To understand the changes made here, we first need to discuss the Key-Value Cache. Inside of the transformer we have 3 vectors that are critical for attention to work key, value, and query. From a high level, attention is how we pass along critical information about the previous tokens to the current token so that it can predict the next token. In the example of self-attention with one head, we multiply the query vector on the current token with the key vectors from the previous tokens and then normalize the resulting matrix (the resulting matrix we call the attention pattern). We now multiply the value vectors with the attention pattern to get the updates to each token. This data is then added to the current tokens embedding so that it now has the context to determine what comes next.

We create the attention pattern for every single new token we create, so while the queries tend to change, the keys and the values are constant. Consequently, the current architectures try to reduce compute time by caching the key and value vectors as they are generated by each successive round of attention. This cache is called the Key-Value Cache.

While architectures like encoder-only and encoder-decoder transformer models have had success, the authors posit that the autoregression shown above, and the speed it allows its models, is the reason why decoder-only models are the most commonly used today.

To understand the YOCO architecture, we have to start out by understanding how it sets out its layers.

For one half of the model, we use one type of attention to generate the vectors needed to fill the KV Cache. Once it crosses into the second half, it will use the KV Cache exclusively for the key and value vectors respectively, now generating the output token embeddings.

This new architecture requires two types of attention efficient self-attention and cross-attention. Well go into each below.

Efficient Self-Attention (ESA) is designed to achieve a constant inference memory. Put differently we want the cache complexity to rely not on the input length but on the number of layers in our block. In the below equation, the authors abstracted ESA, but the remainder of the self-decoder is consistent as shown below.

Lets go through the equation step by step. X^l is our token embedding and Y^l is an intermediary variable used to generate the next token embedding X^l+1. In the equation, ESA is Efficient Self-Attention, LN is the layer normalization function which here was always Root Mean Square Norm (RMSNorm ), and finally SwiGLU. SwiGLU is defined by the below:

Here swish = x*sigmoid (Wg * x), where Wg is a trainable parameter. We then find the element-wise product (Hadamard Product) between that result and X*W1 before then multiplying that whole product by W2. The goal with SwiGLU is to get an activation function that will conditionally pass through different amounts of information through the layer to the next token.

Now that we see how the self-decoder works, lets go into the two ways the authors considered implementing ESA.

First, they considered what is called Gated Retention. Retention and self-attention are admittedly very similar, with the authors of the Retentive Network: A Successor to Transformer for Large Language Models paper saying that the key difference lies in the activation function retention removes softmax allowing for a recurrent formulation. They use this recurrent formulation along with the parallelizability to drive memory efficiencies.

To dive into the mathematical details:

We have our typical matrices of Q, K, and V each of which are multiplied by the learnable weights associated with each matrix. We then find the Hadamard product between the weighted matrices and the scalar . The goal in using is to create exponential decay, while we then use the D matrix to help with casual masking (stopping future tokens from interacting with current tokens) and activation.

Gated Retention is distinct from retention via the value. Here the matrix W is used to allow our ESA to be data-driven.

Sliding Window ESA introduces the idea of limiting how many tokens the attention window should pay attention to. While in regular self-attention all previous tokens are attended to in some way (even if their value is 0), in sliding window ESA, we choose some constant value C that limits the size of these matrices. This means that during inference time the KV cache can be reduced to a constant complexity.

To again dive into the math:

We have our matrices being scaled by their corresponding weights. Next, we compute the head similar to how multi-head attention is computed, where B acts both as a causal map and also to make sure only the tokens C back are attended to.

Read more here:

Understanding You Only Cache Once | by Matthew Gunton | Jun, 2024 - Towards Data Science

Embracing uncertainty, right people, and learning from the data Picture by Cottonbro, on Pexels

This blog post is an updated version of part of a conference talk I gave at GOTO Amsterdam last year. The talk is also available to watch online.

Providing value and positive impact through machine learning product initiatives is not an easy job. One of the main reasons for this complexity is the fact that, in ML initiatives developed for digital products, two sources of uncertainty intersect. On one hand, there is the uncertainty related to the ML solution itself (will we be able to predict what we need to predict with good enough quality?). On the other hand, there is the uncertainty related to the impact the whole system will be able to provide (will users like this new functionality? will it really help solve the problem we are trying to solve?).

All this uncertainty means failure in ML product initiatives is something relatively frequent. Still, there are strategies to manage and improve the probabilities of success (or at least to survive through them with dignity!). Starting ML initiatives on the right foot is key. I discussed my top learnings in that area in a previous post: start with the problem (and define how predictions will be used from the beginning), start small (and maintain small if you can), and prioritize the right data (quality, volume, history).

However, starting a project is just the beginning. The challenge to successfully manage an ML initiative and provide a positive impact continues throughout the whole project lifecycle. In this post, Ill share my top three learnings on how to survive and thrive during ML initiatives:

It is really hard (impossible even!) to plan ML initiatives beforehand and to develop them according to that initial plan.

The most popular project plan for ML initiatives is the ML Lifecycle, which splits the phases of an ML project into business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Although these phases are drawn as consecutive steps, in many representations of this lifecycle youll find arrows pointing backward: at any point in the project, you might learn something that forces you to go back to a previous phase.

This translates into projects where it is really hard to know when they will finish. For example, during the evaluation step, you might realize thanks to model explainability techniques that a specific feature wasnt well encoded, and this forces you to go back to the data preparation phase. It could also happen that the model isnt able to predict with the quality you need, and might force you to go back to the beginning in the business understanding phase to redefine the project and business logic.

Whatever your role in an ML initiative or project is, it is key to acknowledge things wont go according to plan, to embrace all this uncertainty from the beginning, and to use it to your advantage. This is important both to managing stakeholders (expectations, trust) and for yourself and the rest of the team (motivation, frustration). How?

Any project starts with people. The right combination of people, skills, perspectives, and a network that empowers you.

The days when Machine Learning (ML) models were confined to the Data Scientists laptop are over. Today, the true potential of ML is realised when models are deployed and integrated into the companys processes. This means more people and skills need to collaborate to make that possible (Data Scientists, Machine Learning Engineers, Backend Developers, Data Engineers).

The first step is identifying the skills and roles that are required to successfully build the end-to-end ML solution. However, more than a group of roles covering a list of skills is required. Having a diverse team that can bring different perspectives and empathize with different user segments has proven to help teams improve their ways of working and build better solutions (why having a diverse team will make your products better).

People dont talk about this enough, but the key people to deliver a project go beyond the team itself. I refer to these other people as the network. The network is people you know are really good at specific things, you trust to ask for help and advice when needed, and can unblock, accelerate, or empower you and the team. The network can be your business stakeholders, manager, staff engineers, user researchers, data scientists from other teams, customer support team Ensure you build your own network and identify who is that ally who you can go to depending on each specific situation or need.

A project is a continuous learning opportunity, and many times learnings and insights come from checking the right data and monitors.

In ML initiatives there are 3 big groups of metrics and measures that can bring a lot of value in terms of learnings and insights: model performance monitoring, service performance, and final impact monitoring. In a previous post I deep dive into this topic.

Checking at the right data and monitors while developing or deploying ML solutions is key to:

Effectively managing ML initiatives from beginning until end is a complex task with multiple dimensions. In this blogpost I shared, based on my experience first as Data Scientist and lately as ML Product Manager, the factors I consider key when dealing with an ML project: embracing uncertainty, surrounding yourself with the right people, and learning from the data.

I hope these insights help you successfully manage your ML initiatives and drive positive impact through them. Stay tuned for more posts about the intersection of Machine Learning and Product Management 🙂

Read the original here:

Effective Strategies for Managing ML Initiatives | by Anna Via | Jun, 2024 - Towards Data Science

DevOps (Development Operations) and MLOps (Machine Learning Operations) are almost the same and share a wide variety of tools. As a DevOps engineer, you will deploy, maintain, and monitor applications, whereas as an MLOps engineer, you deploy, manage, and motor manufacturing models into production. So, it is beneficial to learn about DevOps tools as it opens a wide array of job opportunities for you. DevOps refers to a set of practices and tools designed to increase a company's ability to deliver applications and services faster and more efficiently than traditional software development processes.

In this blog, you will learn about essential and popular tools for versioning, CI/CD, testing, automation, containerization, workflow orchestration, cloud, IT management, and monitoring applications in production.

Git is the backbone of modern software development. It is a distributed version control tool that allows multiple developers to work on the same codebase without interfering with each other. Understanding Git is fundamental if you are getting started with software development.

Learn about 14 Essential Git Commands for versioning and collaborating on data science projects.

GitHub Actions simplifies the automation of your software workflows, enabling you to build, test, and deploy your code directly from GitHub with just a few lines of code. As a core function of DevOps engineering, mastering Continuous Integration and Continuous Development (CI/CD) is crucial for success in the field. By learning to automate workflows, generate logs, and troubleshoot issues, you will significantly enhance your job prospects.

Remember it is all about experience and portfolio in the operations related careers.

Learn how to automate machine learning training and evaluation by following GitHub Actions For Machine Learning Beginners.

Selenium is a powerful tool primarily used for automating web browser interactions, allowing you to efficiently test your web application. With just a few lines of code, you can harness the power of Selenium to control a web browser, simulate user interactions, and perform automated testing on your web application, ensuring its functionality, reliability, and performance.

Since many servers use Linux, understanding this operating system can be crucial. Linux commands and scripts form the foundation of many operations in the DevOps world, from basic file manipulation to automating the entire workflow. In fact, many seasoned developers rely heavily on Linux scripting, particularly Bash, to develop custom solutions for data loading, manipulation, automation, logging, and numerous other tasks.

Learn about the most commonly used Linux command by checking out Linux for Data Science cheat sheet.

Familiarity with Cloud Platforms like AWS, Azure, or Google Cloud Platform is essential for landing a job in the industry. The majority of services and applications that we use every day are deployed on the Cloud.

Cloud platforms offer services that can help you deploy, manage, and scale applications. By gaining expertise in Cloud platforms, you'll be able to harness the power of scalability, flexibility, and cost-effectiveness, making you a highly sought-after professional in the job market.

Start the Beginners Guide to Cloud Computing and learn how cloud computing works, top cloud platforms, and applications.

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package.

Learn more about Dockers by following the Docker Tutorial for Data Scientists.

Kubernetes is a powerful container orchestration tool that automates the deployment, scaling, and management of containers across diverse environments. As a DevOps engineer, mastering Kubernetes is essential to efficiently scale, distribute, and manage containerized applications, ensuring high availability, reliability, and performance.

Read Kubernetes In Action: Second Edition book to learn about the essential tool for anyone deploying and managing cloud-native applications.

Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud. It enables you to monitor a wide range of metrics and receive alerts in real-time, providing unparalleled insights into your system's performance and health. By learning Prometheus, you will be able to identify issues quickly, optimize systems efficiency, and ensure high uptime and availability.

Terraform, an open-source infrastructure as code (IaC) tool developed by HashiCorp, enables you to provision, manage, and version infrastructure resources across multiple cloud and on-premises environments with ease and precision. It supports a wide range of existing service providers, as well as custom in-house solutions, allowing you to create, modify, and track infrastructure changes safely, efficiently, and consistently.

Ansible is a simple, yet powerful, IT automation engine that streamlines provisioning, configuration management, application deployment, orchestration, and a multitude of other IT processes. By automating repetitive tasks, deploying applications, and managing configurations across diverse environments - including cloud, on-premises, and hybrid infrastructures - Ansible empowers users to increase efficiency, reduce errors, and improve overall IT agility.

Learning about these tools is just the starting point for your journey in the world of DevOps. Remember, DevOps is about more than just toolsit is about creating a culture that values collaboration, continuous improvement, and innovation. By mastering these tools, you will build a solid foundation for a successful career in DevOps. So, begin your journey today and take the first step towards a highly paid and exciting career.

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

Read the original here:

10 Essential DevOps Tools Every Beginner Should Learn - KDnuggets

Traveling abroad has been part of Jimin Jeons life for as long as she can remember. She traveled with her mom during every school vacation, which allowed her to visit 23 countries by the time she was a college student. Being exposed to different cultures from a young age helped her develop a desire to pursue her college education abroad. That brought her to Mason Korea after 12 years of Korean public school education.

While the thought of studying abroad was exciting, I felt burdened by the language barrier to study abroad in the U.S. right after graduating high school, Jeon said. Mason Korea was an alternative to ease that transition by improving my English skills in a more familiar setting in South Korea.

Jeon was part of the first cohort in the newly established Computational and Data Sciences Department at Mason Korea. Although her frequent travels around the world prompted her to major in global affairs, she had her mind set on the world of big data since high school. Thus, after her freshman year, once the new major was opened, she made the jump to the STEM field.

Jeon found she had direct opportunities to engage in data analysis. Her favorite part of Mason Korea was the Career Development Center, which allowed students like her to be exposed to opportunities in data analytics to gain technical hands-on experiences. Her first work experience was through the center as a data science intern at a real estate AI valuation startup during her junior year.

It was a special opportunity to see how the knowledge about programming languages I acquired in the classroom be applied in the real workforce and identify the areas that I need to continue to improve to be a more competent data scientist, said Jeon.

Transitioning to the Fairfax Campus in the fall semester of 2023, Jeon stayed true to her goal of diversifying her experiences. Her last semester at George Mason included working as a teaching assistant for the Computational and Data Sciences Department in the College of Science, performing data cleaning for an on-campus project, and helping students practice their Korean through the language exchange program. She took advantage of the language environment so that she could build her English skills.

Jeon is now a proud graduate in computational and data sciences, one of the few who enrolled in the major in 2020. She is excited about the job opportunities she has and wants to encourage all those who have just closed their four-year journey.

For students just like myself, who have spent their whole life in the Korean education system, going to Mason Korea alone is a challenge, she said. Learning about various topics at a more sophisticated level in a language that you are not familiar with was also not an easy task for me. Yet, the four-year voyage of diverse experiences and success itself shows that I can take on any challenge at any point in my life.

See the rest here:

Going global as Mason Korea's first computational and data sciences graduate - George Mason University

Nearly two and a half millennia since his time, Aristotle and his "virtue ethics" in short, to live a life of good character are every bit as relevant to budding statisticians as the technical skills they learn to build AI models, according to Elizabeth Karns, senior lecturer of statistics and data science at Cornell Bowers CIS and at the ILR School.

An epidemiologist and lawyer, Karns launched Integrated Ethics in Data Science (STSCI 3600), a seven-week course offered twice each spring semester, several years ago in response to what she viewed as a disconnect between statisticians and the high-powered, high-consequence statistical models they were being asked to build.

I started thinking more about algorithms and how we are not preparing students sufficiently to be confronted with workplace pressures to just get the model done Put in the data, don't question it, and just use it, she said.

The problem, as she sees it, is that these models are largely unregulated, have no governing body, and thus skirt rigorous scientific testing and evaluation. Lacking such oversight, ethics and fairness, then, become a matter of discretion on the part of the statisticians developing the models; personal values and virtues are brought into the equation, and this is where Aristotles wisdom proves vital, she said.

At this point in our lack of regulation, we need to depend on ethical people, Karns said. I want students to learn to pause and reflect before making decisions, and to ask, How well does this align with my values? Is this a situation that could lead to problems for the company or users? Is this something I want to be associated with? Thats the core of the class.

For the course, Karns with the help of Cornells Center for Teaching Innovation (CTI) developed an immersive video, Nobodys Fault: An Interactive Experience in Data Science Practice, which challenges students to consider a moral conflict brought about by a bad model.

I tell my students that we're going to be in situations in this class where there's not a clear right or wrong answer, she said. And that's the point to struggle with that ambiguity now and get some comfort in that gray space. That way, when they get out into the workplace, they can be more effective.

To read more about the work Bowers CIS is doing to develop responsible AI, click here. Louis DiPietro is a public relations and content specialist for Cornell Bowers CIS.

More here:

Aristotle, AI and the Data Scientist | The ILR School - Cornell University | ILR School