Hold onto your hats, folks, because the world of Artificial Intelligence has just been given a significant shake-up. Meta has unveiled their latest marvels: the Llama 4 herd, marking what they’re calling “the beginning of a new era of natively multimodal AI innovation”. This isn’t just another incremental update; it’s a leap forward that promises to reshape how we interact with and build upon AI.
At the heart of this announcement are two groundbreaking models available right now: Llama 4 Scout and Llama 4 Maverick. What makes these models so special? Well, for starters, they are natively multimodal. Forget about AI that only understands text – Llama 4 can seamlessly process and understand both text and images. This opens up a universe of possibilities for creating more personalized and intuitive AI experiences.
Let’s take a closer look at these two impressive members of the Llama 4 family:
- Llama 4 Scout: This model boasts 17 billion active parameters and leverages a clever architecture with 16 experts. But the real showstopper here is its industry-leading context window of 10 million tokens. To put that into perspective, it means Scout can remember and process an absolutely enormous amount of information at once – imagine summarising lengthy documents or understanding intricate codebases with remarkable ease. What’s more, it’s powerful enough to outperform previous Llama models and even rivals like Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1 on various benchmarks, all while fitting onto a single NVIDIA H100 GPU .
- Llama 4 Maverick: Sharing the same 17 billion active parameters as Scout, Maverick kicks things up a notch with a staggering 128 experts. This sophisticated design makes it a best-in-class multimodal model, outperforming even giants like GPT-4o and Gemini 2.0 Flash across a wide range of tests. It even holds its own against the significantly larger DeepSeek v3.1 in reasoning and coding tasks. Meta highlights its best-in-class performance to cost ratio, making advanced AI capabilities more accessible.
These models aren’t just conjured out of thin air. They are the result of rigorous research and a novel training approach. A crucial element in their architecture is the mixture-of-experts (MoE). Think of an MoE model as having a team of specialized AI brains. When it encounters a piece of information, it intelligently routes that information to the most relevant expert (or a small group of experts) to process it. This makes the models incredibly efficient both during training and when they’re being used, allowing for higher quality within a given compute budget. For example, while Llama 4 Maverick has a total of 400 billion parameters, only 17 billion are actively engaged for any single task. This clever trick reduces serving costs and latency, making deployment on platforms like a single NVIDIA H100 DGX as the host to be feasible.
How does Llama 4 compare to previous models?
The Llama 4, represented by Llama 4 Scout and Llama 4 Maverick, offers several key advancements and improvements compared to previous generation Llama models, according to the sources.
Firstly, Llama 4 Scout is described as “more powerful than all previous generation Llama models”. This immediately establishes a general superiority in terms of capability.
Secondly, Llama 4 offers native multimodality, a significant step forward as it integrates text and vision tokens into a unified model backbone using early fusion. This allows for joint pre-training with large amounts of unlabeled text, image, and video data, a capability not explicitly mentioned for previous Llama versions in this source. The vision encoder has also been improved in Llama 4, based on MetaCLIP but trained separately to better adapt to the LLM.
Thirdly, Llama 4 models utilise a mixture-of-experts (MoE) architecture, which is a first for the Llama series. This architecture leads to more compute-efficient training and inference, delivering higher quality for a fixed training FLOPs budget compared to dense models. For example, Llama 4 Maverick has 17 billion active parameters but 400 billion total parameters, improving inference efficiency by activating only a subset of the total parameters during serving.
In terms of context length, Llama 4 Scout offers an “industry-leading context window of 10M” tokens, a dramatic increase from the 128K context length of Llama 3. Llama 4 Scout was pre-trained and post-trained with a 256K context length, enabling advanced length generalisation.
Regarding performance, Llama 4 Scout is described as the “best multimodal model in the world in its class”. It reportedly delivers better results than Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1 across a broad range of benchmarks. Similarly, Llama 4 Maverick is also the “best multimodal model in its class”, outperforming GPT-4o and Gemini 2.0 Flash across various benchmarks and achieving comparable results to DeepSeek v3 on reasoning and coding with less than half the active parameters. Llama 4 Maverick also offers a better performance-to-cost ratio.
The development of Llama 4 benefited from distillation from Llama 4 Behemoth, a significantly larger and more powerful teacher model. This codistillation process resulted in substantial quality improvements in Llama 4 Scout and Llama 4 Maverick.
Furthermore, Llama 4 demonstrates improved multilingual capabilities, being pre-trained on 200 languages, including over 100 with over 1 billion tokens each, representing a 10x increase in multilingual tokens compared to Llama 3.
In terms of bias, Llama 4 shows significant progress compared to Llama 3.3. It refuses less on debated political and social topics, is dramatically more balanced in which prompts it refuses, and exhibits a rate of strong political lean comparable to Grok, which is half the rate of Llama 3.3.
The post-training process for Llama 4 models has also been revamped, involving lightweight SFT on harder data, online reinforcement learning with continuous data filtering, and lightweight DPO, leading to improved balance between intelligence and conversational abilities.
Finally, Llama 4 models are designed for efficient deployment, with Llama 4 Scout fitting on a single NVIDIA H100 GPU (with Int4 quantisation) and Llama 4 Maverick fitting on a single H100 host.
In summary, Llama 4 represents a significant leap forward from previous Llama models in terms of multimodality, architecture, context length, performance across various benchmarks, multilingual capabilities, bias reduction, and training methodologies.
The Inner Workings: Architecture and Training
The secret sauce behind Llama 4’s capabilities lies in its innovative architecture and the meticulous training process. Beyond the Mixture of Experts, native multimodality is a cornerstone of the design. Llama 4 incorporates early fusion, which means it integrates text and vision tokens right at the beginning of its processing, using a unified model backbone. This is a significant advancement, allowing the model to be pre-trained on vast datasets of unlabeled text, images, and videos simultaneously. The vision encoder itself has also been improved, building upon MetaCLIP but trained specifically to better align with the language model.
The training methodology also introduces new techniques. MetaP is a novel technique that allows for reliable setting of critical model hyperparameters, ensuring consistent performance across different scales and training configurations. Furthermore, Llama 4 has been pre-trained on a massive 30 trillion tokens of data, encompassing over 200 languages – ten times more multilingual tokens than its predecessor, Llama 3. This enhanced multilingual capability will undoubtedly fuel more inclusive and globally applicable AI applications.
To achieve the impressive 10 million token context window of Llama 4 Scout, Meta employed “mid-training” techniques, using specialized datasets to extend the model’s ability to handle long sequences of information. A key architectural innovation contributing to this is the use of interleaved attention layers without positional embeddings, a design they call iRoPE architecture. This architecture, with its “interleaved” attention layers and “Rotary Position Embeddings” (RoPE) in most layers, is a step towards supporting potentially “infinite” context length in the future.
Behind the impressive performance of Scout and Maverick is an even larger and more powerful model: Llama 4 Behemoth. This colossal model boasts 288 billion active parameters and nearly two trillion total parameters. While still in training, Behemoth is already demonstrating state-of-the-art performance on math, multilingual, and image benchmarks, outperforming models like GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on STEM-focused evaluations. Behemoth serves as a teacher model, and through a process called codistillation, its vast knowledge is transferred to the smaller, more deployable Llama 4 Scout and Maverick models, leading to significant quality improvements.
Post-training also plays a crucial role in shaping the capabilities of these models. For Llama 4 Maverick, Meta revamped its post-training pipeline, using a sequence of lightweight supervised fine-tuning (SFT), online reinforcement learning (RL), and lightweight direct preference optimization (DPO). A key insight was that excessive SFT and DPO could limit the model’s exploration during RL, hindering accuracy in reasoning and coding. To address this, they strategically filtered out “easy” data and focused on harder prompts during the online RL stage, even implementing a continuous online RL strategy with adaptive data filtering. This meticulous approach resulted in a general-purpose chat model with industry-leading intelligence and image understanding. The post-training for the massive Llama 4 Behemoth required an even more significant overhaul, including aggressive data pruning and a refined RL recipe focused on sampling hard prompts and dynamically filtering zero-advantage prompts.
Comparing LLama 4 Scout and LLama 4 maverick
Llama 4 Scout and Llama 4 Maverick are the first models in the Llama 4 herd, designed to enable more personalised multimodal experiences. Both models represent a significant advancement in the Llama ecosystem. Here’s a comparison and contrast based on the provided source:
Similarities:
- Both Llama 4 Scout and Llama 4 Maverick have 17 billion active parameters.
- Both are natively multimodal models, incorporating early fusion to integrate text and vision tokens into a unified model backbone. This allows them to be jointly pre-trained with large amounts of unlabeled text, image, and video data.
- They are both considered general-purpose models.
- Both models have been codistilled from Llama 4 Behemoth, a 288 billion active parameter teacher model, resulting in substantial quality improvements.
- You can download both Llama 4 Scout and Llama 4 Maverick today on llama.com and Hugging Face. They are also used on the web in Meta AI which can be tried in WhatsApp, Messenger, Instagram Direct, and on the web.
Key Differences:
- Number of Experts: Llama 4 Scout has 16 experts, while Llama 4 Maverick has a significantly larger number with 128 experts.
- Context Window: Llama 4 Scout offers an industry-leading context window of 10 million tokens. In contrast, the source does not explicitly state the context window size for Llama 4 Maverick, but it highlights Scout’s as a key differentiator. Scout was pre-trained and post-trained with a 256K context length, empowering its base model with advanced length generalisation capability.
- Performance on Benchmarks:
- Llama 4 Scout delivers better results than Gemma 3, Gemini 2.0 Flash-Lite, and Mistral 3.1 across a broad range of widely reported benchmarks. It also exceeds comparable models on coding, reasoning, long context, and image benchmarks and offers stronger performance than all previous Llama models.
- Llama 4 Maverick is the best multimodal model in its class, outperforming GPT-4o and Gemini 2.0 Flash across a broad range of widely reported benchmarks. It achieves comparable results to the new DeepSeek v3.1 on reasoning and coding—at less than half the active parameters.
- Intended Use Cases and Strengths:
- Llama 4 Scout with its 10 million token context length opens up possibilities for tasks like multi-document summarisation, parsing extensive user activity for personalised tasks, and reasoning over vast codebases. It is also best-in-class on image grounding, able to align user prompts with relevant visual concepts and anchor model responses to regions in the image.
- Llama 4 Maverick offers unparalleled, industry-leading performance in image and text understanding, making it suitable for creating sophisticated AI applications that bridge language barriers. It’s considered a “product workhorse” model for general assistant and chat use cases, excelling in precise image understanding and creative writing. It also offers a best-in-class performance to cost ratio with an experimental chat version scoring an ELO of 1417 on LMArena.
- Hardware Requirements: Llama 4 Scout fits on a single NVIDIA H100 GPU (with Int4 quantisation). Llama 4 Maverick fits on a single H100 host, making deployment relatively easy.
What motivated the development of Llama 4?
The development of the Llama 4 herd, specifically Llama 4 Scout and Llama 4 Maverick, was motivated by several key factors, as highlighted in the sources i researched:
- Enabling more personalised multimodal experiences: A primary motivation was to create models that would allow people to build more personalised multimodal experiences. This suggests a drive towards AI that can better understand and interact with the world through various modalities like text and vision.
- Advancing the Llama ecosystem: These models mark the beginning of a new era for the Llama ecosystem, indicating a continuous effort to push the boundaries of their AI capabilities. The development signifies a step forward in their foundational models.
- Openly available leading models: Meta believes it’s important for leading models and systems to be openly available so that everyone can contribute to building the future of personalised experiences. Making Llama 4 Scout and Maverick open-weight reflects this commitment to open innovation.
- Creating more intelligent systems: There’s a belief that the most intelligent systems need to be capable of taking generalised actions, conversing naturally with humans, and working through challenging problems they haven’t encountered before. Developing Llama 4 is a step towards giving Llama “superpowers” in these areas.
- Improving products for people and opportunities for developers: This advancement in AI is intended to lead to better products for users on Meta’s platforms and create more opportunities for developers to innovate in consumer and business use cases.
- Offering multimodal intelligence at a compelling price: The new models are designed to offer multimodal intelligence while outperforming models of significantly larger sizes, suggesting a motivation for efficiency and cost-effectiveness.
- Pushing the boundaries of context length and performance: The development of Llama 4 Scout, with its industry-leading 10 million token context window, and Llama 4 Maverick, with its top-tier multimodal performance, indicates a drive to excel in specific capabilities and outperform existing models in their class.
- Learning from a powerful teacher model (Llama 4 Behemoth): The codistillation of Llama 4 Scout and Maverick from Llama 4 Behemoth, a highly capable 288 billion parameter model, was a significant motivation to enhance the quality and performance of the smaller models. Llama 4 Behemoth’s superior performance on STEM benchmarks reinforced its suitability as a teacher.
- Addressing limitations of previous models: The improvements in areas like bias reduction in Llama 4 compared to Llama 3 suggest a motivation to address known issues and create more balanced and reliable models.
- Supporting a full technology stack: The development of these models is part of a focus on the entire AI ecosystem, including product integrations, indicating a desire to bring advanced AI capabilities to real-world applications.
- Driving innovation: Ultimately, the development of Llama 4 is driven by a core belief that openness fosters innovation and benefits developers, Meta, and the wider world. The release of these models is intended to empower the community to build new and exciting experiences.
How Does LLama 4 Address LLM Bias?
Llama 4 addresses the well-known issue of bias in leading LLMs, which historically have leaned left on debated political and social topics due to the nature of internet training data. Meta’s goal is to remove bias from their AI models and ensure Llama can understand and articulate both sides of contentious issues. They aim to make Llama more responsive, able to answer questions and respond to various viewpoints without judgment or favouring specific opinions.
Significant improvements have been made in Llama 4 compared to Llama 3 in addressing this issue:
- Reduced refusals on debated topics: Llama 4 refuses less on debated political and social topics overall, decreasing from 7% in Llama 3.3 to below 2%.
- Improved balance in refusals: Llama 4 demonstrates dramatically more balance in which prompts it refuses to respond to. The proportion of unequal response refusals is now less than 1% on a set of debated topical questions.
- Comparable political lean to Grok: Testing shows that Llama 4 responds with a strong political lean at a rate comparable to Grok, which is half the rate observed in Llama 3.3 on contentious political or social topics.
While Meta is proud of this progress, they acknowledge that more work is needed and they will continue to strive to further reduce this rate.
The approach to addressing bias involves ongoing efforts to make Llama more responsive and balanced in its viewpoints. The reported improvements in refusal rates and balance suggest that the strategies employed in the development of Llama 4 have been effective in mitigating some aspects of LLM bias.
What Performance Advantage Does LLama4 Behemoth Offer?
I would say that drawing on the information in the source, Llama 4 Behemoth offers several significant performance advantages.
Firstly, Llama 4 Behemoth is described as “our most powerful yet and among the world’s smartest LLMs”. This highlights its overall high level of intelligence and capability.
Specifically, it demonstrates state-of-the-art performance for non-reasoning models on math, multilinguality, and image benchmarks. Furthermore, on STEM-focused benchmarks such as MATH-500 and GPQA Diamond, Llama 4 Behemoth outperforms GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro. This clearly positions it as a leading model in these specific areas.
Another key advantage of Llama 4 Behemoth is its role as a “teacher model”. It was used to codistill the Llama 4 Maverick model, which resulted in substantial quality improvements across end task evaluation metrics. This indicates that Behemoth possesses superior knowledge and capabilities that can be effectively transferred to smaller models. The source mentions a “novel distillation loss function that dynamically weights the soft and hard targets through training”, which contributed to this effective knowledge transfer.
In terms of training efficiency, while pre-training using FP8 and 32K GPUs, Llama 4 Behemoth achieved 390 TFLOPs/GPU. The post-training recipe for Behemoth, involving the pruning of 95% of SFT data and large-scale reinforcement learning, led to “even more significant improvements in reasoning and coding abilities of the model”. This suggests advanced capabilities in complex reasoning and coding tasks.
The source also mentions that the development of Llama 4 Behemoth involved revamping the RL infrastructure due to its unprecedented scale, leading to a “~10x improvement in training efficiency over previous generations”. While this doesn’t directly describe a performance advantage in terms of output quality, it signifies advancements in the underlying technology that enabled the creation of such a powerful model.
In summary, Llama 4 Behemoth offers performance advantages through its:
- Overall high intelligence and power.
- Superior performance on STEM benchmarks compared to leading models like GPT-4.5.
- State-of-the-art capabilities in math, multilinguality, and image benchmarks for non-reasoning tasks.
- Effective role as a teacher model, significantly enhancing the quality of smaller models like Llama 4 Maverick.
- Advanced reasoning and coding abilities achieved through its specialized post-training process.
It is important to note that Llama 4 Behemoth is still training and has not yet been released. However, its demonstrated capabilities make it a significant advancement in the Llama ecosystem.
How is llama4 available to developers for use?
Llama 4 Scout and Llama 4 Maverick are available to developers for use through several avenues or platforms.
Firstly, developers can download the Llama 4 Scout and Llama 4 Maverick models directly from llama-downloads and Hugging Face. This open-weight availability allows developers to integrate the models into their own projects and workflows.
Secondly, Meta is making these models available via their partners in the coming days. The source also lists a wide array of partners across cloud and data platforms, edge silicon providers, and global service integrators who are supporting this work, suggesting broader accessibility in the near future.
Thirdly, developers can explore the capabilities of Llama 4 through Meta AI, which is built using Llama 4 and can be tried in WhatsApp, Messenger, Instagram Direct, and on the Meta.AI website. This allows developers to understand the potential of the models and potentially integrate with these platforms.
Meta believes that making leading models and systems openly available is crucial for everyone to build the future of personalised experiences, highlighting their commitment to empowering developers. They consider Llama 4 Scout and Llama 4 Maverick as the “best choices for adding next-generation intelligence to your products”.
Furthermore, the models are designed to be efficient, with Llama 4 Scout fitting on a single NVIDIA H100 GPU (with Int4 quantisation) and Llama 4 Maverick fitting on a single H100 host, making deployment relatively accessible.
Meta has also open-sourced several safeguard tools Llama-Guard-3-8Blike Llama Guard and Prompt Guard, which developers can integrate into their Llama-supported applications to ensure safety and policy compliance.
In summary, developers can access and utilise Llama 4 through direct downloads of the model weights, future availability via partners across various platforms, and by experimenting with Meta AI powered by Llama 4. This open and accessible approach aims to foster innovation and enable developers to build advanced AI-powered experiences.
Briefly outline the post-training process for Llama 4 models
The post-training process for the new Llama 4 models involves a refined approach compared to previous iterations. For Llama 4 Maverick, the process consists of the following key stages:
- Lightweight Supervised Fine-Tuning (SFT): Initially, a lightweight SFT is performed, but with a specific focus on a harder dataset. This is achieved by using Llama models as a judge to filter out more than 50% of the data tagged as easy and performing SFT on the remaining harder set.
- Online Reinforcement Learning (RL): This stage is conducted online and multimodally. A continuous online RL strategy is implemented where the model is trained and then used to continually filter and retain only medium-to-hard difficulty prompts. This iterative process helps to improve performance efficiently.
- Lightweight Direct Preference Optimization (DPO): A final lightweight DPO step is used to handle corner cases related to model response quality, aiming to strike a good balance between the model’s intelligence and conversational abilities.
It’s worth noting that for the even larger Llama 4 Behemoth, the post-training process involved pruning a significant portion (95%) of the SFT data to focus on quality and efficiency, followed by large-scale reinforcement learning (RL) with an emphasis on sampling hard prompts and dynamically filtering out prompts with zero advantage. This highlights that the post-training recipe is adapted based on the model’s scale. For Llama 4 Scout, the source mentions it was both pre-trained and post-trained with a 256K context length.
Putting Llama 4 to Use: Performance, Safety, and Getting Started
The impressive architecture and training translate into tangible performance gains. Llama 4 Maverick excels in image and text understanding, making it ideal for applications requiring sophisticated AI that can bridge language barriers and understand visual cues. It serves as a “product workhorse” for general assistant and chat functionalities, demonstrating strength in precise image understanding and creative writing. Llama 4 Scout, while also a general-purpose model, shines with its unprecedented 10 million token context length, opening doors for tasks like multi-document summarization and reasoning over vast amounts of code. It also boasts best-in-class performance for its size on various benchmarks, including coding, reasoning, long context, and image understanding, and excels in image grounding, accurately aligning user prompts with visual concepts.
Beyond performance, Meta has placed a strong emphasis on safeguards and protections in the development of Llama 4. They’ve integrated mitigations at every stage, from pre-training data filtering to post-training techniques and tunable system-level safeguards. They have even open-sourced tools like:
- Llama Guard: An input/output safety model to detect policy violations.
- Prompt Guard: A classifier to identify malicious prompts and prompt injections.
- CyberSecEval: Evaluations to help developers understand and reduce cybersecurity risks.
Meta believes in providing developers with open solutions that can be tailored to their specific applications. They also conduct rigorous evaluations and red-teaming, including their new Generative Offensive Agent Testing (GOAT) framework, to proactively identify and mitigate potential risks.
Addressing bias in LLMs is another key focus. Meta acknowledges the historical left-leaning bias in many large language models due to training data. With Llama 4, they have made significant strides in reducing this bias. The models now refuse less on debated political and social topics and exhibit a much more balanced refusal rate across different viewpoints. Their testing shows Llama 4 responds with a strong political lean at a rate comparable to Grok and significantly lower than Llama 3.3, though they acknowledge there’s still work to be done.
So, how can you get your hands on this exciting new technology? Right now, you can download Llama 4 Scout and Llama 4 Maverick today on llama.com and Hugging Face. Furthermore, you can experience Meta AI, which is built using Llama 4, in platforms you likely use every day: WhatsApp, Messenger, Instagram Direct, and on the web via the MetaAI’s Website.
