Llama 3.1 405B

Model Overview

Llama 3.1 405B is Meta's flagship open-weights foundation model. With 405 billion parameters, it is one of the largest and most capable openly available models in the world, rivaling the performance of top closed-source models across a wide range of benchmarks.

Capabilities

• State-of-the-Art Open Weights: Competes directly with leading proprietary models like GPT-4o and Claude 3.5 Sonnet in general knowledge, steerability, math, tool use, and multilingual translation.
• Massive Scale: Trained on over 15 trillion tokens of data using 16,000 H100 GPUs, giving it unprecedented capacity for reasoning and knowledge retention for an open model.
• Extended Context: Supports an expanded context window of 128K tokens, allowing it to process long documents, entire codebases, and extensive chat histories.
• Ecosystem Enabler: Its open availability allows developers to use it for synthetic data generation, model distillation, and fine-tuning, empowering a new wave of specialized AI models.

Example Use Cases

• Model Distillation: Using the 405B model's outputs to train smaller, more efficient models (like the 8B and 70B variants) for specific tasks.
• Synthetic Data Generation: Creating high-quality datasets for training other AI systems.
• Enterprise AI: Deploying powerful AI entirely on-premises for maximum data privacy and security.

Performance & Benchmarks

Llama 3.1 405B pushes the boundaries of open-source AI:

• MMLU (General Knowledge): 88.6%
• HumanEval (Code): 89.0%
• MATH: 73.8%
• Multilingual: Supports 8 languages with high fluency.

Open Source Philosophy

Meta released Llama 3.1 405B under a permissive license (with a 700 million MAU commercial limit) to democratize access to frontier-level AI. Meta believes that open-sourcing AI models drives innovation, improves safety through community scrutiny, and prevents a monopoly on AI technology.

About Meta

Meta is a technology company focused on building products that help people connect, find communities, and grow businesses. Its AI research division, FAIR, is a leader in advancing the state of the art in artificial intelligence.