DeepSeek-R1: Advancing Large Language Model Reasoning
Created on January 20|Last edited on January 20
Comment
DeepSeek-R1 represents the next step in reasoning-focused large language model (LLM) development, following the release of DeepSeek-R1-Zero. DeepSeek-R1-Zero was trained entirely using large-scale reinforcement learning (RL) without relying on supervised fine-tuning (SFT). This allowed the model to explore reasoning through techniques like chain-of-thought (CoT), self-verification, and reflection, all emerging naturally. However, challenges such as language mixing, endless repetition, and low readability in DeepSeek-R1-Zero revealed the need for further improvements. DeepSeek-R1 addresses these limitations by integrating cold-start data before the RL process, significantly enhancing reasoning capabilities. As a result, DeepSeek-R1 achieves performance comparable to OpenAI-o1 across tasks involving mathematics, code, and reasoning.
To support further research and innovation, DeepSeek-AI has open-sourced both DeepSeek-R1 and DeepSeek-R1-Zero, along with six dense models distilled from DeepSeek-R1. Among these, DeepSeek-R1-Distill-Qwen-32B sets new performance benchmarks, outperforming OpenAI-o1-mini across multiple evaluations.
Innovative training with reinforcement learning
The development of DeepSeek-R1-Zero marked a breakthrough in the application of reinforcement learning for reasoning tasks. By bypassing supervised fine-tuning, the model relied entirely on RL to discover and refine reasoning behaviors, such as creating detailed CoT outputs. DeepSeek-R1-Zero validated that purely RL-based reasoning training is feasible and effective, paving the way for future advancements in LLMs.
The DeepSeek-R1 pipeline builds on this foundation with a two-stage RL process aimed at uncovering better reasoning patterns and aligning the model with human preferences. Additionally, two SFT stages were introduced as a "seed" to enhance both reasoning and general language capabilities. This hybrid approach ensures stronger performance across reasoning-intensive tasks, creating a model that is more aligned with user needs.
Smaller models through distillation
In addition to developing powerful large models, the DeepSeek-R1 project focuses on distillation, transferring reasoning capabilities from larger models to smaller ones. This process allows for smaller models to outperform baseline RL-trained small models. Using reasoning data generated by DeepSeek-R1, six dense models—ranging from 1.5 billion to 70 billion parameters—were fine-tuned for use by the research community. These distilled models achieve exceptional benchmark results, combining efficiency and high performance. Models such as DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B are now available as open-source resources.
Model downloads and compatibility
DeepSeek-AI has released all DeepSeek-R1 and DeepSeek-R1-Distill models through platforms like Hugging Face. The flagship models, DeepSeek-R1 and DeepSeek-R1-Zero, feature 671 billion total parameters, with 37 billion activated during reasoning tasks. Distilled versions, such as DeepSeek-R1-Distill-Qwen-32B, offer high performance with reduced computational demands.
The models can be run locally using frameworks such as vLLM and SGLang.
Benchmarking and performance evaluation
DeepSeek-R1 and its distilled counterparts have undergone extensive evaluation across a wide range of benchmarks. On English reasoning datasets like MMLU and DROP, DeepSeek-R1 delivers performance comparable to or better than state-of-the-art models like OpenAI-o1. For math-heavy tasks such as AIME and MATH-500, the models achieve some of the highest pass@1 scores, including a remarkable 97.3% on MATH-500. In coding benchmarks such as Codeforces, DeepSeek-R1 demonstrates elite performance, nearing the ratings achieved by top models like OpenAI-o1-1217.
Smaller distilled models also excel in evaluation, proving that reasoning capabilities can be effectively transferred to models with fewer parameters. For example, DeepSeek-R1-Distill-Qwen-32B outperforms most competitors on coding, math, and reasoning benchmarks while maintaining an efficient parameter count.
Open-source collaboration and accessibility
DeepSeek-AI is committed to advancing LLM research and accessibility. All models, from the flagship DeepSeek-R1 to the distilled versions, are available under open-source licenses, supporting commercial use and modifications. Researchers and developers can utilize these models to create derivative works, including fine-tuning for specific applications.
For interactive use, DeepSeek-R1 is accessible through chat platforms and an OpenAI-compatible API on DeepSeek’s official website and platform. These resources aim to make high-performance reasoning models accessible to a wider audience.
Conclusion
DeepSeek-R1 is a significant step forward in reasoning-focused LLM development. By combining reinforcement learning with strategic SFT and distillation techniques, it sets new benchmarks for both large and small models across diverse tasks. With its open-source release, DeepSeek-AI enables the broader research community to benefit from these advancements, fostering innovation and collaboration in AI reasoning and language modeling.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.