Microsoft Releases Phi-2
All you need are textbooks and attention! Phi-2 outperforms Mistral-7B with half of the parameters!
Created on December 12|Last edited on December 12
Comment
Microsoft Research's Machine Learning Foundations team has recently introduced Phi-2, a new 2.7 billion-parameter language model. This model is a continuation of their earlier efforts with Phi-1 and Phi-1.5, which showed exceptional abilities in Python coding and common sense reasoning. Phi-2 stands out for its compact size yet powerful performance, rivaling models many times its size.
The key to Phi-2's success lies in its training approach. The model uses high-quality, "textbook-quality" data, including synthetic datasets specifically created to teach common sense reasoning and general knowledge. This approach is based on the idea that training data quality is crucial for model performance. Additionally, Phi-2 employs scaled knowledge transfer from the 1.3 billion parameter model, Phi-1.5, which helps in accelerating training convergence and improving benchmark scores.
Improvement over 1.5
In terms of performance, Phi-2 has shown to outperform its predecessor, Phi-1.5, across various tasks like common sense reasoning, language understanding, math, and coding. It competes with or even surpasses much larger models such as Mistral-7B and Llama-2 and shows comparable capabilities to Google's Gemini Nano 2 despite its smaller size.
An important aspect of Phi-2 is its relatively better behavior in terms of toxicity and bias compared to other models, despite not undergoing alignment through reinforcement learning from human feedback (RLHF). This is attributed to the tailored data curation technique employed during its training.
High Performance
Phi-2 has been rigorously tested on academic benchmarks and Microsoft's internal proprietary datasets. Its performance has consistently been impressive, outperforming models like Mistral-7B and Llama-2 in various tasks. Notably, it has shown adeptness in multi-step reasoning tasks, such as coding and math problems.
The model's practical applications have been demonstrated through tests on commonly used prompts in the research community. Phi-2 effectively solved physics problems and was able to correct student mistakes, illustrating its practical utility and effectiveness.
Available on Azure AI Studio
Phi-2 is now available in the Azure AI Studio model catalog, making it accessible for further research and development in the field of language models. This release signifies a notable advancement in the field of AI, particularly in the development and application of small language models. Its performance and capabilities mark it as a significant tool for both academic research and practical AI implementations.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.