Falcon 180B: A New Giant in Openly Available Language Models
Falcon just got its wings, with a 4.5x boost in parameters!
Created on September 6|Last edited on September 6
Comment
Hugging Face now hosts Falcon 180B, a language model that takes the throne as the most expansive open-source model to date. Developed by TII, this model is endowed with 180 billion parameters and trained on a colossal 3.5 trillion-token dataset.
What is Falcon 180B?
Falcon 180B is the latest addition to TII's Falcon series, and it's significantly more powerful. The model leverages advanced architectural features like multiquery attention and was trained on Amazon SageMaker, utilizing around 7 million GPU hours. This makes it 2.5 times larger than Llama 2, and utilizes 4 times more computing resources.
Dataset and Capabilities
The RefinedWeb dataset serves as the primary training material, covering about 85% of the data. Falcon 180B also trains on a diverse range of documents, from technical papers to chat data. The model exhibits unparalleled performance across a range of natural language tasks, challenging even proprietary models like PaLM-2.
Commercial Use
Although designed for broad application, Falcon 180B does come with commercial limitations.
Performance Metrics
In head-to-head comparisons, Falcon 180B outclasses other open-source contenders. It scores 68.74 on the Hugging Face Leaderboard, comfortably above Llama 2’s 67.35. Its metrics remain robust even when the model is quantized, affirming its versatile capabilities.
Demo
If you’re interested in trying out Falcon 180B, it's fully integrated into the Hugging Face ecosystem from version 4.33. A demo is also available for those looking to get a firsthand experience.
Falcon 180B breaks new ground as an open-source language model. Its superior architecture, vast training dataset, and top-notch performance make it a powerful resource for both research and practical applications. With the backing of TII and Hugging Face, it's poised to become a critical tool in advancing our understanding and utilization of natural language processing technologies.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.