Eagle 7B: A New RNN LLM
RNN's are all you need?
Created on January 29|Last edited on January 29
Comment
Eagle 7B, a new artificial intelligence model, represents a significant step forward in the realm of language processing. Built on the innovative RWKV-v5 architecture, a modified version of the classic Recurrent Neural Network (RNN), this model is making strides in both multi-lingual capabilities and efficiency. With its ability to handle an extensive range of languages and its environmentally conscious design, Eagle 7B may be the beginning of a shift away from attention and back to RNN’s for many use cases.
Pros and Cons of the RNN
To understand the significance of Eagle 7B, it's helpful to understand the basics of the RNN and how RWKV-v5 modifies it. Classic RNNs are a foundational concept in AI, designed to process sequences of data (like language) by maintaining a 'hidden state' – a memory of sorts that carries information from previously processed data to help in understanding the current input. However, RNNs have limitations, especially when it comes to training large models, due to their sequential processing nature, which is not ideal for parallel computation, and also faces issues with problems like vanishing gradients.
RWKV-v5, the architecture behind Eagle 7B, addresses these limitations. It restructures the RNN approach by dividing the network into multiple layers that can handle parts of the data processing task in parallel. This change allows Eagle 7B to efficiently handle large-scale language data, crucial for a model that can be trained on its extensive dataset of 1.1 trillion tokens
Efficient Inference
Eagle 7B stands out for its efficiency. Thanks to the RWKV-v5 architecture's design, which lowers inference costs(by 10x-100x), Eagle 7B is recognized as one of the greenest models in its class. This aspect is particularly important in today's context, where the environmental impact of technology is a growing concern.
Multilingual Benchmarks
The performance of Eagle 7B has been rigorously tested across multiple benchmarks to evaluate its linguistic capabilities. In multi-lingual assessments, Eagle 7B underwent testing using benchmarks such as xLAMBDA, xStoryCloze, xWinograd, and xCopa, covering a comprehensive range of 23 languages. These benchmarks are designed to gauge common sense reasoning in each language. The results from these tests have indicated a substantial improvement in multi-lingual performance when comparing the RWKV v4 and the enhanced v5 architectures, both utilizing the v2 world dataset.

English Benchmarks
In terms of English language performance, Eagle 7B's capabilities were examined through 12 distinct benchmarks, focusing on both commonsense reasoning and world knowledge. The model demonstrated a significant leap in performance when transitioning from the RWKV v4 to the v5 architecture. In particular, Eagle 7B has shown to compete closely and even surpass established models like Falcon and LLaMA2 in several benchmarks, including LAMBADA, StoryCloze16, WinoGrande, HeadQA_en, and Sciq. This level of performance places Eagle 7B on par with the expected standards for transformers trained with a similar token count.
When compared with other prominent models, such as Mistral-7B – which is rumored to be trained on a massive 2-7 trillion tokens – Eagle 7B is steadily narrowing the performance gap. The RWKV team is not resting on its laurels; they plan to further enhance Eagle 7B by training it on an additional 1 trillion tokens. This ambitious effort is aimed at outperforming models like LLaMA2 and possibly reaching or even surpassing the prowess of Mistral-7B. The ongoing development and enhancement of Eagle 7B underscore its potential as a leading model in the realm of AI language processing, both in English and across a multitude of other languages.

Apache 2.0 License
Eagle 7B's release under the Apache 2.0 license, making it freely available for both personal and commercial use, further emphasizes its accessibility. This move aligns with the broader goal of building inclusive AI systems that cater to diverse global needs, not limited by language barriers. The model's requirement for additional fine-tuning for specific applications also demonstrates its versatility.
In summary, Eagle 7B represents a notable advancement in AI, especially in language processing. Its use of the RWKV-v5 architecture, a modification of the classic RNN, allows it to efficiently process large volumes of multi-lingual data, making it a powerful tool in the global AI toolkit. The model's environmental efficiency and accessibility further enhance its appeal, marking it as a significant development in the world of artificial intelligence.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.