Open Source Mixture-of-Agents Outperforms GPT-4o
By combining multiple open source models, researchers outperform GPT-4o!
Created on June 24|Last edited on June 24
Comment
Collaboration appears to be the key to unlocking new levels of performance for LLMs. Researchers from Duke University, Together AI, the University of Chicago, and Stanford University have introduced an innovative approach called Mixture-of-Agents (MoA) that enhances the capabilities of large language models (LLMs). This method leverages the strengths of multiple LLMs working together, producing results that surpass the current state-of-the-art.
The Concept
The idea behind MoA is simple yet powerful. By organizing multiple LLM agents in a layered structure, where each agent processes and refines outputs from the previous layer, the collective system generates more accurate and coherent responses. This iterative process continues until a final, high-quality output is produced.
Implementation Details
The MoA methodology is designed to maximize the performance of LLMs through collaborative processing. Here is how it is implemented:
Layered Structure: MoA consists of multiple layers, each containing several LLM agents. Each agent in a layer independently generates responses to a given prompt.
Initial Prompting: In the first layer, all agents receive the same initial prompt. Each agent then generates its own response based on this prompt.
Intermediate Outputs: The responses from the first layer are collected and passed as auxiliary information to the agents in the second layer. This means each agent in the second layer has access to all responses generated by the agents in the first layer.
Refinement Process: Agents in the second layer use the collected responses as additional context to generate their own refined responses. This process is repeated across multiple layers, with each layer's agents refining the outputs of the previous layer.
Final Aggregation: The final layer's agents synthesize the information from all previous layers to produce a single, high-quality response. This layer uses a special Aggregate-and-Synthesize prompt to ensure the final output is comprehensive, accurate, and well-structured.
Mathematical Formulation
The mathematical representation of the MoA process can be summarized as follows:
Here is an explanation to the equation:
yi: This represents the output of the i-th layer in the Mixture-of-Agents framework.
⊕: This denotes the concatenation operation.
Ai,j(xi): This is the output of the j-th agent in the i-th layer when given input xi.
xi: This is the input to the i-th layer.
x1: This is the original input to the entire Mixture-of-Agents model.
xi+1: This is the input to the next (i+1)-th layer, which is the same as the output of the i-th layer (yi).
Each agent in a given layer processes the input it receives and generates an output. These outputs are then combined with the original input and passed to the next layer. This iterative process continues across multiple layers, refining the responses until the final output is generated.
Key Components
Within the MoA framework, agents can play the role of proposers or aggregators. Proposers generate diverse initial responses, while aggregators synthesize these responses into a cohesive output. The selection of agents for each role is based on their performance metrics and diversity considerations. The success of MoA heavily relies on the diversity of the models used. Different LLMs bring unique strengths to the table, enhancing the overall quality of the final output through varied perspectives.
Advantages of MoA
By leveraging multiple models, MoA enhances the quality of responses beyond what a single model can achieve. This approach reduces the need for extensive retraining and scaling of individual models, making it more cost-effective. Additionally, MoA can be applied to any set of LLMs, providing a scalable and adaptable solution for various AI applications.
The Mixture-of-Agents approach represents a significant advancement in AI, demonstrating that collaboration among models can lead to superior performance. This methodology enhances the interpretability and accessibility of AI-driven systems, paving the way for more effective and trustworthy AI applications across various domains.
Results
The Mixture-of-Agents approach has demonstrated significant improvements across several key benchmarks. On the AlpacaEval 2.0, MoA achieved a leading score of 65.1%, significantly surpassing the previous best of 57.5% by GPT-4 Omni. In the MT-Bench, although improvements were more incremental, MoA still secured top positions, proving its effectiveness even in highly optimized benchmarks. Furthermore, in the FLASK benchmark, MoA showed substantial gains in various metrics, including robustness, correctness, efficiency, and factuality. These results highlight the potential of MoA to set new standards in natural language understanding and generation tasks.
In conclusion, the Mixture-of-Agents approach is a promising development in the field of AI, showcasing the power of collaborative intelligence to push the boundaries of what LLMs can achieve.
Add a comment