Researchers unveil MindSearch: A SOTA open-source answer engine!
A new open source tool for searching the web with AI!
Created on August 5|Last edited on August 5
Comment
Researchers from the University of Science and Technology of China and the Shanghai AI Laboratory have introduced a groundbreaking AI framework called MindSearch, designed to tackle the complexities of web information retrieval and integration. MindSearch leverages a multi-agent system to mimic human cognitive processes, combining the strengths of search engines and Large Language Models (LLMs) to deliver accurate and comprehensive responses to user queries.
Information Seeking Challenges
Information seeking on the web is a cognitively demanding task that involves sifting through vast amounts of data to find relevant information. Traditional search engines, while powerful, often struggle to align with complex human intentions, leading to inefficiencies and incomplete results. Recent attempts to combine LLMs with search engines have shown promise but still face significant challenges. These include the inability of search engines to retrieve all necessary information in a single query, the dispersion of relevant information across multiple web pages mixed with irrelevant content, and the limited context length of LLMs, which makes it difficult to integrate information from many long documents.
The MindSearch Framework
MindSearch addresses these challenges through a multi-agent framework composed of two main components: WebPlanner and WebSearcher. The WebPlanner acts as the high-level orchestrator of the information retrieval process. It decomposes the user’s query into smaller, manageable sub-queries, modeling this decomposition as a dynamic graph. This graph represents the query and its sub-components as nodes, enabling a structured approach to information seeking. The WebPlanner uses a directed acyclic graph (DAG) to organize and process sub-queries, ensuring a logical flow of information retrieval steps.
The WebSearcher executes the sub-queries generated by the WebPlanner. It performs hierarchical information retrieval using a coarse-to-fine strategy to manage large volumes of web content. The coarse level broadly generates multiple related queries to ensure wide coverage of the topic and executes these queries via search engines to gather a comprehensive set of initial results. The intermediate level aggregates and filters these results, removing duplicates and irrelevant entries, and selects the most valuable pages based on relevance. The fine level conducts detailed reading and summarization of selected pages, extracting key insights and integrating them into a coherent summary.
Coarse-to-Fine Strategy
The coarse-to-fine strategy is crucial for efficient and effective information retrieval. It starts with generating diverse queries related to the user’s input, ensuring a wide range of information is covered. It then aggregates search results, merges duplicates, and selects the most promising sources for detailed analysis. Finally, it reads and summarizes the full content of selected pages, providing a focused and relevant response.
Long-Context Management
Managing long contexts is a significant challenge in information retrieval. MindSearch’s multi-agent framework naturally facilitates this. The WebPlanner focuses on query decomposition, while the WebSearcher handles the detailed searches. This separation allows each agent to specialize in its task, reducing computational load and improving efficiency. The DAG structure helps maintain context across agents. WebSearcher’s responses include contextual information from previous nodes, ensuring continuity and relevance.
MindSearch’s ability to integrate information from over 300 web pages in under three minutes showcases its efficiency. This performance is significantly faster than what human experts could achieve, demonstrating the practical benefits of the framework.
Experimental Results
MindSearch was evaluated using two primary categories of Question Answering (QA) tasks: open-set QA and closed-set QA. The evaluations aimed to reflect both subjective and objective judgments of MindSearch’s performance, using only the BING search API without extra reference sources.
In the open-set QA evaluation, 100 real-world human queries were curated, and responses were collected from MindSearch (using InternLM2.5-7b-chat), Perplexity.ai (Pro version), and ChatGPT with a search plugin. Five human experts evaluated the responses based on depth, breadth, and factuality. The results showed that MindSearch significantly improved in terms of depth and breadth of responses. This validates the superiority of the WebPlanner, which effectively decomposes complex queries into executable sub-queries. However, MindSearch did not perform as well in factuality, suggesting that more detailed search results might distract the model from the initial problem, especially given the incomplete long-context capabilities of LLMs. Future work will focus on alleviating hallucination issues during the web browsing process.
In closed-set QA tasks, including datasets like Bamboogle, Musique, and HotpotQA, MindSearch was compared against raw LLMs without search engines and a ReAct-style interaction with search engines. The results showed that MindSearch outperformed these baselines by a large margin, further proving the effectiveness of the proposed method. These advantages were amplified when transferring from closed-source LLMs to open-source LLMs, indicating that MindSearch provides a simple approach to enhance weak LLMs with broader knowledge and reduce hallucination issues.
Conclusion
MindSearch represents a significant advancement in AI-driven web information retrieval. By mimicking human cognitive processes through a multi-agent system and employing a coarse-to-fine retrieval strategy, MindSearch addresses the limitations of traditional search engines and LLMs. This innovative approach not only enhances the accuracy and comprehensiveness of responses but also significantly improves the efficiency of handling complex queries.
The framework’s open-source nature, available here, invites further development and optimization by the research community. MindSearch sets a new standard for AI-driven search solutions, paving the way for more advanced and efficient information retrieval systems.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.