Qdrant Unveils BM42: A Cutting-Edge Pure Vector-Based Hybrid Search Algorithm Optimizing RAG and AI Applications

Qdrant, a leading provider of vector search technology, has introduced BM42, a new algorithm designed to revolutionize hybrid search. For the past four decades, BM25 has been the standard algorithm used by search engines, from Google to Yahoo. However, the advent of vector search and the introduction of Retrieval-Augmented Generation (RAG) have highlighted the need for a more advanced solution. BM42 aims to bridge this gap by combining the strengths of BM25 with modern transformer models, offering a significant upgrade for search applications.

The Legacy of BM25

BM25 has remained relevant for a long time due to its simple yet effective formula, which calculates the relevance of documents based on term frequency and inverse document frequency (IDF). This method excels in traditional web search environments where document length and query structures are consistent. However, the landscape of text retrieval has shifted dramatically with the rise of RAG systems, which require handling shorter, more varied documents and queries. BM25’s reliance on document statistics, such as term frequency and document length, becomes less effective in these scenarios.

The Introduction of BM42

BM42 addresses these challenges by integrating the core principles of BM25 with the capabilities of transformer models. The key innovation in BM42 is using attention matrices from transformers to determine the importance of the term within documents. Transformers generate a range of outputs, including embeddings and attention matrices, highlighting each token’s significance in the input sequence. By leveraging the attention row corresponding to the special [CLS] token, BM42 can accurately gauge the importance of each token in a document, even for shorter texts typical in RAG applications.

Advantages of BM42

BM42 offers several advantages over BM25 and SPLADE, another modern alternative that uses transformers to create sparse embeddings. While SPLADE has shown superior performance in academic benchmarks, it needs to improve its performance, including the need for extensive computational resources and issues with tokenization and domain dependency. BM42, on the other hand, retains the interpretability and simplicity of BM25 while overcoming SPLADE’s limitations.

One of BM42’s primary benefits is its efficiency. The algorithm can perform document and query inferences quickly, making it suitable for real-time applications. It also has a low memory footprint, ensuring it can handle large datasets without significant resource demands. BM42 supports multiple languages and domains, provided a suitable transformer model is available, making it highly versatile.

Practical Implementation

BM42 can be seamlessly integrated into Qdrant’s vector search engine. The implementation involves setting up a collection for hybrid search with BM42 and using dense embeddings from models like jina.ai. This combination allows for a balanced approach, where sparse and dense embeddings complement each other to enhance retrieval accuracy. Benchmarks conducted by Qdrant demonstrate that BM42 outperforms BM25 in scenarios involving short texts, a common use case in modern search applications.

Encouraging Community Engagement

Qdrant’s release of BM42 introduces a new algorithm and fosters community engagement and innovation. The company invites developers and researchers to experiment with BM42, share their projects, and contribute to its ongoing development. By providing this powerful tool, Qdrant aims to empower its community to push the boundaries of what is possible in search technology.

Conclusion

The release of BM42 by Qdrant marks a significant milestone in the evolution of search algorithms. By combining the robustness of BM25 with the intelligence of transformers, BM42 sets a new standard for hybrid search. It addresses the limitations of earlier methods and modern alternatives, offering a versatile, efficient, and highly accurate solution for today’s search applications.