BM25

BM25 (Best Matching 25) remains the industry-standard algorithm for sparse, lexical search engines. It calculates a relevance score between a search query and a document by evaluating how often query terms appear in the document (Term Frequency - TF) while penalizing terms that are common across the entire corpus (Inverse Document Frequency - IDF). BM25 improves upon older TF-IDF models by introducing a non-linear term frequency saturation curve and document length normalization, preventing excessively long documents from being unfairly penalized or rewarded. Architecturally, BM25 operates on inverted indexes, where text is tokenized, stemmed, and mapped to the documents containing those tokens. When a query is executed, the engine quickly looks up the matching tokens and computes the BM25 score. It is the default scoring algorithm under the hood of widely-used search infrastructure like Elasticsearch, OpenSearch, and Apache Lucene. In the era of LLMs and RAG, BM25 remains highly important because dense vector embeddings often struggle with exact keyword matching, acronyms, IDs, or domain-specific jargon. By combining BM25's lexical precision with vector search's semantic understanding in a Hybrid Search architecture, RAG systems achieve much higher overall recall and relevance, ensuring that specific entities and semantic concepts are both accurately retrieved.