Lexical Search
Authoritative Definition
Traditional keyword-based text search that matches exact words, substrings, or tokenized stems using inverted index structures.
Overview & Technical Description
Lexical search relies on the exact occurrences of tokens within a document corpus. By utilizing techniques such as tokenization, stemming, and stop-word removal, it builds highly efficient inverted indices that map terms to document locations. It employs term-weighting algorithms like TF-IDF or BM25 to score and rank documents based on keyword frequency and rarity. In AI infrastructure and RAG architectures, lexical search remains a foundational component due to its unparalleled ability to handle out-of-vocabulary terms, specific identifiers, and exact match queries. While dense vector search struggles with short, highly specific strings (such as product SKUs, error codes, or specific names), lexical search excels at retrieving these exact matches instantly. When integrated into modern retrieval pipelines, lexical search is typically deployed alongside semantic vector search in a hybrid configuration. This dual approach ensures that systems benefit from both the exactitude of keyword matching and the conceptual understanding of dense embeddings.
Editorial Notes
Essential for exact entity, phone number, and code symbol searches. Lexical search systems often require extensive tuning of analyzers, tokenizers, and synonym lists to optimize retrieval performance for domain-specific jargon.