The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

Overview

• Speaker: Jo Kristian Bergum
• Channel: AI Engineer
• Main topic: The resurgence and effectiveness of BM25 lexical scoring for agentic information retrieval loops
• Purpose: To explain why lexical search (BM25) combined with simple tools like grep outperforms complex vector/embedding retrieval setups in agentic search workflows. Jo Kristian Bergum, CEO of Hornet.dev, explains why a 30-year-old lexical scoring function like BM25 is experiencing a strong comeback in the era of LLMs and agentic search. He defines agentic search as search running inside an agent loop consisting of an LLM model, a harness, and a retrieval engine. Modern LLM users are more powerful, issuing complex queries, using syntactic operators, and rewriting queries dynamically. BM25 provides exact matching, low cost, and interpretability, making it an ideal foundational primitive when paired with simple tools like grep.

Topic Map

Introduction and Agentic Search Definition

• Explanation: Introduction of the speaker, company background, and definition of agentic search as search inside an agent loop.
• Key claims:
  • Agentic search is search inside an agent loop.
  • An agentic search system requires a capable model, a harness, and a retrieval engine.
• Examples:
  • Writing code, answering questions, deep research tasks.
• Terminology:
  • Agentic search
  • Harness
  • Retrieval engine
  • LLM
• Why it matters: Establishes the foundational architecture of AI agents requiring external retrieval tools.

What is BM25

• Explanation: Definition of BM25 (Best Match 25) as a 30-year-old lexical scoring function based on term frequency, inverse document frequency, and document length normalization.
• Key claims:
  • BM25 is a scoring function evaluating term rarity, term frequency, and document length normalization.
  • Top-k retrieval algorithms accelerate finding the highest scoring documents.
• Examples:
  • Scoring query-document pairs using inverted indexes and algorithms like WAND.
• Terminology:
  • BM25
  • Best Match 25
  • Scoring function
  • Top-k retrieval
  • Inverted index
  • WAND
• Why it matters: Clarifies the mechanics of the baseline retrieval function discussed throughout the talk.

User Change: Why BM25 Makes a Comeback

• Explanation: BM25 hasn't changed, but the user has. LLMs possess general knowledge, detailed queries, and relevance feedback across turns.
• Key claims:
  • BM25 did not become magic; the user changed.
  • LLMs are powerful users capable of typing complex, multi-term queries and syntax operators.
• Examples:
  • LLMs using site operators, phrasing, and entity filters.
• Terminology:
  • General knowledge in model
  • Query formulation
  • Relevance feedback
• Why it matters: Explains the paradigm shift from single-shot human search to multi-turn agentic query generation.

BrowseComp-Plus Benchmark and Context Windows

• Explanation: Analysis of the BrowseComp-Plus benchmark containing riddle-like multi-hop questions over web documents.
• Key claims:
  • Retrieval quality changes end-to-end task accuracy drastically.
  • Context window limitations make retrieval essential even with massive LLM context windows.
• Examples:
  • Comparing 736M token corpus against floppy disk sized LLM context windows (~370k tokens).
• Terminology:
  • BrowseComp-Plus
  • Multi-hop question
  • Oracle evidence
  • Context window
  • Floppy disk
• Why it matters: Demonstrates empirically that retrieval quality is the core bottleneck for agent accuracy.

BM25 + Grep Is All You Need

• Explanation: Discussion of Waterloo's research paper on scaling direct corpus interaction via dynamic workspace expansion.
• Key claims:
  • Combining BM25 retrieval with file system workspaces and grep tools achieves superior end-to-end accuracy.
  • Exposing retrieved documents in virtual file systems (vfs) enables progressive disclosure and inspectable reasoning.
• Examples:
  • Using rg/grep, cat, read lines, inspect, verify tools in agent loops.
• Terminology:
  • Dynamic workspace expansion
  • Progressive disclosure
  • vfs
  • grep
• Why it matters: Provides a blueprint for building high-performance, cost-effective agentic search infrastructure.

Key Points

BM25 as an interpretable primitive

• Explanation: BM25 scores based on exact term matches, making it fully explainable and easy for LLMs to reason about.
• Evidence: Outperforms complex embedding models when paired with sophisticated agent query generation.
• Practical implication: Engineering teams should use BM25 as a robust baseline before adopting heavy embedding or neural retrieval models.

Context window vs corpus size

• Explanation: Even large LLM context windows are tiny fractions of real-world enterprise corpora (billions of documents).
• Evidence: Corpora like BrowseComp-Plus have 736M tokens while model context windows hold ~370k tokens.
• Practical implication: Retrieval infrastructure remains indispensable regardless of how large LLM context windows grow.

Evaluating agent + retriever as a system

• Explanation: Traditional IR evaluation metrics based on static single queries are obsolete for agents.
• Evidence: Agents generate query trajectories, rewriting queries dynamically over multiple turns.
• Practical implication: Evaluation must measure end-to-end task success, latency, and tool call efficiency rather than static NDCG@10.

Frameworks, Models & Processes

Agentic Search Loop

• How it works: An LLM agent determines when to search and formulates queries, interacts with a retrieval engine to get snippets, and refines results in a loop until it finds enough evidence for a final answer.
• Components:
  • Capable model (LLM)
  • Harness
  • Retrieval engine
• When to use: When building complex QA, coding assistants, or deep research agents over large corpora.

BM25 + Grep Workspace Architecture

• How it works: Retrieves candidate documents using BM25, places them in a virtual file system workspace, and uses Unix-like tools (grep, cat) for progressive disclosure and inspection.
• Components:
  • BM25 retriever
  • Virtual file system (vfs)
  • Grep / read tools
• When to use: For scaling direct corpus interaction in developer tools and enterprise RAG systems.

Examples & Case Studies

Evaluation of BrowseComp-Plus benchmark with oracle evidence versus retriever pipelines.

• Illustrates: When the exact evidence document is provided upfront in the context window, GPT-4 reaches 93% accuracy.
• Lesson: Reasoning is not the bottleneck; retrieval quality determines overall task success.

Comparing human search queries (median 2 terms) with GPT-5 search trajectories (median 10 terms).

• Illustrates: LLMs act as power users leveraging advanced query syntax, operators, and specific keywords.
• Lesson: Search engines optimized for human users must adapt to high-frequency, complex LLM-generated queries.

Actionable Takeaways

• Immediate:
  • Adopt BM25 as a strong, cheap, and interpretable retrieval baseline.
  • Implement file-system-like workspaces (vfs) for agent document inspection.
• Strategic:
  • Evaluate retrieval and LLM agents as a single integrated system rather than isolated components.
  • Leverage simple Unix tools (grep, cat) alongside lexical search to match modern agent workflows.
• Questions to investigate:
  • How do BM25 hyperparameters (k1, b) impact long document retrieval in agentic workflows?
  • What is the optimal balance between vector search and lexical BM25 in multi-modal agent systems?

Claims Worth Verifying

• Oracle evidence in BrowseComp-Plus reaches 93% accuracy on multi-hop questions. (Benchmark statistic)
• BM25 with grep and virtual file systems outperforms standard vector RAG pipelines on specific agent benchmarks. (Research finding)

Notable Quotes

"The unreasonable effectiveness of BM25 for agentic search." (at 0:19) "BM25 did not become magic. The user changed." (at 5:32) "Retrieval quality changes answer accuracy." (at 10:48) "BM25 plus grep is all you need." (at 20:05)

Compressed Summary

• BM25 is a 30-year-old lexical scoring function experiencing a major comeback in agentic search.
• LLMs act as power users issuing complex, multi-term, syntax-heavy queries compared to human users.
• Retrieval quality is the primary bottleneck in agentic accuracy, not LLM reasoning capability.
• Combining BM25 with grep and virtual file systems (vfs) provides a powerful, interpretable, and low-cost retrieval architecture.
• Keywords: bm25, agentic search, retrieval, grep, llm
• Core insight: BM25 makes a powerful comeback in agentic search because modern LLM agents act as power users capable of exploiting lexical scoring, exact matching, and simple grep tools for superior reasoning accuracy.