AI Engineer · Published 2026-10-07

Grep or Embeddings? Agentic Search Over Company Documents — George He, LlamaIndex

privateReview: unreviewedConfidence: high
RAGvector-searchhybrid-searchllamaparseagentic-searchmulti-tenancyrerankingcontext-engineering
Open on YouTube ↗

Summary

Overview

  • Speaker: George He
  • Channel: AI Engineer
  • Main topic: Agentic search, document retrieval, hybrid search (embeddings + grep), and document parsing in enterprise environments
  • Purpose: To provide engineers with a practical framework and architectural blueprint for building scalable, high-performance agentic document search pipelines that combine keyword search, vector embeddings, and structured parsing. George He, Head of Engineering at LlamaIndex, discusses the trade-offs and design decisions between vector embeddings (semantic search) and grep-based local file search (keyword/file traversal) for enterprise AI agents. He argues that as models improve in instruction-following, developers face a debate on when and why agents need vector indexes. Through examining code-bases like Claude Code which favor grep, versus complex unstructured enterprise document repositories requiring hybrid approaches, he breaks down the 3-stage syncing process, multi-tenancy, permissions, and LlamaParse architecture.

Topic Map

The Debate: Grep vs. Embeddings

  • Explanation: Examining why coding agents like Claude Code dropped RAG in favor of simple local grep/glob search, and contrasting that with enterprise document sets containing PDFs, spreadsheets, and images.
  • Key claims:
    • Models are now good at instruction-following and local file traversal.
    • Code is small, structured, plain text, and already on disk, making grep effective.
    • Enterprise documents lack folder structures like code and have complex formats like tables and charts.
  • Examples:
    • Claude Code using glob, grep, and read instead of vector databases for codebase navigation.
  • Terminology:
    • RAG
    • Vector Index
    • Grep
    • Glob
    • Context Engineering
  • Why it matters: Determines whether agents should use file system traversal or pre-indexed vector stores depending on data scale and structure.

The Real Corpus & Enterprise Challenges

  • Explanation: Analyzing the realities of company data: scale, freshness, security, and format complexity across millions of files.
  • Key claims:
    • Enterprise data is massive, multi-format (PDFs, slides, spreadsheets), and changes continuously.
    • Security and access control are critical at enterprise scale.
  • Examples:
    • Contracts, filings, scanned PDFs, manufacturing diagrams.
  • Terminology:
    • Scale
    • Freshness
    • Security
    • Format
  • Why it matters: Shows why simple local file search fails for large, unstructured, multi-tenant company data.

The Costs of Context

  • Explanation: Evaluating the GPU memory, latency, and financial costs of stuffing entire corpora into large model context windows.
  • Key claims:
    • Every token in the context window costs GPU memory and compute on every query.
    • Model capabilities are growing (up to 1M+ tokens), but willingness to pay for full context ingestion decreases.
  • Examples:
    • 40 GB GPU memory for 128k context on NVIDIA hardware.
  • Terminology:
    • Context window
    • Token cost
    • Time to first token
  • Why it matters: Retrieval flips raw context ingestion by finding the handful of chunks that matter rather than sending the whole corpus every time.

Building a Harness: The Five Tools

  • Explanation: Presenting five core tools for agentic search: hybrid retrieve, list files, grep, read, and get page screenshot.
  • Key claims:
    • Agents need a harness that provides tools and lets the agent decide how to look.
    • Combining broad semantic search with precise file grep and multi-modal pre-rendering gives the best results.
  • Examples:
    • Using hybrid retrieve for broad compass direction, then grep/read to confirm exact wording.
  • Terminology:
    • Hybrid search
    • Filesystem tools
    • Re-ranking
    • Multimodal
  • Why it matters: Provides a robust architectural pattern for agentic document interaction.

Inside the Index: Syncing, Storage, and Permissions

  • Explanation: Breaking down the mechanics of index synchronization (ingest, parse, export), vector storage at scale, and access control tags.
  • Key claims:
    • Syncing is three independent jobs (ingest, parse, export) to keep data fresh without rebuilding from scratch.
    • Vector storage must be tenant-isolated with permissions tagged directly into vector embeddings.
  • Examples:
    • Using TurboPuffer for vector storage with filterable permissions metadata.
  • Terminology:
    • Incremental sync
    • Multi-tenancy
    • Access control
    • Vector storage
  • Why it matters: Addresses production challenges of data freshness, tenancy, and authorization.

Key Points

Hybrid Search Outperforms Single Methods

  • Explanation: Combining vector embeddings (semantic meaning) with keyword search (exact terms) and reranking via LLMs yields higher retrieval quality than either alone.
  • Evidence: Benchmark charts comparing keyword (40.6), vector (42), fusion (48.3), and rerank (48.5).
  • Practical implication: Enterprise search pipelines should default to hybrid fusion search with reranking.

Syncing as Three Decoupled Jobs

  • Explanation: To maintain freshness without full re-indexing, split synchronization into ingest (fingerprinting), parse (converting formats), and export (chunking and storing).
  • Evidence: Pipeline breakdown diagrams showing stage 01 (ingest), stage 02 (parse), and stage 03 (export).
  • Practical implication: Saves compute and prevents stale indexes in production.

Embed Permissions in Vector Storage

  • Explanation: Access control lists and metadata (e.g., allowed groups) should be tagged directly to vectors during ingestion rather than bolted on during retrieval.
  • Evidence: Turbopuffer metadata filtering example where chunk metadata includes allowed groups.
  • Practical implication: Prevents unauthorized data leaks and avoids breaking ranking with post-filtering.

Frameworks, Models & Processes

Agentic Search Tool Harness

  • How it works: Equips an AI agent with five primitive tools to discover, inspect, and verify documents dynamically.
  • Components:
    • hybrid_retrieve (Semantic + Keyword search across the index)
    • list_files (Filesystem traversal by metadata and directory)
    • grep (Exact pattern matching across files)
    • read (Reading specific file offsets and line ranges)
    • get_page_screenshot (Multimodal visual rendering for layouts, charts, and tables)
  • When to use: When building enterprise agents that need to navigate thousands of unstructured files accurately and cost-effectively.

Three-Stage Syncing Pipeline

  • How it works: Decouples the document processing pipeline into three independent, checkpointed stages.
  • Components:
    • Stage 01: Ingest (Fingerprint source files to detect changes)
    • Stage 02: Parse (Convert messy documents into clean text/markdown using LlamaParse)
    • Stage 03: Export (Chunk, embed, delete, and insert into the vector index)
  • When to use: For maintaining data freshness and scale in enterprise document systems.

Examples & Case Studies

Claude Code dropped vector-based RAG in favor of local glob, grep, and read commands.

  • Illustrates: Why codebases work well with grep due to small size, structure, and plaintext format.
  • Lesson: File search primitives work well when data is small and uniform, but enterprise data requires hybrid approaches.

Querying Alphabet 10-K financial reports for cash flow statements using hybrid search, grep, and file read tools.

  • Illustrates: An agent autonomously navigating metadata, searching broad financial statements, and verifying table chunks.
  • Lesson: Agents can successfully ground answers in multi-file enterprise corpora when provided with the right harness tools.

Actionable Takeaways

  • Immediate:
    • Combine keyword and vector search for hybrid retrieval.
    • Use structured parsing (like LlamaParse) for PDFs, tables, and charts before embedding.
  • Strategic:
    • Scale retrieval pipelines by separating ingest, parse, and export into decoupled jobs.
    • Tag permissions and tenancy metadata directly into vector storage during ingestion.
  • Questions to investigate:
    • How does context window pricing impact long-term agent architecture?
    • What is the optimal hybrid search balance between keyword and semantic weight for specific enterprise verticals?

Claims Worth Verifying

  • Hybrid search with reranking consistently outperforms pure vector or pure keyword search on enterprise benchmarks. (Technical Benchmark Claim)
  • Decoupling synchronization into ingest, parse, and export prevents full index rebuilds from scratch. (Architectural Best Practice)

Notable Quotes

"Everyone talks about search. What about the results?" "Search finds it. The harness makes it a result you can trust."

Compressed Summary

  • Hybrid search combines vector embeddings and keyword search for superior results.
  • Claude Code relies on grep because code is small, structured, and local.
  • Enterprise data requires structured parsing and agentic search harnesses.
  • Syncing should be split into ingest, parse, and export stages.
  • Permissions must be tagged directly into vector storage.
  • Keywords: rag, vector-search, hybrid-search, llamaparse, agentic-search
  • Core insight: Search finds information across an index, but an agentic harness with hybrid tools, structured parsing, and permissioned storage makes results trustworthy at enterprise scale.

Core insights

6
Mental Modelhigh noveltymoderate evidence

The grep-vs-embeddings decision is governed by corpus properties (size, structure, format homogeneity), not by model capability. Coding agents can drop RAG because code is small, structured, plain text, and already on disk, whereas enterprise documents lack folder structure and contain tables, charts, and scanned formats.

Why it matters

Frames the choice of retrieval substrate as a function of the data, preventing teams from copying the 'no-RAG' pattern from coding agents into document corpora where it fails.

Generalization

Retrieval architecture should be selected by measuring corpus structure and heterogeneity, not by current model strength.

Code is small, structured, plain text, and already on disk, making grep effective.
Open source video
Enterprise documents lack folder structures like code and have complex formats like tables and charts.
Open source video
Empirical Resultmedium noveltystrong evidence

Hybrid fusion search with LLM reranking measurably beats either keyword or vector search alone (keyword 40.6, vector 42, fusion 48.3, rerank 48.5), indicating the quality gains come from combining exact-term and semantic signals rather than from either retriever in isolation.

Why it matters

Provides concrete numbers that justify defaulting enterprise pipelines to hybrid fusion + rerank instead of single-method retrieval.

Generalization

Retrieval quality is a composition problem: fusion and reranking stages add gains that no single retriever achieves alone.

Benchmark charts comparing keyword (40.6), vector (42), fusion (48.3), and rerank (48.5).
Open source video
Mechanismmedium noveltymoderate evidence

Index synchronization should be decomposed into three independent, checkpointed jobs — ingest (fingerprinting to detect changes), parse (format conversion), and export (chunk, embed, delete, insert) — so freshness is maintained without rebuilding the whole index.

Why it matters

Enables incremental freshness and fault isolation, avoiding full re-index cost and stale indexes in production.

Generalization

Treat ingestion pipelines as decoupled checkpointed stages so any stage can be re-run without cascading reprocessing.

Syncing is three independent jobs (ingest, parse, export) to keep data fresh without rebuilding from scratch.
Open source video
Architecturemedium noveltymoderate evidence

Access control should be baked into vector records at ingestion time by tagging allowed groups into chunk metadata, rather than applied as a post-retrieval filter.

Why it matters

Prevents unauthorized data leaks and avoids post-filtering that breaks ranking by removing results after top-k selection.

Generalization

Security constraints belong at the storage/ingestion layer so filtering is native to retrieval rather than bolted on afterward.

Access control lists and metadata (e.g., allowed groups) should be tagged directly to vectors during ingestion rather than bolted on during retrieval.
Open source video
Vector storage must be tenant-isolated with permissions tagged directly into vector embeddings.
Open source video
Architecturehigh noveltymoderate evidence

Agentic document search should be implemented as a tool harness — hybrid_retrieve, list_files, grep, read, get_page_screenshot — that lets the agent choose its own retrieval strategy, combining broad semantic search with precise grep/read and multimodal page rendering.

Why it matters

Shifts responsibility for choosing retrieval strategy from a fixed pipeline to the agent, mixing coarse semantic pointers with exact verification.

Generalization

Expose retrieval as a set of composable primitives rather than a single endpoint, so the agent can escalate from broad to precise as needed.

Agents need a harness that provides tools and lets the agent decide how to look.
Open source video
Combining broad semantic search with precise file grep and multi-modal pre-rendering gives the best results.
Open source video
Tradeoffmedium noveltymoderate evidence

Stuffing corpora into context windows is economically constrained: every token costs GPU memory and compute on every query (e.g., ~40 GB GPU memory for 128k context on NVIDIA hardware), so retrieval acts as a cost-flip by selecting only the few relevant chunks.

Why it matters

Gives a concrete hardware-cost anchor for why retrieval persists even as context windows grow to 1M+ tokens.

Generalization

Retrieval is fundamentally a cost-optimization over repeated context ingestion, not just a quality technique.

Every token in the context window costs GPU memory and compute on every query.
Open source video
40 GB GPU memory for 128k context on NVIDIA hardware.
Open source video
Retrieval flips raw context ingestion by finding the handful of chunks that matter rather than sending the whole corpus every time.
Open source video

Deep dives

4

The grep-to-embeddings crossover point for enterprise corpora

Research question

At what combination of corpus size, file-format heterogeneity, and structural depth does grep/glob/file-traversal retrieval stop being viable and a vector or hybrid index become necessary?

Why

Pass-1 frames the retrieval-substrate choice as a function of corpus properties but supplies no threshold, so teams cannot decide when to copy the 'no-RAG' coding-agent pattern and when it will silently fail on document sets.

Code is small, structured, plain text, and already on disk, making grep effective.
Open source video
Enterprise documents lack folder structures like code and have complex formats like tables and charts.
Open source video
Source video

Harness-level evaluation of agent-selected retrieval

Research question

Does letting an agent choose among hybrid_retrieve, list_files, grep, read, and get_page_screenshot beat a fixed hybrid-retrieve pipeline on task success, tool-call efficiency, and per-task cost?

Why

Reported numbers cover keyword/vector/fusion/rerank but not end-to-end harness performance, so the central architectural claim of the talk (shifting strategy choice to the agent) is currently unevaluated.

Agents need a harness that provides tools and lets the agent decide how to look.
Open source video
Combining broad semantic search with precise file grep and multi-modal pre-rendering gives the best results.
Open source video
Source video

Scaling ingestion-time permission tagging under ACL churn

Research question

Does tagging allowed-group metadata into vector records at ingestion time hold up under high-cardinality, rapidly changing access control lists and many tenants, or does it force costly re-embedding/upsert churn?

Why

Pass-1 recommends moving access control from post-filter to ingestion-time tagging but leaves update frequency and ACL cardinality unaddressed, which is exactly where the recommendation could break in production.

Access control lists and metadata (e.g., allowed groups) should be tagged directly to vectors during ingestion rather than bolted on during retrieval.
Open source video
Vector storage must be tenant-isolated with permissions tagged directly into vector embeddings.
Open source video
Source video

The per-query cost curve of context-window ingestion vs retrieval

Research question

How does the marginal GPU memory and time-to-first-token cost of full-corpus context ingestion scale with corpus size and query volume compared with a chunk-selective retrieval path?

Why

The retrieval-vs-context debate is often argued on quality alone; turning it into a cost curve gives an engineering decision rule as context windows grow past 1M tokens.

Every token in the context window costs GPU memory and compute on every query.
Open source video
40 GB GPU memory for 128k context on NVIDIA hardware.
Open source video
Retrieval flips raw context ingestion by finding the handful of chunks that matter rather than sending the whole corpus every time.
Open source video
Source video

Article ideas

4

Your Retrieval Architecture Is a Property of Your Corpus, Not Your Model

The grep-vs-embeddings argument is misframed as a model-capability debate; the correct decision variable is corpus structure, size, and format homogeneity, which is why coding-agent patterns do not transfer to enterprise documents.

Angle

Contrast the coding-agent corpus (small, structured, on-disk plain text) with enterprise corpora (scanned PDFs, slides, tables, charts, no folder structure) and derive a decision procedure from corpus measurements rather than model benchmarks.

Source video

Your Agent Doesn't Need a Better Retriever — It Needs a Better Harness

The retrieval-quality ceiling of a single endpoint is lower than a primitive-exposing harness where the agent escalates from broad semantic retrieval to precise grep/read and multimodal page rendering.

Angle

Use the reported fusion/rerank gains as the baseline a harness must beat, then argue that tool schemas, escalation policy, and stopping criteria become the primary engineering surface.

Source video

Retrieval Is a Cost Technology Wearing a Quality Costume

Even as context windows reach 1M+ tokens, retrieval persists because repeated context ingestion carries per-query GPU memory and compute cost, making retrieval fundamentally a cost optimization over tokens rather than a relevance technique.

Angle

Anchor the argument with the ~40 GB for 128k context figure and analyze willingness-to-pay decay as token load grows; reframe RAG roadmaps as capacity plans.

Source video

Freshness Without Full Rebuilds: Decoupling Ingest, Parse, and Export

Splitting index synchronization into three independently checkpointed jobs removes the false choice between stale indexes and full-corpus re-indexing cost.

Angle

Walk through fingerprint-based ingest change detection, parse as a format-conversion boundary, and export as the chunk/embed/delete/insert step, focusing on inter-stage state and fault isolation.

Source video

Project ideas

4

Grep/Embedding Corpus Crossover Harness

beyond-evals

For a fixed task set, grep-based traversal will match or beat embedding retrieval below a measurable corpus-size and format-heterogeneity threshold and degrade sharply above it, producing a reproducible crossover surface.

Proof of concept

Sweep synthetic and real document corpora across size, format mix (plain text vs PDF vs spreadsheet vs scanned image), and folder-structure depth; run identical agent tasks against grep-only, vector-only, and fusion pipelines.

Measurement

Task success rate and retrieval precision per corpus configuration, with the crossover point reported as a function of corpus size, format entropy, and structural depth.

Source video

Tool-Harness vs Fixed-Pipeline Benchmark

beyond-evals

An agent given hybrid_retrieve, list_files, grep, read, and get_page_screenshot will exceed a fixed hybrid-retrieve plus rerank pipeline on task success at equal or lower token cost.

Proof of concept

Implement both retrieval regimes over the same corpus and task suite; log every tool invocation, token spend, and terminal answer for paired comparison.

Measurement

Task success rate, tool-call count per solved task, tokens consumed, and latency, compared against the reported fusion (48.3) and rerank (48.5) baselines.

Source video

ACL Churn Stress Test for Vector Metadata Filtering

gatehouse

Ingestion-time permission tagging preserves ranking quality and leak safety up to a bounded ACL update rate, beyond which re-embedding/upsert churn makes post-filtering comparatively cheaper.

Proof of concept

Build a tenant-isolated vector store with allowed-group metadata on chunks; drive it with scripted ACL change streams at varying frequency and cardinality, comparing ingestion-time tagging against post-retrieval filtering.

Measurement

Ranking quality delta (tagged vs post-filtered), leaked-record count, ingestion/upsert cost per ACL change, and query latency under each ACL churn regime.

Source video

Multimodal Page Rendering ROI Study

new

For documents dominated by tables and charts, get_page_screenshot improves answer correctness over text-only parsing by a margin that justifies its added latency and cost, and the margin vanishes on text-dominant documents.

Proof of concept

Split a mixed corpus by layout type; run the same agent tasks with and without the page-screenshot tool and record correctness alongside per-call latency and token cost.

Measurement

Answer correctness delta on table/chart documents vs text documents, plus added latency and cost per solved task attributable to the screenshot tool.

Source video

Architectural implications

3

Retrieval strategy selection moves from a static pipeline to an agent-controlled tool choice.

Before

A fixed RAG pipeline pre-retrieves top-k chunks before the model sees anything.

After

The agent has hybrid_retrieve, list_files, grep, read, and get_page_screenshot and decides which to invoke.

Consequence

Harness design, tool schemas, and stopping criteria become central engineering artifacts rather than the retriever itself.

Source video

Security filtering migrates from retrieval-time to ingestion-time.

Before

Access control applied as a post-filter on retrieved results.

After

Allowed-group metadata tagged into vectors and enforced via native metadata filtering.

Consequence

Ranking integrity is preserved and leak risk reduced, but ingestion must carry permission context end-to-end.

Source video

Synchronization is decomposed into independent checkpointed stages.

Before

Monolithic re-index that reprocesses the entire corpus on change.

After

Ingest (fingerprint), parse, and export as separate jobs with their own checkpoints.

Consequence

Lower compute overhead and fresher indexes, at the cost of managing inter-stage state and coordination.

Source video

Tradeoffs and failure modes

3

Grep/file traversal instead of vector indexes

Benefit

Simple, exact, index-free search that works well for small structured corpora like code.

Cost or risk

Fails on large, unstructured, multi-format enterprise document sets lacking folder structure.

Enterprise documents lack folder structures like code and have complex formats like tables and charts.
Open source video
Source video

Full-context ingestion

Benefit

Avoids retrieval complexity by placing entire corpora in context.

Cost or risk

High GPU memory and compute cost per query; willingness to pay decreases as token load grows.

Every token in the context window costs GPU memory and compute on every query.
Open source video
Source video

Post-retrieval permission filtering

Benefit

Simple to bolt access control onto an existing retrieval flow.

Cost or risk

Post-filtering can break ranking and risks unauthorized data exposure, motivating ingestion-time tagging.

Access control lists and metadata (e.g., allowed groups) should be tagged directly to vectors during ingestion rather than bolted on during retrieval.
Open source video
Source video

Open questions

4

At what corpus size, structural homogeneity, and format complexity does grep-based traversal stop being viable and a vector index become necessary?

Why unresolved

The summary describes the qualitative trade-off but provides no threshold or measurement for when the switch occurs.

Research direction

Build a benchmark sweeping corpus size, file formats, and structure to find the crossover point between grep and embedding retrieval.

Source video

How should the agentic tool harness itself be evaluated, as opposed to individual retrievers?

Why unresolved

Reported benchmark numbers cover keyword/vector/fusion/rerank but not end-to-end harness performance.

Research direction

Define harness-level evaluations measuring task success, tool-call efficiency, and cost for the five-tool agent.

Source video

Does embedding permission metadata into vectors scale with high-cardinality or rapidly changing access control lists?

Why unresolved

The summary recommends ingestion-time tagging but does not address update frequency or ACL cardinality tradeoffs.

Research direction

Test metadata-filtered vector stores under frequent ACL changes and high tenant counts.

Source video

What does the get_page_screenshot multimodal tool add over text extraction, and when is its latency/cost justified?

Why unresolved

The summary lists it as a primitive for layouts, charts, and tables but gives no measured benefit or cost.

Research direction

Compare multimodal page rendering versus text-only parsing on documents with tables and charts.

Source video

Key claims

6
comparativeVerification needed

Hybrid fusion search with reranking outperforms keyword-only and vector-only retrieval.

Evidence

Benchmark charts comparing keyword (40.6), vector (42), fusion (48.3), and rerank (48.5).

Question

What benchmark, dataset, and metric produced these specific scores, and are they reproducible?

Source video
causalVerification needed

Code corpora are small, structured, and plain text, which is why grep suffices for coding agents.

Evidence

Code is small, structured, plain text, and already on disk, making grep effective.

Question

At what codebase scale does grep-based retrieval begin to degrade relative to embeddings?

Source video
opinionVerification not requested

Index synchronization should be split into three independent jobs: ingest, parse, and export.

Evidence

Syncing is three independent jobs (ingest, parse, export) to keep data fresh without rebuilding from scratch.

Source video
causalVerification needed

Permissions should be embedded into vector metadata at ingestion rather than applied post-retrieval.

Evidence

Access control lists and metadata (e.g., allowed groups) should be tagged directly to vectors during ingestion rather than bolted on during retrieval.

Question

Does ingestion-time permission tagging measurably improve ranking quality and reduce leak risk versus post-filtering?

Source video
factualVerification needed

Context-window ingestion incurs substantial GPU cost, illustrated by ~40 GB for 128k context.

Evidence

40 GB GPU memory for 128k context on NVIDIA hardware.

Question

Which NVIDIA hardware and configuration yields 40 GB for 128k context, and how does it scale?

Source video
causalVerification needed

A five-primitive tool harness lets agents choose how to look, outperforming a single fixed retrieval method.

Evidence

Agents need a harness that provides tools and lets the agent decide how to look.

Question

Does agent-selected tool use beat a fixed hybrid-retrieve pipeline on task success and cost?

Source video

Connections

4