AI Engineer · Published 2026-09-23

From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto

privateReview: unreviewedConfidence: high
document intelligenceagentic OCRvision-language modelsRAGagent harnessesdata ingestiondocument-intelligenceagentic-ocrvision-language-modelsdata-ingestionpipeline-architecturestage-level-evaluationspeculative-decoding
Open on YouTube ↗

Summary

Overview

  • Speaker: Adit Abraham
  • Channel: AI Engineer
  • Main topic: Better agents through better data: how AI teams ingest unstructured data at scale.
  • Purpose: Provide AI engineers with actionable insights, frameworks, and ingestion pipeline recommendations for building reliable document intelligence and agentic workflows. Adit Abraham, co-founder and CEO of Reducto, discusses the practical challenges and strategies of building AI agents that process unstructured documents like PDFs, spreadsheets, and images at scale. He highlights the shift in AI applications from simple information synthesis (RAG) to autonomous execution (agents) and emphasizes that bad data inputs lead to compounding bad outputs in multi-step agent pipelines. The talk covers why PDFs are difficult, the tradeoff between traditional computer vision and vision-language models (VLMs), agentic OCR (speculative decoding and agent-in-the-loop verification), formatting strategies for LLMs versus embedding models, and agent harnesses for complex data tasks.

Topic Map

The Shift from Information Synthesis to Autonomous Action

  • Explanation: AI applications have shifted from retrieval-augmented generation (RAG) and question-answering search tools to autonomous agents executing multi-step end-to-end tasks like reading, writing, and delivering complete workflows.
  • Key claims:
    • Initial AI products were information synthesis tools (search, chatbots, summarization).
    • Today's teams build agents to execute real human work end-to-end.
    • Agentic workflows introduce compounding risks when fed bad data inputs.
  • Examples:
    • Search interfaces vs. agents executing end-to-end tasks in finance, insurance, and legal sectors.
  • Terminology:
    • Information synthesis
    • Autonomous action
    • RAG
    • Agent pipelines
  • Why it matters: When agents make sequential decisions across multiple files, data errors compound and lead to catastrophic pipeline failures.

The Enterprise Data Bottleneck

  • Explanation: Enterprise data is predominantly unstructured, scattered across various repositories (Google Drive, Box, local drives), and multimodal.
  • Key claims:
    • Most enterprise data lives in unstructured formats like PDFs, spreadsheets, and scanned images.
    • Data is scattered across organizational silos with no uniform schema.
    • Downstream problems include parsing accuracy, context retrieval, and editing.
  • Examples:
    • Global financial institutions and insurance organizations with decades of historical unstructured records.
  • Terminology:
    • Unstructured data
    • Multimodal
    • Parsing extraction accuracy
    • Context retrieval
  • Why it matters: Language model capabilities only apply within the context to which intelligence is applied; poor ingestion bottlenecks agent performance.

Why PDFs Are Still Hard

  • Explanation: PDFs are fundamentally an image format designed in the 1990s for printing fidelity, making semantic extraction exceptionally challenging.
  • Key claims:
    • Current frontier models score around 30% on rigorous benchmarks like GDP-PDF.
    • PDFs include handwritten text, large tables, poor scans, multi-column layouts, and embedded charts.
    • Traditional OCR and post-processing fail on complex or non-standard document layouts.
  • Examples:
    • W-2 forms, financial tables with merged cells, line charts, and messy handwritten prescriptions.
  • Terminology:
    • GDP-PDF benchmark
    • OCR
    • Layout analysis
    • Multimodal
  • Why it matters: Enterprise workflows rely heavily on PDFs, yet foundation models struggle to accurately read and reason over them out of the box.

Traditional Computer Vision vs. Vision-Language Models (VLMs)

  • Explanation: Balancing traditional sub-100M parameter computer vision models for structure detection with VLMs for semantic reasoning.
  • Key claims:
    • Traditional CV models (sub-100M parameters) running on CPU are highly efficient at layout segmentation, object detection, and reading order.
    • VLMs introduce horizontal semantic understanding and handle long-tail elements like handwriting.
    • A hybrid approach combining traditional CV and VLMs yields superior accuracy.
  • Examples:
    • Using classical object detection for table and bounding box extraction combined with VLM reasoning for semantics.
  • Terminology:
    • Computer Vision (CV)
    • Vision-Language Models (VLMs)
    • Object detection
    • Semantics
  • Why it matters: Neither approach alone solves all document processing challenges at enterprise scale without performance or latency trade-offs.

Agentic OCR and Agent-in-the-Loop Frameworks

  • Explanation: Moving from human-in-the-loop to agent-in-the-loop using speculative decoding and token-level edits to correct OCR errors automatically.
  • Key claims:
    • Agentic OCR replaces manual review teams with verification and correction agents.
    • Speculative decoding applies token-level edits to correct OCR hallucinations or extraction errors.
    • Agents can cross-verify table totals and correct digit misreadings autonomously.
  • Examples:
    • Correcting 'LO' to '10' in medication dosages or fixing table sums.
  • Terminology:
    • Agentic OCR
    • Speculative decoding
    • Token-level edits
    • Agent-in-the-loop
  • Why it matters: It achieves high-confidence outputs without the high latency and cost of manual human review.

Data Formatting and Optimizing for RAG

  • Explanation: Tailoring document representations specifically for LLM reasoning versus embedding model retrieval.
  • Key claims:
    • Simple tables are token-efficient in Markdown; complex tables with merged cells require HTML.
    • Embedding models struggle with HTML tag noise and prefer natural language representations.
    • Splitting parses into dual representations (HTML for LLMs, natural language for embeddings) optimizes retrieval and reasoning.
  • Examples:
    • Transforming financial table data into natural language summaries for embeddings while preserving HTML tables for LLM reasoning.
  • Terminology:
    • Markdown
    • HTML tables
    • Natural language representation
    • Embedding models
  • Why it matters: Matching data format to the specific model (LLM vs. embedding) drastically improves retrieval accuracy and reduces token costs.

Effective Ingestion Pipelines and Pipeline Stages

  • Explanation: Decomposing ingestion into modular pipeline stages: classification, parsing, splitting, extraction, and formatting.
  • Key claims:
    • Classification and splitting are underinvestested stages that unlock real agent performance.
    • Routing documents to specialized parsers prevents quality erosion from bloated context windows.
    • Evaluating every stage independently (parse, retrieve, format, agent output) enables precise debugging.
  • Examples:
    • Routing hundreds of pages of mixed paper mail to appropriate parsing sub-pipelines.
  • Terminology:
    • Ingestion pipeline
    • Classification
    • Splitting
    • Parsing
    • Stage-level evaluation
  • Why it matters: End-to-end metrics do not tell you where errors occur; instrumenting every pipeline stage allows targeted investment and debugging.

Agent Harnesses as a New Frontier for Data

  • Explanation: Using structured agent loops (Generate, Evaluate, Refine) with code interpreters and multi-agent harnesses to solve complex extraction and chart analysis tasks.
  • Key claims:
    • Agent harnesses let models repeatedly improve their own output through structured loops.
    • Line chart extraction and complex structured extraction benefit from iterative self-correction loops.
    • File-based agent architectures allow agents to navigate parsed files directly instead of pre-computed snippets.
  • Examples:
    • Reconstructing complex line charts and multi-field CBP forms using iterative code execution and validation.
  • Terminology:
    • Agent harnesses
    • Generate-Evaluate-Refine loop
    • Code interpreter
    • File-based agent architecture
  • Why it matters: Agent harnesses overcome the limitations of single-shot model inference, closing the gap on precision and recall for complex long-tail documents.

Key Points

Bad inputs lead to bad outputs in multi-step agent pipelines

  • Explanation: When agents execute multi-step workflows across diverse files, initial ingestion errors compound throughout the pipeline.
  • Evidence: Insurance loss development factor calculations resulting in wrong outcomes due to incorrect table parsing.
  • Practical implication: AI teams must invest heavily in upstream data ingestion quality and verification before feeding documents to agents.

Hybrid CV and VLM architectures maximize parsing accuracy

  • Explanation: Combining sub-100M parameter traditional computer vision models for layout detection with VLMs for semantic reasoning creates superior efficiency and accuracy.
  • Evidence: Traditional CV efficiently handles object detection, layout understanding, and table segmentation on CPUs.
  • Practical implication: Avoid relying solely on massive foundation models for low-level structure tasks when smaller CV models are faster and cheaper.

Dual representation optimizes RAG for both LLMs and embeddings

  • Explanation: LLMs reason best over structured markup (HTML/Markdown), while embedding models perform better with natural language prose.
  • Evidence: HTML table structures contain too much tag noise for embedding models to correlate with natural language queries.
  • Practical implication: Generate separate representations of parsed content: HTML for LLM reasoning and natural language summaries for vector embeddings.

Stage-level evaluation is essential for production reliability

  • Explanation: Evaluating only end-to-end metrics obscures where failures happen in complex document processing pipelines.
  • Evidence: Best-performing teams evaluate parse, retrieve, format, and agent output independently.
  • Practical implication: Instrument every stage of your ingestion and agent pipeline to pinpoint and fix errors quickly.

Frameworks, Models & Processes

Agentic OCR Framework

  • How it works: Combines traditional CV/VLM parsing with an automated verification and correction layer that applies token-level edits (speculative decoding) to fix OCR mistakes.
  • Components:
    • Document input
    • CV + VLM parse
    • Verification and correction layer
    • High-confidence output
  • When to use: When processing enterprise documents with handwriting, low-quality scans, or tables where 100% data fidelity is critical.

Harness Pattern (Generate-Evaluate-Refine)

  • How it works: A structured agent loop where the model generates an initial extraction, the output is checked against defined criteria (consistency, completeness, structural correctness), and flagged issues trigger targeted re-runs.
  • Components:
    • Generate (initial extraction or structured output)
    • Evaluate (check against criteria)
    • Refine (targeted re-run on flagged issues)
  • When to use: For complex extraction tasks like line charts, large financial tables, or multi-page forms requiring high precision.

File-Based Agent Architecture

  • How it works: Agents navigate parsed files directly using code and CLI tools, querying content in context while fetching metadata on-demand rather than relying on bloated static context windows.
  • Components:
    • Parse (structure preserved)
    • Classify (route to type)
    • Agent navigates (real-time decisions)
    • Full context / metadata on demand
  • When to use: When building advanced AI agents operating over large enterprise document corpuses and complex multi-page files.

Examples & Case Studies

An insurance agent processing loss development factors pulled data from the wrong row in an unstructured table, resulting in a confident, citation-backed but entirely wrong calculation.

  • Illustrates: How bad document ingestion inputs cause agents to generate plausible-sounding but incorrect final outputs.
  • Lesson: Upstream parsing accuracy and verification are vital before agent reasoning.

Reconstructing messy line charts and complex financial tables by pairing vision models with code interpreters in an iterative refinement harness.

  • Illustrates: How agent harnesses successfully extract pixel-level data from charts that single-shot models fail to process.
  • Lesson: Equipping agents with tools and evaluation loops solves long-tail document extraction problems.

Actionable Takeaways

  • Immediate:
    • Decompose parsing problems into specialized tools rather than relying on a single monolithic model.
    • Implement agentic verification and speculative decoding to catch OCR and extraction errors.
    • Format data appropriately for its consumer (HTML/Markdown for LLMs, natural language for embeddings).
  • Strategic:
    • Invest in classification and splitting stages in ingestion pipelines to prevent context bloat.
    • Build stage-level evaluations across parsing, retrieval, formatting, and agent outputs.
    • Transition from static RAG pipelines to file-based agent architectures with bidirectional read/write capabilities.
  • Questions to investigate:
    • How can we optimize token consumption during multi-step agent document reasoning?
    • What is the optimal balance between traditional CV models and frontier VLMs for our specific document types?
    • How do we systematically evaluate retrieval recall on complex multi-page enterprise documents?

Claims Worth Verifying

  • Current frontier models score about 30% on surge data lab GDP-PDF benchmark. (benchmark performance)
  • Sub-100M parameter computer vision models running on CPU are highly effective for document layout detection. (architectural efficiency)

Notable Quotes

"Better agents through better data." (at 0:17) "An agent acting on bad data doesn't just return a wrong answer — it creates a wrong outcome." (at 6:47) "PDFs are fundamentally an image format with a broad range of encodings." (at 10:05) "VLM's unlock a lot — but the long tail is where production breaks." (at 12:04) "Evaluations should underpin your decisions." (at 31:42)

Compressed Summary

  • AI shift from RAG to autonomous agents increases the criticality of clean document ingestion data.
  • PDFs remain difficult due to legacy image-based encodings, complex tables, and handwriting.
  • Combining traditional CV for layout and VLMs for semantics optimizes accuracy and latency.
  • Agentic OCR uses speculative decoding and agent verification to automatically correct extraction errors.
  • Dual representation (HTML for LLMs, natural language for embeddings) improves RAG performance.
  • Stage-level evaluation across classification, parsing, retrieval, and formatting is essential for production agents.
  • Keywords: document intelligence, agentic OCR, vision-language models, RAG, agent harnesses, data ingestion
  • Core insight: Building reliable AI agents requires robust upstream document ingestion pipelines, agentic verification loops, and stage-level evaluations rather than relying on single-shot model inference.

Core insights

6
Mechanismhigh noveltymoderate evidence

Serve documents in multiple representations split by consumer model: HTML for LLM reasoning (preserves merged-cell/complex table structure) and natural-language renderings for embedding models (which are degraded by HTML tag noise). The 'parse' becomes a routing artifact rather than a single canonical text blob.

Why it matters

A single text extraction cannot simultaneously satisfy LLM reasoning fidelity and embedding retrieval quality; choosing one format forces a measurable loss in either retrieval accuracy or reasoning accuracy, plus token waste.

Generalization

Any RAG system should generate per-consumer views of the same source artifact and route each view to the model class that consumes it, rather than normalizing to one 'canonical' chunk format.

Splitting parses into dual representations (HTML for LLMs, natural language for embeddings) optimizes retrieval and reasoning.
Open source video
Embedding models struggle with HTML tag noise and prefer natural language representations.
Open source video
Simple tables are token-efficient in Markdown; complex tables with merged cells require HTML.
Open source video
Architecturemedium noveltymoderate evidence

Hybrid document understanding: cheap sub-100M-parameter CV models on CPU handle deterministic structure (layout segmentation, object detection, reading order), while VLMs are reserved for semantic and long-tail elements like handwriting. Capability is tiered by compute cost rather than applying a frontier VLM to everything.

Why it matters

Determines the latency/cost envelope of an ingestion service: heavy VLM calls on every page are economical only where semantics actually require them, while structure can be extracted at CPU cost.

Generalization

Layer specialized small models for deterministic sub-tasks beneath a large model used only for judgment/long-tail cases; the boundary is task determinism, not model quality.

Traditional CV models (sub-100M parameters) running on CPU are highly efficient at layout segmentation, object detection, and reading order.
Open source video
A hybrid approach combining traditional CV and VLMs yields superior accuracy.
Open source video
VLMs introduce horizontal semantic understanding and handle long-tail elements like handwriting.
Open source video
Mechanismhigh noveltymoderate evidence

Agentic OCR replaces human-in-the-loop review with agent-in-the-loop: verification/correction agents apply speculative decoding token-level edits to fix OCR hallucinations, and independently cross-check derived quantities (e.g., re-summing table totals, correcting a misread digit).

Why it matters

Moves quality assurance from a manual, latency-bound review queue into the pipeline itself, changing where trust and verification responsibilities live in the architecture.

Generalization

Wherever a model produces a structured artifact with internal consistency constraints (sums, units, schema validity), a verifier agent can check invariants and patch at the token/field level instead of escalating to a human.

Agentic OCR replaces manual review teams with verification and correction agents.
Open source video
Speculative decoding applies token-level edits to correct OCR hallucinations or extraction errors.
Open source video
Agents can cross-verify table totals and correct digit misreadings autonomously.
Open source video
Mental Modelmedium noveltymoderate evidence

The shift from information synthesis to multi-step autonomous execution changes the error model: in RAG a bad parse degrades one answer, but in an agent pipeline errors compound across sequential decisions over multiple files into catastrophic failures.

Why it matters

Ingestion accuracy becomes a reliability-critical boundary rather than a quality-of-life concern; per-step error budgets must be multiplied, not averaged, across pipeline stages.

Generalization

For any agentic pipeline, upstream data fidelity requirements scale with the number of dependent downstream decisions; reliability engineering should target the earliest shared input.

Agentic workflows introduce compounding risks when fed bad data inputs.
Open source video
When agents make sequential decisions across multiple files, data errors compound and lead to catastrophic pipeline failures.
Open source video
Practicemedium noveltyweak evidence

Classification and splitting are under-invested stages that gate downstream agent performance: routing pages/documents to specialized parsers preserves context quality and avoids degradation from over-stuffed context windows.

Why it matters

Reframes ingestion as a routing problem, not just a parsing problem — a document sent to the wrong parser silently poisons every downstream stage.

Generalization

Multi-format workloads should be partitioned before expensive processing; heterogeneous inputs fed to one general parser dilute context and reduce per-item quality.

Classification and splitting are underinvestested stages that unlock real agent performance.
Open source video
Routing documents to specialized parsers prevents quality erosion from bloated context windows.
Open source video
Practicelow noveltyweak evidence

Independently evaluate every pipeline stage (parse, retrieve, format, agent output), because end-to-end metrics cannot localize where errors originate.

Why it matters

Without stage-level instrumentation, a failed agent result is unattributable, making iteration rate on ingestion pipelines effectively zero.

Generalization

Agent systems need per-stage eval harnesses and traceable intermediate artifacts, mirroring distributed-tracing discipline applied to model quality.

Evaluating every stage independently (parse, retrieve, format, agent output) enables precise debugging.
Open source video

Deep dives

5

Error compounding and error budgets in multi-step agent pipelines

Research question

How does a single upstream parse error amplify across N sequential agent decisions, and what per-stage accuracy budget is required to keep end-to-end failure probability below a threshold?

Why

Ingestion accuracy becomes a reliability-critical boundary; teams need quantitative models to decide where to invest in data quality versus agent robustness.

Agentic workflows introduce compounding risks when fed bad data inputs.
Open source video
When agents make sequential decisions across multiple files, data errors compound and lead to catastrophic pipeline failures.
Open source video
Source video

Consumer-aware dual representations: consistency, routing, and storage cost

Research question

When the same document is rendered as HTML for LLM reasoning and natural language for embeddings, how often do the views diverge, and what is the retrieval/reasoning penalty when they disagree?

Why

A single canonical text cannot satisfy both consumer classes; dual views create new consistency and storage challenges that must be instrumented.

Splitting parses into dual representations (HTML for LLMs, natural language for embeddings) optimizes retrieval and reasoning.
Open source video
Embedding models struggle with HTML tag noise and prefer natural language representations.
Open source video
Simple tables are token-efficient in Markdown; complex tables with merged cells require HTML.
Open source video
Source video

Agentic OCR correction precision and escalation policies

Research question

What is the precision/recall of autonomous token-level corrections (speculative decoding) versus human review, and at what residual error rate must agentic corrections escalate to humans?

Why

Replacing human review with agent-in-the-loop shifts trust into the pipeline; without measured correction precision, silent errors become harder to detect than obvious OCR noise.

Agentic OCR replaces manual review teams with verification and correction agents.
Open source video
Speculative decoding applies token-level edits to correct OCR hallucinations or extraction errors.
Open source video
Agents can cross-verify table totals and correct digit misreadings autonomously.
Open source video
Source video

Optimal routing boundary between CPU CV and VLM in document ingestion

Research question

For a given document corpus, what routing rule between sub-100M CPU CV models and VLMs achieves the best accuracy/cost trade-off, and how much does misrouting degrade end-to-end quality?

Why

Hybrid architectures are claimed superior, but the boundary is a design decision with cost, latency, and error-propagation consequences.

Traditional CV models (sub-100M parameters) running on CPU are highly efficient at layout segmentation, object detection, and reading order.
Open source video
A hybrid approach combining traditional CV and VLMs yields superior accuracy.
Open source video
VLMs introduce horizontal semantic understanding and handle long-tail elements like handwriting.
Open source video
Source video

Stage-level evaluation and error attribution in lossy ingestion pipelines

Research question

How can end-to-end errors be attributed to specific pipeline stages when intermediate artifacts are lossy and interdependent, and what controlled error injection methods reveal attribution?

Why

Without stage-level instrumentation, a failed agent result is unattributable, making iteration rate effectively zero.

Evaluating every stage independently (parse, retrieve, format, agent output) enables precise debugging.
Open source video
Source video

Article ideas

4

The Error-Compounding Tax: Why Agentic Pipelines Force a New Ingestion Accuracy Bar

In multi-step agent pipelines, upstream data errors multiply across sequential decisions rather than averaging out, so ingestion fidelity requirements scale with pipeline depth—making the earliest shared input the highest-leverage reliability investment.

Angle

Quantitative reliability engineering for AI pipelines: model error amplification, derive per-stage accuracy budgets, and argue against treating ingestion as a pre-processing afterthought.

Source video

Kill the Canonical Chunk: Consumer-Aware Document Representations for RAG and Agents

A single text extraction cannot serve both embedding models and LLMs; production pipelines must emit multiple, explicitly routed views (HTML for reasoning, natural language for retrieval) and accept the consistency/storage overhead as a first-class design cost.

Angle

Architecture argument: the 'parse' stage becomes a routing artifact, forcing index/context stores to diverge and formatting to get its own evaluation.

Source video

Agentic OCR Won't Replace Human Review Until We Measure Correction Precision

Autonomous token-level corrections introduce silent errors that are harder to detect than obvious OCR noise; replacing human review depends on measurable per-error-type precision/recall and explicit escalation policies, not capability demos.

Angle

Evidence-based caution against agent-in-the-loop hype; proposes a benchmark framework and threshold-setting methodology.

Source video

The Hybrid Ingestion Stack: Cost-Tiered CV + VLM as the Default Architecture

Deterministic structure extraction belongs on CPU with small models while VLMs should handle only semantic long-tail cases; this boundary is a cost/latency design decision, and misrouting is a first-class error source that must be instrumented.

Angle

Practical architecture blueprint with cost/accuracy trade-offs, routing error analysis, and failure modes.

Source video

Project ideas

4

Error Amplification Simulator for Agentic Pipelines

movement-lab

A single upstream parse error (e.g., one wrong digit) propagates through N sequential agent decisions with a measurable amplification factor; final task failure probability increases superlinearly with step count and error severity, allowing derivation of per-stage accuracy budgets.

Proof of concept

Build a mock multi-step agent pipeline (extract → compute → summarize → decide) fed with controlled parse errors of varying types and severities; instrument each step and measure final task success vs. error severity and number of steps.

Measurement

Amplification factor (increase in final error probability per step), error-sensitive stages, and the per-stage accuracy threshold required to keep end-to-end failure below 5%.

Source video

DualView Consistency Checker

gatehouse

HTML and natural-language representations of the same document can drift; an automated divergence score (entity, numeric, table-structure mismatches) correlates with downstream retrieval-reasoning disagreement and can predict agent failures before they occur.

Proof of concept

Parse a corpus into both views, compute divergence metrics, then run retrieval and QA tasks using each view; correlate divergence scores with inconsistencies in answers and retrievals.

Measurement

Correlation between divergence score and drop in retrieval/QA accuracy; precision of flagging documents whose dual views cause agent errors.

Source video

Agentic Correction Precision Benchmark

beyond-evals

Autonomous token-level corrections (speculative decoding) have lower precision on numeric digits and table totals than on prose, and precision varies by document class; a benchmark can derive safe escalation thresholds per error type.

Proof of concept

Take a labeled OCR error corpus, run a correction agent that applies token-level edits, and measure precision/recall per error type and document class against a human review baseline.

Measurement

Precision/recall of corrections, false-correction rate, and the threshold where correction precision drops below acceptable levels for autonomous use.

Source video

Hybrid Router Cost-Accuracy Profiler

new

For a document corpus, there exists a routing rule between CPU CV and VLM that achieves ≥90% of VLM-only accuracy at ≤30% of cost; misrouting cost (accuracy loss from routing errors) dominates end-to-end accuracy more than parser quality differences.

Proof of concept

Implement a page-level router that classifies pages into deterministic-structure vs. semantic-long-tail, and compare accuracy/cost against VLM-only and CV-only baselines across corpora.

Measurement

Accuracy-cost Pareto frontier, sensitivity of end-to-end accuracy to routing error rate, and the optimal routing threshold.

Source video

Architectural implications

3

Document representations must become consumer-aware rather than canonical.

Before

One text extraction per document, shared by the embedding index and the LLM context.

After

Each parse emits multiple views (structured HTML for LLM reasoning, natural-language for embeddings) with explicit routing.

Consequence

Formatting becomes a first-class pipeline stage with its own evaluation, and index/context stores can diverge in representation.

Source video

Verification moves inside the data pipeline as an agentic stage.

Before

Human-in-the-loop review queue corrects extraction errors post-hoc.

After

Agent-in-the-loop verifiers perform token-level corrective edits and invariant checks autonomously.

Consequence

New failure surface: an autonomous correction can silently introduce errors, so correction confidence and audit trails become required instrumentation.

Source video

Ingestion decomposes into modular, independently evaluable stages (classification, parsing, splitting, extraction, formatting).

Before

Monolithic parse-to-text with a single end-to-end accuracy metric.

After

Stage-scoped pipelines where each boundary has its own eval and can be swapped.

Consequence

Per-stage observability enables error attribution and targeted improvement, but requires intermediate artifacts to be persisted and versioned.

Source video

Tradeoffs and failure modes

4

Hybrid CV + VLM pipeline

Benefit

Superior accuracy with efficient CPU structure extraction plus VLM semantics.

Cost or risk

Two model tiers to maintain, with routing decisions that can misclassify certain documents, plus latency/throughput variance between CPU and VLM paths.

A hybrid approach combining traditional CV and VLMs yields superior accuracy.
Open source video
Source video

Format selection (Markdown vs HTML)

Benefit

Markdown is token-efficient; HTML preserves merged-cell structure for complex tables.

Cost or risk

HTML structure needed for fidelity degrades embedding retrieval due to tag noise, forcing duplicated representations and higher storage/consistency cost.

Embedding models struggle with HTML tag noise and prefer natural language representations.
Open source video
Source video

Agentic OCR corrections

Benefit

High-confidence output without manual review latency and cost.

Cost or risk

Autonomous token-level edits can silently produce wrong values (e.g., a corrected digit) so errors may become harder to detect than obvious OCR noise.

Speculative decoding applies token-level edits to correct OCR hallucinations or extraction errors.
Open source video
Source video

Parser routing by classification

Benefit

Specialized parsers prevent quality erosion from bloated context windows.

Cost or risk

Misrouting a document sends it through the wrong parser, and classification itself becomes a new error source upstream of everything.

Routing documents to specialized parsers prevents quality erosion from bloated context windows.
Open source video
Source video

Open questions

5

How should error attribution be measured across pipeline stages when stage outputs are lossy and interdependent?

Why unresolved

The summary asserts stage-level evaluation enables debugging but gives no method for assigning blame or quantifying per-stage contributions.

Research direction

Define per-stage eval harnesses with intermediate artifact capture and controlled error injection to build an attribution model.

Source video

What confidence threshold should trigger an agentic correction vs. human escalation?

Why unresolved

The summary frames agent-in-the-loop as replacing human review but does not specify when autonomous edits are safe.

Research direction

Benchmark corrective-edit precision/recall against human review and derive escalation policies from measured error distributions.

Source video

Do dual representations (HTML vs natural language) stay consistent, and how is drift detected?

Why unresolved

Two derived views of one document can diverge; the summary does not address consistency maintenance.

Research direction

Build consistency checks and measure retrieval/reasoning impact when views disagree.

Source video

Does routing/classification quality dominate end-to-end accuracy compared to parser quality?

Why unresolved

Classification is claimed to be under-invested and high-leverage, but no comparative evidence is given.

Research direction

Ablate routing accuracy vs. parser capability to quantify sensitivity of final agent performance.

Source video

How robust is the ~30% frontier benchmark number across document types, and what does it imply for deploying foundation models directly on PDFs?

Why unresolved

Only a single benchmark figure is cited, with no breakdown by document class.

Research direction

Replicate on varied corpora (handwriting, merged-cell tables, scanned forms) to profile where frontier models fail.

Source video

Key claims

8
factualVerification needed

Current frontier models score around 30% on rigorous PDF benchmarks such as GDP-PDF.

Evidence

Current frontier models score around 30% on rigorous benchmarks like GDP-PDF.

Question

What exactly does GDP-PDF measure, and is ~30% reproducible across current frontier models?

Source video
causalVerification needed

PDFs are fundamentally an image format designed in the 1990s for printing fidelity, making semantic extraction exceptionally challenging.

Evidence

PDFs are fundamentally an image format designed in the 1990s for printing fidelity, making semantic extraction exceptionally challenging.

Question

To what degree does the print-fidelity origin of PDF actually explain observed extraction failures vs. other factors like scan quality?

Source video
comparativeVerification needed

Embedding models are degraded by HTML tag noise and perform better on natural-language representations.

Evidence

Embedding models struggle with HTML tag noise and prefer natural language representations.

Question

How large is the retrieval quality delta between HTML and natural-language table representations across embedding models?

Source video
comparativeVerification needed

A hybrid approach combining traditional CV and VLMs yields superior accuracy to either alone.

Evidence

A hybrid approach combining traditional CV and VLMs yields superior accuracy.

Question

On which document classes and by what margin does the hybrid outperform VLM-only or CV-only pipelines?

Source video
causalVerification needed

In multi-step agent pipelines, data errors compound and lead to catastrophic pipeline failures.

Evidence

When agents make sequential decisions across multiple files, data errors compound and lead to catastrophic pipeline failures.

Question

What is the empirical amplification factor of a single upstream parse error across N agent steps?

Source video
opinionVerification not requested

Classification and splitting are under-invested stages that unlock real agent performance.

Evidence

Classification and splitting are underinvestested stages that unlock real agent performance.

Source video
predictionVerification needed

Agentic OCR can replace manual review teams with autonomous verification and correction agents.

Evidence

Agentic OCR replaces manual review teams with verification and correction agents.

Question

At what residual error rate do agentic corrections require human oversight, measured on real enterprise document mixes?

Source video
causalVerification needed

Routing documents to specialized parsers prevents quality erosion caused by bloated context windows.

Evidence

Routing documents to specialized parsers prevents quality erosion from bloated context windows.

Question

How does parsing accuracy degrade with context length/window size for a general parser vs. specialized parsers?

Source video

Connections

5