AI Engineer · Published 2026-07-23

Why We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates

privateReview: unreviewedConfidence: high
multi-agentknowledge graphcontrol surfacepharma analyticsdeterminismknowledge-graphcontrol-planecontext-handoffagent-architectureagentic-loop
Open on YouTube ↗

Summary

Overview

  • Speaker: Subbiah Sethuraman and Abhilash Asokan
  • Channel: AI Engineer
  • Main topic: Lessons from building multi-agent AI pipelines for pharma commercial teams and why they transitioned to a single agent with a knowledge graph.
  • Purpose: To share practical engineering lessons, architectural failures, and successful design patterns when building agentic workflows for commercial analytics in pharma. Subbiah Sethuraman and Abhilash Asokan from ZS Associates share their journey of building multi-agent AI pipelines for pharma commercial teams. They detail why their initial multi-agent system failed—specifically due to language models deciding signals, context loss at handoffs, and lack of shared business domain understanding. They explain how they replaced the multi-agent pipeline with a bare agent, a database, and a knowledge graph acting as a deterministic control surface running an iterative loop (traverse, query, reason, conclude).

Topic Map

Pharma Commercial Domain & Analyst Workflow

  • Explanation: Overview of pharma commercial functions and the 4-step workflow of analysts: Signal, Why, Action, and Outlook.
  • Key claims:
    • Pharma has R&D and commercial functions.
    • Analysts work in four steps: Signal detection, Why is it happening, Action, Outlook.
  • Examples:
    • Prescription water drop detection
    • Competitor drug impact analysis
  • Terminology:
    • Signal Detection
    • Source Localization
    • Driver Attribution
    • Synthesis
  • Why it matters: Understanding the human analyst's analytical flow helps in mirroring it with agentic systems.

Why the Multi-Agent Pipeline Failed

  • Explanation: Analysis of the failure modes of the multi-agent architecture: LLM deciding signals, context loss across handoffs, and lack of shared business understanding.
  • Key claims:
    • A language model was deciding signals instead of deterministic statistics.
    • Context was lost at every agent handoff.
    • Agents lacked shared understanding of business metrics and table relationships.
  • Examples:
    • Brand A prescriptions dropping 18% in territory X due to insurance tier changes, leading to mismatched action generation.
  • Terminology:
    • Multi-Agent Pipeline
    • Context Hand-off
    • Driver Attribution Agent
  • Why it matters: Highlights the dangers of blindly splitting reasoning tasks across multiple communicating LLM agents without deterministic boundaries.

Deriving the Architecture with Knowledge Graphs

  • Explanation: Moving from designed architectures to derived architectures using a bare agent, database, and knowledge graph as a control surface.
  • Key claims:
    • Architecture should be derived, not designed.
    • Separate the deterministic statistics from the agentic reasoning.
    • The knowledge graph acts as the agent's control surface and restricts hypothesis evaluation.
  • Examples:
    • Statistically finding drop signals before agent wake-up; graph constraining agent traversal paths.
  • Terminology:
    • Knowledge Graph
    • Control Surface
    • Graph Contract
    • Sub-agents
  • Why it matters: Provides a robust pattern for controlling LLM agent behavior and preventing hallucinations in enterprise data workflows.

Key Points

Statistics finds the signal, the agent explains it.

  • Explanation: Signal detection should be handled by deterministic statistical methods with guardrails and thresholds before any LLM agent is invoked.
  • Evidence: Running automated pipelines that scan KPI anomalies and queue signals for agents.
  • Practical implication: Eliminates LLM non-determinism in basic metric monitoring and anomaly detection.

The knowledge graph is the agent's control surface.

  • Explanation: Instead of free-form reasoning over raw database tables, agents navigate a structured knowledge graph that defines valid entity relationships and business rules.
  • Evidence: Graph contract defines valid entity transitions (entity -> has category -> driver category).
  • Practical implication: Reduces hallucinated relationships and bounds the agent's investigation space.

Architecture should be derived, not designed.

  • Explanation: Instead of pre-designing complex multi-agent orchestrations, start with a bare agent, bash, and a database, then derive the structure based on observed behavior and bottlenecks.
  • Evidence: Building up from bare cloud code and observing where context was lost.
  • Practical implication: Leads to simpler, more reliable systems tailored to actual runtime challenges.

Frameworks, Models & Processes

Analyst Workflow Framework

  • How it works: Answers four questions in order: Signal (Is the change real?), Why (What's causing it?), Action (What should we do?), Outlook (What happens next?).
  • Components:
    • Signal Detection
    • Source Localization
    • Driver Attribution
    • Synthesis
  • When to use: When designing automated commercial analytics platforms for pharma.

Agentic Loop Framework

  • How it works: Agent traverses the knowledge graph to propose the next move, queries real numbers from the database, reasons over evidence, and repeats until root cause is found.
  • Components:
    • Traverse
    • Query
    • Reason
    • Conclude
  • When to use: When investigating complex business metrics and root causes over structured data graphs.

Examples & Case Studies

Brand A new prescriptions dropped 18% in territory X over four weeks due to an insurer moving the brand to a worse coverage tier.

  • Illustrates: Failure of multi-agent pipeline where the cause was correctly identified, but the action agent recommended sending sales reps to visit more doctors without addressing insurance coverage.
  • Lesson: No single agent saw the whole picture, leading to incoherent end-to-end reasoning across handoffs.

Actionable Takeaways

  • Immediate:
  • Strategic:
    • Separate deterministic data processing from agentic reasoning.
    • Use knowledge graphs to constrain agent reasoning paths and provide business domain context.
  • Questions to investigate:
    • How do we scale knowledge graph construction across diverse enterprise data sources?
    • What are the token cost trade-offs of the traverse-query-reason loop?

Claims Worth Verifying

  • A language model should not be used to decide whether a KPI drop is a real signal or noise. (Architectural principle)
  • Giving agents a knowledge graph control surface prevents invalid relationship inferences. (Design pattern)

Notable Quotes

"A language model is a reasoning engine, not a statistics engine." (at 11:56) "Architecture should be derived, not designed." (at 22:27) "Graph cannot be treated just as a lookup layer. Graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically." (at 23:45)

Compressed Summary

  • Multi-agent pipelines failed due to LLM signal detection, context loss at handoffs, and lack of domain understanding.
  • Replaced multi-agent systems with a single agent powered by a knowledge graph and deterministic statistical guardrails.
  • The knowledge graph acts as a control surface ensuring agents only evaluate valid business hypotheses.
  • The agent runs a strict loop: traverse the graph, query real numbers, reason over evidence, and conclude.
  • Keywords: multi-agent, knowledge graph, control surface, pharma analytics, determinism
  • Core insight: Complex agentic systems fail when LLMs handle deterministic statistics and multi-agent handoffs lose context; separating deterministic checks from agent reasoning over a knowledge graph control plane solves this.

Core insights

5
Architecturemedium noveltystrong evidence

Signal detection must be relegated to deterministic statistics with guardrails and thresholds; the LLM is invoked only to explain an already-detected anomaly. The LLM is treated as a reasoning engine, never as a statistics engine, drawing a hard architectural boundary between anomaly detection and explanation.

Why it matters

Places the non-determinism of the LLM downstream of the correctness-critical decision, so a false or noisy signal can never originate from model sampling. It defines where the deterministic/agentic seam belongs in a monitoring pipeline.

Generalization

Any agentic system that must react to numeric or time-series conditions should compute the triggering condition deterministically and let the agent operate only on confirmed events, not on raw metric streams.

A language model is a reasoning engine, not a statistics engine.
Open source video
A language model was deciding signals instead of deterministic statistics.
Open source video
Separate the deterministic statistics from the agentic reasoning.
Open source video
Mechanismhigh noveltystrong evidence

The knowledge graph is used as a control plane, not a retrieval/lookup layer: it constrains which entity transitions and hypotheses the agent may pursue, operating as a 'graph contract' (e.g. entity -> has category -> driver category).

Why it matters

Turns the graph from passive context into an active governor of the agent's search space, bounding investigation and filtering out relationships the business domain declares invalid. This is a concrete mechanism for constraining agent behavior rather than prompting for it.

Generalization

Structured domain schemas can act as runtime navigation controllers for agents — a general pattern for grounding and bounding agent reasoning in any enterprise domain with well-defined entity relations.

Graph cannot be treated just as a lookup layer. Graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically.
Open source video
Graph contract defines valid entity transitions (entity -> has category -> driver category).
Open source video
Reduces hallucinated relationships and bounds the agent's investigation space.
Open source video
Failure Modemedium noveltystrong evidence

The multi-agent pipeline failed primarily through context loss at handoffs: each agent held a valid local view, but no single agent saw the whole picture, producing end-to-end incoherence where a correct root-cause diagnosis was paired with an action that ignored the actual cause.

Why it matters

Identifies a specific, reproducible failure signature of decomposed LLM pipelines — correct intermediate steps composing into a wrong final decision — which is not caught by per-agent evaluation.

Generalization

Task decomposition across LLM agents risks semantic drift at every serialized handoff; end-to-end coherence must be evaluated at the pipeline level, not at each agent.

Context was lost at every agent handoff.
Open source video
No single agent saw the whole picture, leading to incoherent end-to-end reasoning across handoffs.
Open source video
the cause was correctly identified, but the action agent recommended sending sales reps to visit more doctors without addressing insurance coverage
Open source video
Mental Modelmedium noveltymoderate evidence

Architecture should be derived from observed runtime behavior rather than designed up front: start from a bare agent, bash, and a database, then add structure where context loss and bottlenecks actually appear.

Why it matters

Reverses the common practice of pre-designing multi-agent orchestrations, arguing that complexity should be earned by observed failure rather than assumed from a diagram.

Generalization

For agentic systems with uncertain failure surfaces, build the minimal loop first and let topology emerge from instrumentation rather than from anticipated decomposition.

Architecture should be derived, not designed.
Open source video
Building up from bare cloud code and observing where context was lost.
Open source video
Mechanismmedium noveltymoderate evidence

The agent runs an explicit traverse-query-reason-conclude loop in which the graph proposes the next move while the actual numbers are retrieved from the database at each step, keeping hypothesis generation separate from evidence retrieval.

Why it matters

Provides a concrete control flow for evidence-grounded root-cause investigation: the agent cannot fabricate the evidence because every claim in the loop is re-queried from the database.

Generalization

Agent loops over structured data can separate 'where to look next' (proposed by a structured index or schema) from 'what the data says' (retrieved deterministically from the source of truth).

Agent traverses the knowledge graph to propose the next move, queries real numbers from the database, reasons over evidence, and repeats until root cause is found.
Open source video
Statistically finding drop signals before agent wake-up; graph constraining agent traversal paths.
Open source video

Deep dives

5

Knowledge graph as runtime control plane versus retrieval layer

Research question

Does constraining an agent's traversal to a declared graph contract measurably prevent invalid relationship inferences, or only reduce their frequency, and how does that change as graph coverage becomes incomplete?

Why

The control-plane framing is the highest-novelty claim in the talk and the whole architecture's correctness argument rests on it. If the effect is only partial, the 'graph contract' becomes a soft prior rather than a guarantee, which changes how much the agent loop still needs prompt-level and eval-level guardrails.

Graph cannot be treated just as a lookup layer. Graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically.
Open source video
Graph contract defines valid entity transitions (entity -> has category -> driver category).
Open source video
Reduces hallucinated relationships and bounds the agent's investigation space.
Open source video
Source video

End-to-end coherence loss at agent handoffs

Research question

Can information loss per handoff be measured, and does it predict end-to-end incoherence where each agent's local output is correct but the composed recommendation is wrong?

Why

The Brand A case is a concrete failure signature that per-agent evaluation cannot catch. If handoff loss is measurable and predictive, pipeline-level coherence becomes a first-class eval target rather than a hope, and decomposition decisions get a quantitative test.

Context was lost at every agent handoff.
Open source video
No single agent saw the whole picture, leading to incoherent end-to-end reasoning across handoffs.
Open source video
the cause was correctly identified, but the action agent recommended sending sales reps to visit more doctors without addressing insurance coverage
Open source video
Source video

Deterministic/agentic seam for anomaly-triggered reasoning

Research question

On the same KPIs, can deterministic anomaly detection with fixed guardrails match or exceed LLM-based signal detection while eliminating sampling-induced false alarms?

Why

This is the falsifiable core of the 'reasoning engine, not a statistics engine' boundary. If thresholds under-detect relative to an LLM, the seam shifts the failure from false positives to false negatives, and the boundary must be justified with detection curves rather than principle.

A language model is a reasoning engine, not a statistics engine.
Open source video
A language model was deciding signals instead of deterministic statistics.
Open source video
Separate the deterministic statistics from the agentic reasoning.
Open source video
Source video

Token and latency economics of the traverse-query-reason-conclude loop

Research question

How do iterations per resolved root cause, re-queried evidence calls, and token spend scale with graph depth, and which traversal-pruning strategies preserve diagnosis accuracy?

Why

The loop's grounding guarantee comes from re-querying at every step, which is also its cost center. Without iteration budgets and pruning results, the pattern is hard to justify for high-volume monitoring versus a one-shot explanation.

Agent traverses the knowledge graph to propose the next move, queries real numbers from the database, reasons over evidence, and repeats until root cause is found.
Open source video
What are the token cost trade-offs of the traverse-query-reason loop?
Open source video
Statistically finding drop signals before agent wake-up; graph constraining agent traversal paths.
Open source video
Source video

Scaling graph-contract construction across heterogeneous enterprise sources

Research question

Can the graph contract (entity -> has category -> driver category) be extracted automatically from heterogeneous enterprise data with human-in-the-loop validation at a cost below manual curation?

Why

The control-plane pattern's value is capped by graph completeness, and the talk flags construction scaling as unresolved. This determines whether the pattern generalizes beyond one curated pharma domain.

How do we scale knowledge graph construction across diverse enterprise data sources?
Open source video
Graph contract defines valid entity transitions (entity -> has category -> driver category).
Open source video
Building up from bare cloud code and observing where context was lost.
Open source video
Source video

Article ideas

4

Your Multi-Agent Pipeline Didn't Fail at Reasoning — It Failed at the Handoff

Decomposed LLM pipelines fail not because each agent reasons poorly but because serialized handoffs destroy the shared context that makes local correctness composable, so per-agent evals give false confidence and only pipeline-level coherence testing catches the real defect.

Angle

Use the Brand A 18% prescription drop case — correct root cause in one agent, action agent recommending rep visits for what was an insurance tier change — as the canonical signature of the failure and argue for end-to-end eval over agent-level scorecards.

Source video

Your Knowledge Graph Is a Control Plane, Not a Vector Store

Treating a knowledge graph as a retrieval layer wastes its real value: it is the runtime governor that bounds an agent's hypothesis space through a declared contract of valid entity transitions, and that is an engineering control, not a prompting technique.

Angle

Contrast lookup-layer graph usage with contract-driven traversal (entity -> has category -> driver category), show how the graph proposes next moves while the database supplies the numbers, and be explicit about the open question of whether this reduces or actually prevents invalid inferences.

Source video

Stop Designing Agent Architectures, Start Measuring Where Context Dies

Agent topology should be earned from observed runtime failure rather than drawn up front; start from a bare agent, a shell, and a database, instrument the loop, and add structure only where context loss and bottlenecks actually appear.

Angle

Argue derive-don't-design as an engineering discipline with an explicit cost — delayed architectural commitment and required instrumentation — rather than as a slogan, and pair it with the counterexample of a pre-designed driver-attribution agent that produced incoherent output.

Source video

A Language Model Is a Reasoning Engine, Not a Statistics Engine

Putting an LLM upstream of anomaly detection means sampling noise can originate the correctness-critical decision; signal detection belongs to deterministic statistics with thresholds, and the model should be invoked only to explain an event that has already been confirmed.

Angle

Frame it as seam placement: the deterministic/agentic boundary should sit downstream of the trigger, and evaluate the trade by asking whether fixed thresholds under-detect compared with LLM signal judgment.

Source video

Project ideas

4

Control-Plane Graph Contract Harness

beyond-evals

Bounding agent traversal to a declared graph contract measurably lowers the rate of invalid relationship inferences compared with free-form reasoning over the same data, but the reduction is partial rather than complete and degrades as graph coverage drops.

Proof of concept

Build two agent variants over the same dataset — one traversing a contract-limited knowledge graph, one reasoning free-form — and adversarially probe both with entity pairs the contract forbids, measuring residual invalid inferences at 100%, 70%, and 40% graph coverage.

Measurement

Rate of invalid relationship inferences per variant at each graph-coverage level, plus traversal paths audited against the contract.

Source video

Handoff Loss Probe for Multi-Agent Chains

beyond-evals

Information lost at each agent handoff is measurable and correlates with end-to-end incoherence, so chains with higher per-handoff loss produce correct local steps that compose into wrong final recommendations.

Proof of concept

Instrument a serialized agent chain with a fixed context payload per handoff, quantify retained versus dropped fields at each step, and score final recommendations against a ground-truth action derived from the true root cause; replicate the prescription-drop-to-rep-visit mismatch as a target case.

Measurement

Per-handoff context retention rate versus pipeline-level correctness, and count of cases where a correct diagnosis is paired with an action that ignores the cause.

Source video

Deterministic Trigger versus LLM Signal Detection Bake-Off

movement-lab

Deterministic anomaly detection with fixed guardrails and thresholds matches or exceeds LLM-based signal detection on KPI streams while eliminating sampling-induced false alarms.

Proof of concept

Run both detectors over the same historical KPI series with known labeled anomalies, then route only confirmed events to an explanation agent and compare downstream explanation quality and auditability.

Measurement

Precision, recall, and false-alarm variance across repeated runs for LLM detection versus single-run determinism for threshold detection.

Source video

Traverse-Query-Reason Loop Budgeter

new

Loop iterations per resolved root cause can be cut substantially by traversal pruning without reducing diagnosis accuracy, because most traversals do not change the concluded cause.

Proof of concept

Instrument a graph-and-database agent loop to log every traversal step and whether it altered the final conclusion, then apply pruning rules that skip low-yield branches and compare accuracy and token spend against the unpruned loop.

Measurement

Iterations and tokens per correctly resolved root cause, accuracy delta versus unpruned baseline, and latency per resolution.

Source video

Architectural implications

3

Responsibility for deciding whether a metric change is a real signal is moved out of the LLM entirely.

Before

A language model decides which KPI movements are genuine signals versus noise.

After

Deterministic statistical pipelines with guardrails and thresholds scan for anomalies and queue signals for an agent that only explains them.

Consequence

Signal detection becomes reproducible and auditable; the LLM's stochasticity is confined to explanation and cannot trigger false alarms.

Source video

The knowledge graph's role is elevated from retrieval substrate to runtime control plane.

Before

The graph is treated as a lookup layer supplying context to the agent.

After

The graph acts as a control plane that the agent navigates and uses to make its next decisions, constrained by a graph contract of valid transitions.

Consequence

The agent's hypothesis space is bounded by domain-valid relationships, reducing hallucinated relationships and making traversal paths auditable.

Source video

Multi-agent decomposition is replaced by a single bare agent plus database plus knowledge graph.

Before

Reasoning is split across multiple communicating agents (e.g. a dedicated driver-attribution agent) that hand off context.

After

One agent runs an iterative traverse-query-reason-conclude loop against a graph and database, with no serialized handoffs.

Consequence

Eliminates the handoff context-loss failure mode at the cost of concentrating reasoning responsibility in one loop that must be bounded by the control surface.

Source video

Tradeoffs and failure modes

4

Multi-agent task decomposition

Benefit

Splits complex analytical reasoning into specialized agents with narrower responsibilities.

Cost or risk

Context is lost at every agent handoff, and no single agent sees the whole picture, so locally correct steps can compose into an incoherent end-to-end recommendation.

No single agent saw the whole picture, leading to incoherent end-to-end reasoning across handoffs.
Open source video
Source video

Knowledge graph as bounded control surface

Benefit

Reduces hallucinated relationships and bounds the agent's investigation space with domain-valid transitions.

Cost or risk

Requires constructing and maintaining the graph/contract across enterprise data sources; scaling that construction is an explicitly open problem.

How do we scale knowledge graph construction across diverse enterprise data sources?
Open source video
Source video

Traverse-query-reason-conclude loop

Benefit

Every step is grounded in real numbers re-queried from the database, producing evidence-backed conclusions.

Cost or risk

Iterative traversals and repeated queries carry an unresolved token-cost overhead.

What are the token cost trade-offs of the traverse-query-reason loop?
Open source video
Source video

Deriving architecture instead of designing it

Benefit

Produces simpler systems matched to observed runtime bottlenecks rather than anticipated ones.

Cost or risk

Requires instrumented observation of where context is lost before structure can be justified, delaying architectural commitment.

Building up from bare cloud code and observing where context was lost.
Open source video
Source video

Open questions

4

How do you scale knowledge graph construction across diverse enterprise data sources?

Why unresolved

The graph-as-control-plane pattern depends on a complete, valid graph contract, but the summary gives no method for building that graph across heterogeneous sources.

Research direction

Automated schema/ontology extraction from heterogeneous enterprise data with human-in-the-loop validation of graph contracts.

Source video

What are the token cost trade-offs of the traverse-query-reason loop?

Why unresolved

The iterative loop re-queries the database and re-reasons at each step, but no cost or latency measurements are reported.

Research direction

Benchmark loop iterations per resolved root cause against token/latency budgets, and test traversal-pruning strategies.

Source video

Does a knowledge graph control surface merely reduce invalid relationship inferences or actually prevent them?

Why unresolved

One claim states the control surface 'reduces' hallucinated relationships while another states it 'prevents' them; the summary does not resolve which is demonstrated.

Research direction

Adversarial evaluation of agents against a graph contract, measuring the residual rate of invalid relationship inferences.

Source video

When is multi-agent decomposition still the right choice versus a single agent with a deterministic control surface?

Why unresolved

The talk argues for derivation over design but gives no boundary conditions for when splitting into agents remains warranted.

Research direction

Comparative study of single-loop agent-plus-graph versus decomposed agents on tasks with differing context-handoff sensitivity.

Source video

Key claims

7
causalVerification needed

A language model was deciding signals instead of deterministic statistics, which was a root cause of the pipeline failure.

Evidence

A language model was deciding signals instead of deterministic statistics.

Question

Can deterministic anomaly detection with fixed thresholds match or exceed LLM-based signal detection on the same KPIs, and how much non-determinism does the LLM introduce?

Source video
causalVerification needed

Context was lost at every agent handoff in the multi-agent pipeline.

Evidence

Context was lost at every agent handoff.

Question

Can the amount of information lost per handoff be measured, and does handoff loss correlate with degraded end-to-end outcomes?

Source video
causalVerification needed

The knowledge graph acts as the agent's control surface and restricts hypothesis evaluation.

Evidence

The knowledge graph acts as the agent's control surface and restricts hypothesis evaluation.

Question

Does bounding traversal to a graph contract measurably reduce invalid inferences compared to free-form reasoning over the same data?

Source video
opinionVerification not requested

Architecture should be derived, not designed.

Evidence

Architecture should be derived, not designed.

Source video
causalVerification needed

Giving agents a knowledge graph control surface prevents invalid relationship inferences.

Evidence

Giving agents a knowledge graph control surface prevents invalid relationship inferences.

Question

Is the reduction in invalid relationship inferences complete (prevention) or partial (reduction), and under what graph completeness conditions?

Source video
factualVerification needed

An 18% four-week new-prescription drop for Brand A in territory X was correctly attributed to an insurer coverage-tier change, but the action agent recommended more rep visits instead.

Evidence

Brand A new prescriptions dropped 18% in territory X over four weeks due to an insurer moving the brand to a worse coverage tier.

Question

Is this case representative of multi-agent handoff failures, or specific to how the action agent's context was truncated?

Source video
factualVerification needed

Architecture was derived by starting from a bare agent, bash, and a database and adding structure only where context losses were observed.

Evidence

Building up from bare cloud code and observing where context was lost.

Question

Does derive-don't-design systematically yield simpler, more reliable systems than upfront multi-agent design across comparable projects?

Source video

Connections

5