AI Engineer · Published 2026-09-15

Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

privateReview: unreviewedConfidence: high
voice agentstranscriptionprefix cachetool injectionrealtime aiSpeculative decodingPrefix cachingttsTool callinglatencycascaded architecture
Open on YouTube ↗

Summary

Overview

  • Speaker: Bohan Li
  • Channel: AI Engineer
  • Main topic: Architecting real-time voice agents using cascading intelligence, streaming transcribers, async tool injection, and audio prefix caching at EliseAI
  • Purpose: To share engineering strategies, architectural patterns, and practical techniques for building fast, intelligent real-time conversational voice agents at scale. Bohan Li from EliseAI discusses the architecture of real-time voice agents that combine low-latency streaming models with frontier-level intelligence. He draws parallels to autonomous vehicle perception, planning, and control systems, breaking down how EliseAI structures transcription layers, language model processing, and text-to-speech pipelines to maintain speed without sacrificing intelligence.

Topic Map

Introduction and Parallels to Autonomous Driving

  • Explanation: Bohan introduces voice agents through the lens of autonomous driving, mapping perception (transcription), planning (language model reasoning), and control (text-to-speech) to voice agent architecture.
  • Key claims:
    • Cascaded voice agents map cleanly to perception, planning, and control architectures found in self-driving cars.
  • Examples:
    • Bounding boxes and lidar in cars correspond to transcription and signal conversion in voice agents.
  • Terminology:
    • perception
    • planning
    • control
    • transcription
  • Why it matters: Provides a mental model for breaking down complex real-time voice systems into discrete engineering layers.

Streaming Speculative Transcriber Layer

  • Explanation: Explains the streaming speculative transcriber layer, which layers a fast streaming transcriber (like Flux) on top of an accurate batch transcriber (like Scribe V2) to balance speed and accuracy.
  • Key claims:
    • Layering fast streaming transcribers with accurate batch transcribers allows for quick initial detections while retaining deep context.
  • Examples:
    • User says name and date of birth; fast transcriber captures early signals while batch transcriber catches phonetic details.
  • Terminology:
    • streaming speculative transcriber
    • flux
    • scribe v2
    • corrective layer
  • Why it matters: Solves the trade-off between transcription speed and contextual accuracy in real-time dialogue.

Async Background Tool Injection

  • Explanation: Discusses reducing round trips and tool-calling inference overhead by using background agents to execute tool calls and push results back into the main agent context without slowing down responses.
  • Key claims:
    • Background agents can handle tool calling asynchronously, pushing tool outputs into the main agent context to reduce inference round trips.
  • Examples:
    • Background lookup of patient name and date of birth running asynchronously during conversation.
  • Terminology:
    • tool calling
    • background agents
    • inference round trips
  • Why it matters: Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.

Audio Prefix Cache and Text-to-Speech Optimization

  • Explanation: Explains audio prefix caching and WebSocket streaming for text-to-speech engines to hide latency and deliver instant responses.
  • Key claims:
    • Audio prefix caching matches generated sequences against previously generated audio to suppress redundant generation and hide latency.
  • Examples:
    • Caching standard phrases like 'You said your name is' and yielding cached audio while waiting for dynamic entity data.
  • Terminology:
    • audio prefix cache
    • cartesia
    • websocket
    • text-to-speech
  • Why it matters: Eliminates perceived latency in voice responses, making AI agents sound completely natural and conversational.

Key Points

Cascaded Architecture for Voice Agents

  • Explanation: Voice agents require a structured pipeline separating perception, planning, and control layers, analogous to robotics.
  • Evidence: Described through transcription, LLM reasoning, and TTS generation steps.
  • Practical implication: Allows modular optimization of speed and accuracy at each pipeline stage.

Streaming Speculative Transcriber

  • Explanation: Combining fast streaming transcription with accurate batch transcription yields both low latency and correct entity recognition.
  • Evidence: Using Flux on top of Scribe V2 with phonetic matching.
  • Practical implication: Prevents transcription errors on names and medical data while maintaining rapid response times.

Background Tool Injection

  • Explanation: Offloading tool calls to background agents prevents blocking the main dialogue loop.
  • Evidence: Background tool calling for database lookups while user is speaking.
  • Practical implication: Reduces round-trip inference overhead in low-parameter language models.

Audio Prefix Caching

  • Explanation: Caching common response prefixes and playing pre-rendered or cached audio segments hides TTS generation latency.
  • Evidence: Cartesia TTS integration caching 'You said your name is' sequences.
  • Practical implication: Delivers human-like conversational responsiveness without awkward pauses.

Frameworks, Models & Processes

Cascaded Voice Agent Architecture

  • How it works: Processes real-world audio signals through a perception layer (transcription), a planning layer (LLM reasoning and tool calling), and a control layer (text-to-speech audio generation).
  • Components:
    • Perception Layer (Streaming Speculative Transcriber)
    • Planning Layer (Language Model & Background Tool Injection)
    • Control Layer (Audio Prefix Cache & TTS Engine)
  • When to use: Building production-grade real-time conversational voice agents for healthcare, housing, or customer service.

Examples & Case Studies

An automated OBGYN appointment booking call where the agent verifies patient name, date of birth, and insurance information in real time.

  • Illustrates: Cascading transcription correction, background database tool lookup, and audio prefix caching in a live medical scheduling scenario.
  • Lesson: Real-time voice agents must handle asynchronous corrections and streaming audio seamlessly to maintain conversational flow.

Actionable Takeaways

  • Immediate:
    • Layer fast streaming transcribers with slower batch transcribers for optimal accuracy and speed.
    • Implement audio prefix caching to eliminate perceived TTS generation latency.
  • Strategic:
    • Treat voice agent pipelines like autonomous driving systems with decoupled perception, planning, and control.
    • Offload tool-calling logic to background agents to protect the main LLM from round-trip overhead.
  • Questions to investigate:
    • How do different TTS engines handle prefix caching at scale under high concurrent load?
    • What are the cost trade-offs of running speculative multi-layer transcription in production?

Claims Worth Verifying

  • Streaming speculative transcribers can cancel inaccurate early transcriptions when higher-context batch transcribers finish. (technical)
  • Audio prefix caching suppresses redundant TTS generation and hides latency for common conversational openings. (technical)

Notable Quotes

"Cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception, planning, and controls" (at 0:51) "Help us improve life's most critical areas through AI automation" (at 20:00)

Compressed Summary

  • Voice agent architecture mirrors autonomous vehicle perception, planning, and control.
  • Streaming speculative transcribers combine Flux and Scribe V2 for speed and accuracy.
  • Async background tool injection reduces round-trip inference overhead.
  • Audio prefix caching hides text-to-speech latency for instant conversational responses.
  • Keywords: voice agents, transcription, prefix cache, tool injection, realtime ai
  • Core insight: Architecting real-time voice agents requires treating transcription, reasoning, and speech synthesis as a decoupled pipeline optimized for both low latency and frontier-level intelligence.

Core insights

5
Mechanismhigh noveltymoderate evidence

Transcription is treated as a speculative execution problem: a fast streaming transcriber emits early tokens to keep the dialogue loop moving, while a slower, higher-context batch transcriber runs behind it as a corrective layer and can cancel or overwrite the early transcript (with phonetic matching used for entity precision).

Why it matters

It decouples time-to-first-signal from final correctness, letting an agent begin reasoning before the audio is fully understood while still recovering hard entities like names, dates of birth, and insurance details. This is the transcription analogue of speculative decoding and it changes where 'truth' lives in the pipeline.

Generalization

Any streaming perception stage feeding a latency-sensitive agent can be split into a cheap provisional channel and an expensive authoritative channel, provided the downstream consumer supports retraction/revision of already-consumed context.

Layering fast streaming transcribers with accurate batch transcribers allows for quick initial detections while retaining deep context.
Open source video
Using Flux on top of Scribe V2 with phonetic matching.
Open source video
Streaming speculative transcribers can cancel inaccurate early transcriptions when higher-context batch transcribers finish.
Open source video
Architecturehigh noveltymoderate evidence

Tool calls are moved off the critical path: background agents execute tool calls and push results into the main agent's context asynchronously, instead of the primary agent blocking on a tool round trip mid-turn.

Why it matters

In a voice loop, each synchronous tool round trip is a silence the user hears. Making tool results an asynchronous context injection rather than a blocking call changes the main agent's contract from 'call and wait' to 'read what has arrived', which is what makes small/fast models viable on the hot path.

Generalization

Separate the agent that decides what to say from the agent that decides what to know; treat tool execution as a background producer writing into shared context rather than a function call in the reasoning loop.

Background agents can handle tool calling asynchronously, pushing tool outputs into the main agent context to reduce inference round trips.
Open source video
Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.
Open source video
Mechanismhigh noveltymoderate evidence

Audio prefix caching matches the model's generated text against previously generated audio so that already-rendered segments can be replayed immediately, hiding TTS generation latency for predictable openings while dynamic entity data is still resolving.

Why it matters

It converts a generation-latency problem into a lookup problem for the most common case, and it explicitly decouples 'start speaking' from 'have all the data'. Perceived latency, not measured latency, becomes the optimization target.

Generalization

In any generative output channel with high-frequency boilerplate (speech, templated responses, UI fragments), cache-and-replay the stable prefix and only generate the variable suffix; schedule the stable prefix to play before the variable suffix is known.

Audio prefix caching matches generated sequences against previously generated audio to suppress redundant generation and hide latency.
Open source video
Caching standard phrases like 'You said your name is' and yielding cached audio while waiting for dynamic entity data.
Open source video
Mental Modelmedium noveltyweak evidence

The architectural decomposition of a voice agent maps onto autonomous-driving perception / planning / control, with each layer independently optimizable for its own speed-accuracy trade-off.

Why it matters

It justifies putting a hard boundary between transcription, reasoning plus tool access, and speech synthesis, so a latency fix in one layer does not force a redesign of the others. It also suggests importing robotics-style budget allocation per stage.

Generalization

For real-time agentic systems, choose boundaries along sense / decide / act rather than along vendor or model boundaries; latency budgets then attach to stages, not to models.

Cascaded voice agents map cleanly to perception, planning, and control architectures found in self-driving cars.
Open source video
Tradeoffmedium noveltyweak evidence

The way to get frontier-level intelligence into a real-time loop is not to shrink the reasoning but to remove serialization: the summary describes small, fast models on the hot path whose main latency cost is inference round trips, addressed by overlapping work (async tools, speculative transcription, cached audio).

Why it matters

It reframes the intelligence-versus-latency problem as a scheduling and overlap problem. If round trips, not tokens, dominate, then architectural concurrency buys more responsiveness than model selection.

Generalization

When p95 latency is dominated by the number of sequential model/host hops rather than per-hop compute, invest in overlapping and pre-computing across the hops before optimizing the model itself.

Reduces round-trip inference overhead in low-parameter language models.
Open source video
Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.
Open source video

Deep dives

4

Retraction and reconciliation semantics for provisional agent context

Research question

What runtime primitives let an agent revise already-consumed transcript tokens and late-arriving tool results without restarting the turn, and how much entity-level error does deferred commitment buy?

Why

Both speculative transcription and async tool injection make the agent's working context mutable and time-stamped, yet the summary describes cancellation of transcripts without describing propagation into the planning layer. Without explicit retraction events, commit points, and staleness rules, the system will confidently act on a wrong name or date of birth — this is the main unengineered burden the patterns introduce.

Streaming speculative transcribers can cancel inaccurate early transcriptions when higher-context batch transcribers finish.
Open source video
Layering fast streaming transcribers with accurate batch transcribers allows for quick initial detections while retaining deep context.
Open source video
Using Flux on top of Scribe V2 with phonetic matching.
Open source video
Source video

Latency attribution: round trips versus model compute in the voice hot path

Research question

What fraction of p95 turn latency is attributable to sequential model/host round trips versus per-hop inference, and does overlapping work beat shrinking the model?

Why

The summary's central tradeoff claim is that synchronous tool execution, not token generation, dominates latency in small intelligent LLMs. If true, the engineering priority shifts from model selection to concurrency, pre-computation, and scheduling — a claim that is measurable and currently unmeasured.

Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.
Open source video
Background agents can handle tool calling asynchronously, pushing tool outputs into the main agent context to reduce inference round trips.
Open source video
Reduces round-trip inference overhead in low-parameter language models.
Open source video
Source video

Cache coherence between replayed audio prefixes and dynamically resolved entities

Research question

How should audio cache keys be constructed, and what divergence rate occurs when a cached boilerplate prefix plays before the entity that determines the suffix has resolved?

Why

Prefix caching converts generation latency into a lookup for predictable openings, but correctness depends on the cached prefix remaining valid once entity data lands. Cache keying, hit-rate observability, and coherence between cached and fresh audio become production concerns, and response phrasing becomes partly constrained by what is cacheable.

Caching standard phrases like 'You said your name is' and yielding cached audio while waiting for dynamic entity data.
Open source video
Audio prefix caching matches generated sequences against previously generated audio to suppress redundant generation and hide latency.
Open source video
Audio prefix caching suppresses redundant TTS generation and hides latency for common conversational openings.
Open source video
Source video

Where frontier intelligence actually sits in a small-model hot loop

Research question

Does escalating hard reasoning to an asynchronous background frontier agent improve task success on complex turns without degrading perceived latency on the hot path?

Why

The title claims frontier intelligence while the described hot path runs small, low-parameter models and optimizes latency. The unresolved topology question — whether a frontier model is one of the background agents — determines whether the architecture is a cascaded small-model system with an escalation tier or something else entirely.

Reduces round-trip inference overhead in low-parameter language models.
Open source video
Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.
Open source video
Cascaded voice agents map cleanly to perception, planning, and control architectures found in self-driving cars.
Open source video
Source video

Article ideas

3

Your Agent's Context Is Now Mutable, and Your Runtime Isn't Ready

Speculative transcription and async tool injection both push provisional information into an agent's working context, so the hard engineering problem in realtime agents is no longer perception quality but reconciliation: versioned context entries, retraction events, and explicit commit points. Teams that adopt the fast-path patterns without retraction semantics will ship agents that confidently state wrong names and dates.

Angle

Take the two latency patterns the talk treats as wins, and show that both smuggle in the same unaddressed requirement — a context store that supports revision — then sketch the primitive set (revision events, staleness timestamps, deferred entity commitment).

Source video

Stop Shrinking the Model, Start Overlapping the Hops

The intelligence-versus-latency trade-off in realtime agents is usually framed as a model-selection problem, but when p95 latency is dominated by the number of sequential round trips rather than per-hop compute, architectural concurrency buys more responsiveness than model choice. Async tool execution, speculative perception, and cached output are three instances of the same move: overlap work instead of compressing it.

Angle

Argue the reframing, then demand the measurement the framing implies — a latency attribution breakdown per hop — and show how teams can instrument it before choosing a model tier.

Source video

Perceived Latency Is the Only Latency: Cache the Boilerplate, Generate the Variable

Audio prefix caching works because it decouples 'start speaking' from 'have all the data', turning a generation problem into a lookup for the most common case. Any high-frequency generative output channel — speech, templated responses, UI fragments — should be split into a stable replayed prefix and a variable generated suffix, with the prefix scheduled to play before the suffix is known. The cost is that phrasing becomes partly constrained by what is cacheable.

Angle

Generalize the audio pattern into a design rule for generative output, and be explicit about the coherence risk when entity data contradicts the cached premise mid-utterance.

Source video

Project ideas

4

Retractable Context Store

gatehouse

Gating commitment on phonetically ambiguous entities until the corrective transcriber confirms reduces entity-level capture errors by at least 50% while adding less than 150ms to p95 turn latency.

Proof of concept

Instrument a context store with versioned entries, revision events, and per-entity commit points; run 200 scripted calls containing hard names, dates of birth, and insurance identifiers through a layered fast-plus-batch transcription path with and without deferred commitment.

Measurement

Entity-level accuracy on names/dates/identifiers, contradiction rate (agent states a value that is later revised), p95 turn latency, and the share of turns where the corrective pass fired.

Source video

Round-Trip Latency Attribution Harness

beyond-evals

At least 40% of p95 turn latency in a cascaded voice loop is attributable to sequential round trips rather than per-hop model compute, and async tool injection cuts p95 turn latency by at least 30% at equal task success.

Proof of concept

Build a trace-instrumented cascaded agent with a toggle between synchronous tool calls and background-agent context injection, replaying a fixed set of scripted conversations through both paths.

Measurement

Per-hop latency attribution breakdown, p95 turn latency, task success rate, inference round trips per turn, and the fraction of turns answered before the async tool result lands.

Source video

Prefix Cache Coherence Bench

movement-lab

On a realistic call distribution at least 60% of assistant opening segments are cacheable and hits cut time-to-first-audio-byte by at least 50%, but more than 5% of hits require mid-utterance divergence because entity data contradicts the cached suffix.

Proof of concept

Interpose a text-to-audio prefix cache between the language model and the TTS stream, with configurable cache keys over boilerplate openings, and run it against a load-replay of representative opening phrases including entity-driven variants.

Measurement

Cache hit rate, time-to-first-audio-byte on hit versus miss, cache-miss tail latency under concurrency, and divergence rate where the cached prefix premise is invalidated.

Source video

Off-Hot-Path Frontier Escalation

new

Routing hard multi-entity reasoning to an asynchronous background frontier agent while a small model holds the hot path preserves perceived latency and raises task success on complex turns by at least 20% versus a uniformly small-model loop.

Proof of concept

Run a small fast model on the spoken hot path with a background escalation agent that consumes the live context, resolves multi-entity reasoning, and injects results; compare against a hot-path-only baseline on a set of deliberately complex turns.

Measurement

Task success on complex turns, time-to-first-audio-byte, p95 turn latency, and the rate at which escalation results arrive before the turn ends.

Source video

Architectural implications

3

Transcripts become provisional and revisable rather than a single append-only log.

Before

One transcriber produces a transcript that the LLM consumes as ground truth.

After

A fast provisional transcript is consumed immediately and a slower corrective pass can invalidate earlier tokens and rewrite context.

Consequence

The agent runtime needs a reconciliation mechanism (revision events, retraction semantics, or deferred commitment on entities) — otherwise the agent acts confidently on a wrong name or date. This is the main unaddressed engineering burden introduced by the pattern.

Source video

Tool execution moves out of the reasoning loop into background producers writing into shared context.

Before

The main agent calls a tool, blocks, receives the result, then continues its turn.

After

Background agents run lookups concurrently with the user speaking and inject results into the main agent's context opportunistically.

Consequence

Context becomes a mutable, time-stamped store with staleness and ordering concerns; the main agent must reason about what it currently knows versus what is still pending, which is a new class of prompt- and state-management work.

Source video

A caching layer is interposed between text generation and speech synthesis.

Before

LLM text streams directly into the TTS engine, and the user waits for audio generation.

After

Text is matched against a cache of previously rendered audio; stable prefixes play from cache while variable suffixes generate.

Consequence

Cache keying, hit-rate observability, and coherence between cached and fresh audio become production concerns; response phrasing becomes partly constrained by what is cacheable.

Source video

Tradeoffs and failure modes

3

Speculative multi-layer transcription

Benefit

Low time-to-first-transcript plus high final accuracy on phonetically difficult entities.

Cost or risk

Two transcribers running concurrently raise per-call cost, and already-consumed incorrect tokens must be cancelled or corrected downstream — the summary does not describe how that cancellation propagates to the LLM context.

Streaming speculative transcribers can cancel inaccurate early transcriptions when higher-context batch transcribers finish.
Open source video
Source video

Audio prefix caching

Benefit

Suppresses redundant generation and removes perceived TTS latency on common openings.

Cost or risk

Only helps for predictable phrasing; correctness depends on the cached prefix matching the eventual full response, so entity-driven branching can force a mid-utterance divergence or discard the cached segment's premise.

Audio prefix caching suppresses redundant TTS generation and hides latency for common conversational openings.
Open source video
Source video

Async background tool injection

Benefit

Removes synchronous tool latency from the dialogue loop and reduces inference round trips.

Cost or risk

The main agent may commit to an utterance before a background result arrives, producing answers inconsistent with data that lands milliseconds later; the summary gives no ordering or staleness guarantees.

Background agents can handle tool calling asynchronously, pushing tool outputs into the main agent context to reduce inference round trips.
Open source video
Source video

Open questions

4

How does a runtime retract already-consumed transcript tokens and tool results from an LLM's working context without restarting the turn?

Why unresolved

The summary asserts that speculative transcriptions can be cancelled but describes no mechanism for propagating that correction into the planning layer.

Research direction

Prototype a context store with versioned, retractable entries plus commit points, and measure agent error rates on entity capture with and without deferred commitment.

Source video

What is the cost premium of running speculative dual-transcription in production, and at what call volume does it stop being viable?

Why unresolved

The summary poses this question itself and provides no numbers.

Research direction

Benchmark cost-per-minute and p50/p95 latency for single-transcriber versus layered transcription, stratified by whether entity correction fires.

Source video

How do TTS engines behave under prefix caching at high concurrent load?

Why unresolved

The summary raises this but reports no measurement; caching behavior is engine-specific and the summary names Cartesia only as an integration example.

Research direction

Load-test cached versus uncached TTS paths for hit rate, time-to-first-audio-byte, and cache-miss tail latency under concurrency.

Source video

What does 'frontier intelligence' actually mean in the hot path when the deployed loop uses small, low-parameter models?

Why unresolved

The title claims frontier intelligence while the evidence describes small intelligent LLMs and latency engineering; the summary never specifies where the larger model sits.

Research direction

Clarify the topology — whether a frontier model runs asynchronously as one of the background agents — and measure whether off-hot-path escalation improves task success without affecting perceived latency.

Source video

Key claims

6
causalVerification needed

Layering a fast streaming transcriber over an accurate batch transcriber yields both low initial latency and high final accuracy.

Evidence

Layering fast streaming transcribers with accurate batch transcribers allows for quick initial detections while retaining deep context.

Question

What is the entity-level word error rate and time-to-first-token for layered versus single-transcriber setups on names, dates, and insurance identifiers?

Source video
factualVerification needed

Early speculative transcriptions can be cancelled once a higher-context batch transcriber finishes.

Evidence

Streaming speculative transcribers can cancel inaccurate early transcriptions when higher-context batch transcribers finish.

Question

How is cancellation propagated to a downstream LLM that has already conditioned on the incorrect tokens?

Source video
causalVerification needed

Background agents handling tool calls asynchronously reduce inference round trips for the main agent.

Evidence

Background agents can handle tool calling asynchronously, pushing tool outputs into the main agent context to reduce inference round trips.

Question

Measured reduction in round trips and in p95 turn latency, and what fraction of turns now answer before the async result lands?

Source video
causalVerification needed

Audio prefix caching suppresses redundant TTS generation and hides latency for common conversational openings.

Evidence

Audio prefix caching suppresses redundant TTS generation and hides latency for common conversational openings.

Question

What cache hit rate is achievable on real call distributions, and what is the time-to-first-audio-byte improvement on hits?

Source video
causalVerification needed

Synchronous tool execution is a dominant latency cost in small intelligent LLMs.

Evidence

Drastically reduces latency caused by synchronous tool execution in small intelligent LLMs.

Question

What share of end-to-end turn latency is attributable to synchronous tool round trips versus model inference?

Source video
opinionVerification not requested

Cascaded voice agent layers correspond cleanly to perception, planning, and control in autonomous driving.

Evidence

Cascaded voice agents map cleanly to perception, planning, and control architectures found in self-driving cars.

Source video

Connections

4