AI Engineer · Published 2026-10-11

Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit

privateReview: unreviewedConfidence: high
latencyvoice-agentturn-detectionlivekitbenchmarkinglatency-budgetvoice-agentsttsasync-toolsconversation-benchmarkingperceived-latencystreaming
Open on YouTube ↗

Summary

Overview

  • Speaker: Jesse Hall
  • Channel: AI Engineer
  • Main topic: Real-time voice AI infrastructure, latency management, turn-taking, and humanlike agent design.
  • Purpose: To educate AI engineers on how to build responsive, humanlike real-time voice agents using LiveKit infrastructure and conversation-level benchmarking. Jesse Hall from LiveKit discusses the limitations of traditional component-based evaluations for real-time voice agents. He explains why component leaderboards lead engineers astray, how latency is a budget rather than just raw speed, and why asynchronous tool execution and audio-based turn detection are necessary to achieve humanlike voice agent behavior.

Topic Map

The Limitations of Component Leaderboards

  • Explanation: Why picking top-ranked components (STT, LLM, TTS) independently fails to deliver a functional real-time conversation pipeline.
  • Key claims:
    • Errors flow downhill in component cascades.
    • Component leaderboards score models in clean lab conditions, not real-world mobile traffic.
    • The leaderboard is a trap: measuring the model is not the same as shipping a conversation.
  • Examples:
    • STT hears 'Tuesday the 30th' instead of 'Tuesday the 3rd', causing downstream LLM to book the wrong date.
  • Terminology:
    • Component Leaderboard
    • Speech-to-Text (STT)
    • Large Language Model (LLM)
    • Text-to-Speech (TTS)
  • Why it matters: Engineers build brittle pipelines by optimizing individual components instead of the end-to-end user conversation experience.

Latency as a Budget vs. Humanlike Goal

  • Explanation: Distinguishing between raw component latency (like time to first byte) and the perceived latency experienced by callers.
  • Key claims:
    • Latency is a budget; humanlike behavior is the goal.
    • Dashboard metrics report time-to-first-byte, but bytes are not sound.
    • Front-loaded silence in TTS models can create up to 750ms of dead air.
  • Examples:
    • A dashboard says 210ms time-to-first-byte, but the user hears silence for 750ms.
  • Terminology:
    • Time to First Byte (TTFB)
    • Time to First Audio
    • Front-loaded silence
  • Why it matters: Optimizing purely for raw speed metrics without considering audio delivery creates unnatural, robotic pauses.

Token Metrics vs. Sentence Metrics

  • Explanation: Why text generation metrics like time-to-first-token fail to capture voice generation realities.
  • Key claims:
    • Tokens aren't sentences; words aren't equal.
    • TTS cannot start speaking on a single token; it needs an entire sentence.
    • Models that stream tokens the fastest do not finish sentences first.
  • Examples:
    • Streaming individual tokens vs complete sentences.
  • Terminology:
    • First Token
    • First Sentence
    • Token streaming
  • Why it matters: Voice agents require sentence-level pacing to sound natural rather than rushed or disjointed.

Async Tools and Conversational Pacing

  • Explanation: How humanlike agents handle long-running tool calls without freezing the conversation.
  • Key claims:
    • Real agents do things, and tools take time (2s, 10s, 40s).
    • Humans don't freeze while booking systems load; they buy time and keep listening.
    • Asynchronous tool execution allows agents to speak interim status updates while working in the background.
  • Examples:
    • Agent saying 'Searching flights. This will take a minute.' while executing async search.
  • Terminology:
    • Async Tool
    • Interim update
    • Conversational thread
  • Why it matters: Enables agents to maintain conversational flow during high-latency backend operations.

Turn-Taking and Audio-Based Detection

  • Explanation: Replacing transcript-based turn detectors with audio-based turn detection to eliminate false cutoffs.
  • Key claims:
    • Humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation.
    • Pipelines that wait for the entire transcript are too slow.
    • LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.
  • Examples:
    • LiveKit audio-based turn detection firing before transcript lands.
  • Terminology:
    • Turn Detector
    • Audio turn detection
    • Last syllable
    • First audible byte
  • Why it matters: Crucial for preventing awkward pauses or premature interruptions in live voice conversations.

Conversation-Level Benchmarking

  • Explanation: A new open-source benchmarking approach that tests complete working agents against simulated callers and outcomes.
  • Key claims:
    • There are as many benchmarks as there are agents.
    • Test the conversation, not the component.
    • Score outcomes (database bookings, correct names/dates) against human-equivalent behaviors.
  • Examples:
    • Running hundreds of simulated callers with mumbling, interruptions, and changing minds against hotel concierge agents.
  • Terminology:
    • Conversation benchmark
    • Outcome scoring
    • Simulated callers
  • Why it matters: Provides a realistic evaluation framework that reflects actual production performance.

Key Points

Errors flow downhill

  • Explanation: An error in the speech-to-text stage propagates through the LLM and TTS without downstream models knowing the input was flawed.
  • Evidence: STT mishearing '3rd' as '30th' causes LLM to plan the wrong date.
  • Practical implication: Component isolation hides systemic conversational failures.

Measured latency vs. Perceived latency

  • Explanation: Dashboards show fast time-to-first-byte numbers, but front-loaded silence leads to slow perceived latency for users.
  • Evidence: Up to 750ms of silence before caller hears audio despite 210ms TTFB.
  • Practical implication: Measure what the user hears, not just what the server outputs.

Async tool execution

  • Explanation: Tools take time; agents must buy time conversationally rather than freezing.
  • Evidence: Using async context updates to say 'Searching flights' while API calls run in the background.
  • Practical implication: Eliminates awkward silence during database lookups.

Frameworks, Models & Processes

Tone-Home Checklist for Voice Agents

  • How it works: A practical checklist for engineering humanlike real-time voice applications.
  • Components:
    • Set the latency budget first
    • Measure first audio and first sentence
    • Spend leftover budget on quality
    • Anything over a second goes async
    • Test conversations, then keep testing
  • When to use: When designing and deploying production-grade voice AI agents.

Examples & Case Studies

Caller books a hotel room for October 10-12, changes guests, asks for ocean view, names Jesse Hall, and later changes dates to December 20-24.

  • Illustrates: A complete real-time hotel concierge agent handling multi-turn updates, interruptions, and async tool execution.
  • Lesson: Agents must handle mid-conversation changes smoothly without awkward pauses.

Actionable Takeaways

  • Immediate:
    • Set a strict latency budget based on human perception windows.
    • Implement async tool execution to keep conversations alive during long API calls.
    • Use audio-based turn detection instead of transcript-based waiting.
  • Strategic:
    • Stop testing components in isolation; test end-to-end conversations.
    • Build or adopt neutral benchmarks that evaluate outcome success against human equivalents.
    • Treat voice agents as real-time sensory systems rather than chatbots with speakers.
  • Questions to investigate:
    • How does your current STT-LLM-TTS pipeline handle front-loaded silence?
    • What percentage of user call interruptions result in false cutoffs in your current agent?

Claims Worth Verifying

  • LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget. (Benchmark performance claim)
  • Gemma 4 31B tuned for voice starts speaking in about 380ms, passing voice evals at 88%. (Model performance claim)

Notable Quotes

"Latency is a budget. Humanlike is the goal." (at 2:07) "The leaderboard is a trap." (at 7:31) "A voice agent isn't a chatbot with a speaker attached." (at 25:30) "It's a realtime system that senses and responds like a person." (at 25:38)

Compressed Summary

  • Component leaderboards fail because errors flow downhill and ignore real-world audio conditions.
  • Latency is a budget bounded by human conversational expectations (~1.5s hold, ~600ms feel human).
  • Async tool execution allows voice agents to speak interim status updates while background work runs.
  • Audio-based turn detection reduces false cutoffs compared to transcript-based waiting.
  • Benchmarking must test complete working conversations and outcomes, not isolated components.
  • Keywords: latency, voice-agent, turn-detection, livekit, benchmarking
  • Core insight: Voice agents require real-time infrastructure optimized for humanlike conversational pacing and async tool handling rather than isolated component speed metrics.

Core insights

6
Mental Modelhigh noveltystrong evidence

Latency must be treated as a budget allocated across the whole conversation, not as a per-component speed metric: server-side time-to-first-byte can read 210ms while the caller experiences up to 750ms of dead air because TTS models front-load silence. The unit of measurement should shift from bytes emitted to audio the caller actually hears (time-to-first-audio).

Why it matters

Engineers who optimize TTFB dashboards will ship agents that feel slow while reporting fast, so the instrumentation itself is misleading. Perceived latency is the only metric a caller can observe, and it requires a different measurement point (first audible byte, not first byte).

Generalization

In any streaming pipeline with a rendering/consumption stage, instrument at the consumer boundary rather than at the producer boundary; intermediate bytes are not the delivered experience (applies to token streaming UIs, video, and audio alike).

Latency is a budget; humanlike behavior is the goal.
Open source video
Dashboard metrics report time-to-first-byte, but bytes are not sound.
Open source video
Front-loaded silence in TTS models can create up to 750ms of dead air.
Open source video
A dashboard says 210ms time-to-first-byte, but the user hears silence for 750ms.
Open source video
Empirical Resulthigh noveltystrong evidence

Audio-based turn detection can replace transcript-based turn detection: because humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation, waiting for a complete transcript is structurally too slow. LiveKit's audio detector fires before the transcript lands and reportedly cuts false cutoffs from 9.9% to 4.5% at the same latency budget.

Why it matters

Turn-taking is a separate, prosody-driven subsystem that should not be coupled to the ASR/transcript path; treating 'end of turn' as a transcript-completion event forces a latency floor and causes premature interruptions or awkward pauses.

Generalization

Control/scheduling decisions in real-time interactive systems should be driven by cheap, low-latency signals (prosody, partial features) rather than by waiting for the full semantic pipeline to complete.

Humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation.
Open source video
Pipelines that wait for the entire transcript are too slow.
Open source video
LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.
Open source video
LiveKit audio-based turn detection firing before transcript lands.
Open source video
Mechanismhigh noveltystrong evidence

Token-level generation metrics do not transfer to voice: TTS cannot begin speaking on a single token, it needs an entire sentence, so the fastest token-streaming model does not necessarily produce the first audible sentence soonest. The meaningful scheduling unit is the sentence, not the token.

Why it matters

Model and pipeline selection based on time-to-first-token will pick the wrong components for voice, and generation logic (buffering to sentence boundaries, chunking policy) becomes an architectural responsibility rather than an implementation detail.

Generalization

When a downstream consumer has a different natural unit than the upstream producer (tokens vs sentences, frames vs batches), optimize and measure at the consumer's unit; streaming granularity is a design parameter, not a constant.

Tokens aren't sentences; words aren't equal.
Open source video
TTS cannot start speaking on a single token; it needs an entire sentence.
Open source video
Models that stream tokens the fastest do not finish sentences first.
Open source video
Measure first audio and first sentence
Open source video
Architecturemedium noveltymoderate evidence

Long-running tool calls should be executed asynchronously while the agent speaks interim status updates, rather than blocking the conversation. Humans do not freeze while booking systems load; they buy time and keep listening, so the agent must be able to hold the conversational thread during 2s/10s/40s operations.

Why it matters

It decouples tool-execution latency from conversational latency, which removes the hard requirement that every backend operation fit inside the response budget and enables a heuristic like 'anything over a second goes async'.

Generalization

Separate the agent's conversational/response clock from its task-execution clock; long-running work belongs in a background channel with explicit progress narration, mirroring how humans handle slow processes.

Real agents do things, and tools take time (2s, 10s, 40s).
Open source video
Asynchronous tool execution allows agents to speak interim status updates while working in the background.
Open source video
Agent saying 'Searching flights. This will take a minute.' while executing async search.
Open source video
Anything over a second goes async
Open source video
Failure Modemedium noveltymoderate evidence

Errors flow downhill in component cascades: an STT misrecognition is silently consumed by the LLM, which plans an action on corrupted input with no downstream model able to detect that the input was wrong (e.g. '3rd' heard as '30th' causes the wrong booking date). Component leaderboards score models in clean lab conditions, not real-world mobile traffic, so independently top-ranked components do not compose into a working conversation.

Why it matters

It invalidates leaderboard-driven component selection as a design method and argues that validation, confidence propagation, and outcome checks must sit at the conversation level rather than at each component boundary.

Generalization

In composed pipelines, per-stage accuracy is not composable into end-to-end reliability; without cross-stage error signaling, an upstream defect is laundered into a confident downstream action.

Errors flow downhill in component cascades.
Open source video
Component leaderboards score models in clean lab conditions, not real-world mobile traffic.
Open source video
The leaderboard is a trap: measuring the model is not the same as shipping a conversation.
Open source video
STT hears 'Tuesday the 30th' instead of 'Tuesday the 3rd', causing downstream LLM to book the wrong date.
Open source video
Practicehigh noveltymoderate evidence

Evaluation should be conversation-level and outcome-scored: run hundreds of simulated callers who mumble, interrupt, and change their minds against a complete working agent, then score end-state effects (database bookings, correct names and dates) rather than component metrics. The hotel concierge case (October 10-12 booking, guest change, ocean view, name, then a change to December 20-24) is the unit of test, not the ASR or LLM.

Why it matters

It provides a testable harness design: the observable is the side effect in the system of record, the workload is adversarial-but-humanlike caller behavior, and it explicitly rejects the 'one benchmark per component' trap.

Generalization

For agents that act on external systems, benchmark by scoring verified side effects under adversarial input variation, not by scoring intermediate model outputs.

Test the conversation, not the component.
Open source video
Score outcomes (database bookings, correct names/dates) against human-equivalent behaviors.
Open source video
Running hundreds of simulated callers with mumbling, interruptions, and changing minds against hotel concierge agents.
Open source video
There are as many benchmarks as there are agents.
Open source video

Deep dives

5

Perceived-latency instrumentation for streaming voice

Research question

Where must latency be measured so dashboards reflect the caller's experienced delay rather than server-side byte emission, and how large is the gap between TTFB and time-to-first-audio in production TTS?

Why

Prevents teams from optimizing a misleading metric and makes front-loaded silence visible instead of hidden.

Latency is a budget; humanlike behavior is the goal.
Open source video
Dashboard metrics report time-to-first-byte, but bytes are not sound.
Open source video
Front-loaded silence in TTS models can create up to 750ms of dead air.
Open source video
A dashboard says 210ms time-to-first-byte, but the user hears silence for 750ms.
Open source video
Source video

Audio-based turn detection generalization and residual error

Research question

Does audio/prosody-based turn detection preserve false-cutoff reductions across languages, accents, and mobile codecs at equal latency budgets?

Why

Turn-taking is a separate subsystem; the 9.9% to 4.5% claim needs replication conditions before it can drive architecture.

Humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation.
Open source video
Pipelines that wait for the entire transcript are too slow.
Open source video
LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.
Open source video
LiveKit audio-based turn detection firing before transcript lands.
Open source video
Source video

Sentence-level scheduling and first-sentence latency

Research question

How should generation and TTS buffering be scheduled so that first-sentence latency, not time-to-first-token, governs when audio starts?

Why

Determines component selection and chunking policy; token metrics mis-rank voice models and mislead pipeline design.

Tokens aren't sentences; words aren't equal.
Open source video
TTS cannot start speaking on a single token; it needs an entire sentence.
Open source video
Models that stream tokens the fastest do not finish sentences first.
Open source video
Measure first audio and first sentence
Open source video
Source video

Async tool lifecycle under caller mind changes

Research question

What cancellation, supersession, and reconciliation semantics are needed when a caller changes a request while a long-running tool call is still executing?

Why

Async tools remove backend latency from conversation but create stale-result and rollback risks that must be designed explicitly.

Real agents do things, and tools take time (2s, 10s, 40s).
Open source video
Asynchronous tool execution allows agents to speak interim status updates while working in the background.
Open source video
Agents must handle mid-conversation changes smoothly without awkward pauses.
Open source video
Anything over a second goes async
Open source video
Source video

Conversation-level outcome benchmarking

Research question

Do simulated-caller, outcome-scored conversation benchmarks predict production success better than component leaderboards?

Why

Provides a falsifiable alternative to component metrics by testing verified side effects under adversarial caller behavior.

Test the conversation, not the component.
Open source video
Score outcomes (database bookings, correct names/dates) against human-equivalent behaviors.
Open source video
Running hundreds of simulated callers with mumbling, interruptions, and changing minds against hotel concierge agents.
Open source video
There are as many benchmarks as there are agents.
Open source video
Source video

Article ideas

4

Your Voice Latency Dashboard Is Measuring the Wrong Byte

The only latency that matters in a voice agent is time-to-first-audio and first-sentence, because TTFB and token metrics hide front-loaded silence; teams should instrument at the caller's ear and allocate a conversation-level budget.

Angle

Instrumentation critique and budget reallocation

Source video

The Component Leaderboard Trap: Why Top Models Do Not a Conversation Make

Independent STT/LLM/TTS rankings are not composable; silent upstream errors flow downhill, so reliability must be evaluated at conversation outcome level rather than per-component benchmarks.

Angle

Failure-mode analysis of cascade design

Source video

Sentences, Not Tokens: The Real Streaming Unit for Humanlike Voice

Optimizing time-to-first-token picks the wrong models for voice because TTS needs sentence boundaries; first-sentence latency should drive generation chunking and model selection.

Angle

Streaming granularity as architecture

Source video

Async Tools Are a Turn-Taking Strategy, Not Just a Backend Optimization

Long-running tool calls should run off the conversational clock with narrated interim status, but this introduces stale-result lifecycle management that must be designed explicitly.

Angle

Turn management meets distributed operations

Source video

Project ideas

4

First-Audio Budget Profiler

movement-lab

For voice-agent TTS models, time-to-first-audio and first-sentence latency rank models differently from time-to-first-byte and time-to-first-token, and front-loaded silence accounts for a measurable share of perceived dead air.

Proof of concept

Instrument a LiveKit-style pipeline with multiple TTS/LLM configurations, timestamping byte emission, first audible sample, and first complete sentence across mobile-like network jitter.

Measurement

Delta between TTFB and time-to-first-audio; rank correlation between TTFB/first-token and time-to-first-audio/first-sentence; dead-air milliseconds per configuration.

Source video

Robust Turn-Detection Replication Harness

new

Audio/prosody-based turn detection maintains a false-cutoff advantage over transcript-complete detection across accents, languages, and noisy mobile audio at matched latency budgets.

Proof of concept

Build a replay harness over noisy, accented, codec-degraded dialogue with ground-truth turn boundaries; compare audio-based detector against transcript-completion baseline at equal latency.

Measurement

False cutoffs, false holds, turn-transition latency, and error rates segmented by language, accent, and SNR.

Source video

Async Tool Lifecycle Simulator

gatehouse

Supersede/cancel/commit semantics for in-flight tool calls reduce incorrect side effects when simulated callers change requests mid-operation, compared with blocking or fire-and-forget execution.

Proof of concept

Simulate a hotel booking agent with 2s/10s/40s tools and callers who change dates or guest counts; implement cancellable tool operations with versioned requests and result reconciliation.

Measurement

Wrong bookings per 100 conversations, stale-result commits, mean conversational silence, and recovery rate after caller change.

Source video

Outcome-Scored Conversation Benchmark

beyond-evals

Outcome-scored simulated-caller benchmarks correlate with real deployment success better than component leaderboard scores for the same agents.

Proof of concept

Create a harness that runs hundreds of adversarial simulated callers against a working concierge agent and scores verified database end states, then compare against component benchmark rankings.

Measurement

Spearman correlation between benchmark outcome scores and production success rates; error attribution to upstream STT vs LLM vs policy.

Source video

Architectural implications

4

The latency budget is set at the conversation level and then allocated downward, implying an end-to-end scheduler/controller rather than independently tuned components.

Before

Pick the top-ranked STT, LLM, and TTS from component leaderboards and wire them into a cascade.

After

Define a conversational latency budget from human perception windows, measure first audio and first sentence, and spend the leftover budget on quality.

Consequence

Model selection becomes a constrained allocation problem against a shared budget, and per-component optimization loses authority over the design.

Source video

Turn detection moves off the transcript path onto the audio/prosody path.

Before

Wait for the full transcript to decide the user has finished their turn.

After

Fire turn decisions on audio signals (tone, pitch, intonation, last syllable) before the transcript lands.

Consequence

Turn-taking becomes an independent low-latency subsystem, decoupling responsiveness from ASR completion time and reducing false cutoffs.

Source video

Tool execution is moved off the conversational critical path into a background channel with narration.

Before

Block the response loop while the tool call runs, producing dead air.

After

Launch async tools and emit interim status speech ('Searching flights. This will take a minute.') while the work proceeds.

Consequence

The agent's response clock no longer bounds backend latency; it instead requires lifecycle management for in-flight operations and their results.

Source video

Observability instrumentation moves from server-emitted bytes to caller-perceived audio and sentence-level completion.

Before

Dashboard reports time-to-first-byte per component.

After

Dashboard reports time-to-first-audio and first-sentence latency for the whole conversation.

Consequence

The metric set changes what engineers can see and therefore what they optimize; front-loaded silence becomes visible instead of hidden.

Source video

Tradeoffs and failure modes

4

Asynchronous tool execution

Benefit

Keeps the conversational thread alive during 2s/10s/40s backend operations, eliminating awkward silence during database lookups.

Cost or risk

In-flight operations can be invalidated by the caller changing their mind mid-flight (as in the hotel example where dates change from October 10-12 to December 20-24), requiring cancellation/reconciliation of stale results.

Agents must handle mid-conversation changes smoothly without awkward pauses.
Open source video
Source video

Audio-based turn detection

Benefit

Reduces false cutoffs from 9.9% to 4.5% at the same latency budget.

Cost or risk

Residual 4.5% false-cutoff rate remains, and detection fires before the transcript is available, so semantic content cannot be used to confirm the decision.

LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.
Open source video
Source video

Sentence-level TTS buffering

Benefit

Enables natural pacing and the first audible sentence sooner.

Cost or risk

Requires aggregating tokens to sentence boundaries, which constrains chunking policy and can delay output when generation is slow to complete a sentence.

TTS cannot start speaking on a single token; it needs an entire sentence.
Open source video
Source video

Component-level optimization / leaderboards

Benefit

Cheap, standardized comparison of individual models in controlled conditions.

Cost or risk

Lab-condition scores do not reflect real-world mobile traffic, and errors flow downhill silently, so top-ranked components can compose into a conversation that fails.

The leaderboard is a trap: measuring the model is not the same as shipping a conversation.
Open source video
Source video

Open questions

4

What is the actual numeric latency budget, and how should it be split across turn detection, STT, LLM, and TTS?

Why unresolved

The summary states 'set the latency budget first' and cites human perception windows and the 750ms/210ms example, but no explicit budget breakdown or target numbers are given.

Research direction

Derive a perception-based budget from first-audio and first-sentence targets, then measure the per-stage contribution to that budget in a production voice pipeline.

Source video

How should results from async tools be reconciled when the caller changes the request while the tool is still running?

Why unresolved

The hotel example shows dates and guest counts changing mid-conversation while tools execute, but the summary does not describe cancellation, stale-result, or rollback semantics.

Research direction

Define an operation lifecycle (cancel, supersede, commit) for in-flight tool calls and test it under adversarial simulated callers who change their minds.

Source video

Does the 9.9% to 4.5% false-cutoff result generalize beyond the tested conditions (languages, accents, noisy mobile audio)?

Why unresolved

The claim is reported as a single comparison at the same latency budget with no stated evaluation conditions or dataset.

Research direction

Replicate audio-based turn detection across languages, accents, and mobile codecs, reporting false-cutoff and false-hold rates against the same latency budget.

Source video

Does conversation-level outcome scoring predict production performance, and is a shared benchmark suite needed?

Why unresolved

The claim 'there are as many benchmarks as there are agents' implies fragmentation, and the summary gives no evidence of correlation between simulated-caller scores and real deployments.

Research direction

Build a reusable simulated-caller harness with outcome scoring and compare benchmark results against real deployment metrics for the same agents.

Source video

Key claims

8
factualVerification needed

Front-loaded silence in TTS models can create up to 750ms of dead air even when time-to-first-byte is 210ms.

Evidence

A dashboard says 210ms time-to-first-byte, but the user hears silence for 750ms.

Question

Which TTS models exhibit how much front-loaded silence, and is it stable across voices, text, and deployment environments?

Source video
comparativeVerification needed

Audio-based turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.

Evidence

LiveKit turn detection reduces false cutoffs from 9.9% to 4.5% at the same latency budget.

Question

What dataset, languages, and acoustic conditions produce this comparison, and is the latency budget identical across both configurations?

Source video
causalVerification needed

TTS cannot start speaking on a single token; it requires an entire sentence, so the fastest token-streaming model need not produce the first spoken sentence first.

Evidence

Models that stream tokens the fastest do not finish sentences first.

Question

Across current TTS architectures, does time-to-first-sentence rank-order models differently than time-to-first-token?

Source video
causalVerification needed

In component cascades, upstream errors propagate unnoticed, so a downstream model acts confidently on corrupted input.

Evidence

STT hears 'Tuesday the 30th' instead of 'Tuesday the 3rd', causing downstream LLM to book the wrong date.

Question

What fraction of end-to-end booking errors trace to a single upstream misrecognition rather than to LLM reasoning failures?

Source video
factualVerification needed

Component leaderboards score models in clean lab conditions rather than real-world mobile traffic.

Evidence

Component leaderboards score models in clean lab conditions, not real-world mobile traffic.

Question

How much does component ranking change when evaluated on real mobile audio and network conditions versus benchmark sets?

Source video
factualVerification needed

Humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation, making transcript-complete turn detection too slow.

Evidence

Humans negotiate turn-taking in hundreds of milliseconds using tone, pitch, and intonation.

Question

What are the measured human turn-transition timings across languages and conversational contexts?

Source video
causalVerification needed

Agents that speak interim status while running async tools maintain conversational flow during long backend operations.

Evidence

Asynchronous tool execution allows agents to speak interim status updates while working in the background.

Question

Do interim status updates measurably improve caller satisfaction or task completion versus silent waiting, and at what verbosity do they become annoying?

Source video
comparativeVerification needed

Conversation-level benchmarks with simulated callers and outcome scoring better reflect production performance than component benchmarks.

Evidence

Test the conversation, not the component.

Question

How well do simulated-caller outcome scores correlate with real deployment success rates for the same agents?

Source video

Connections

5