AI Engineer · Published 2026-09-15

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

privateReview: unreviewedConfidence: high
voice agentlatencytranscriptionTurn detectionnormalizationvoice-agentsttfastt-wertts-normalizationturn-detectionvadstructured-extraction
Open on YouTube ↗

Summary

Overview

  • Speaker: Venky B
  • Channel: AI Engineer
  • Main topic: Voice Agent Failure Modes and Solutions
  • Purpose: To educate AI engineers on the practical production hurdles of building real-time voice AI agents and provide architectural strategies to mitigate latency, transcription errors, structured data collection issues, text-to-speech input formatting, and turn-taking failures. Venky B, founder and COO of Plivo, discusses five critical failure modes encountered when deploying voice AI agents from proof of concept to production. Drawing on Plivo's 14 years in telephony and handling over a billion voice calls monthly, Venky breaks down latency, brittle transcription, data collection challenges, input normalization to TTS, and turn detection/barge-in handling, offering concrete engineering strategies for each.

Topic Map

Introduction and Plivo Background

  • Explanation: Venky introduces himself as the Chief Agent Officer of Plivo, a company with 14 years in the developer API business, processing over a billion voice calls per month across a 90-person team.
  • Key claims:
    • Voice agents sound great in development but fail when moved to production.
    • Plivo provides programmable voice, an AI agent studio, and underlying SIP trunking and audio streaming infrastructure.
  • Examples:
    • Moving a voice agent proof-of-concept to production breaks under real-world conditions.
  • Terminology:
    • voice AI agents
    • SIP trunking
    • audio streaming
    • AI agent studio
  • Why it matters: Establishes credibility based on massive scale telemetry and telephony infrastructure experience.

Failure Mode 1: Latency

  • Explanation: Examines the latency budget of the typical voice AI pipeline (Telephony -> STT -> Turn Detection -> LLM -> TTS), noting the time-to-first-audio (TTFA) threshold and tradeoffs between cost, intelligence, and latency.
  • Key claims:
    • TTFA under 550ms feels natural/good; 750ms-1.2s causes conversation drag; over 1.2s causes users to hang up.
    • Frontier models (OpenAI, Claude, Gemini) have high TTFA (450-500ms) unless using dedicated capacity (Cerebras, Groq).
    • Open-source models like Qwen 3.5 and Gemma 4 offer good speed-intelligence tradeoffs, while mixture-of-experts (MoE) models need careful sizing.
  • Examples:
    • Using frontier models vs. specialized high-speed inference accelerators like Cerebras and Groq.
  • Terminology:
    • TTFA (Time to First Audio)
    • STT
    • TTS
    • turn detection
    • mixture of experts
    • frontier models
  • Why it matters: Latency is the primary reason users abandon voice agents in production.

Failure Mode 2: Brittle Transcription

  • Explanation: Discusses why speech-to-text (STT) engines fail on real-world audio with accents, background noise, domain vocabulary, proper nouns, phone numbers, addresses, and code-switched languages.
  • Key claims:
    • Best STTs achieve 4-6% Word Error Rate (WER) on clean eval sets, but real-world calls perform much worse due to accent and noise.
    • Proper nouns, jargons, phone numbers, and code-switching break downstream LLM understanding.
    • Dynamic keyword/phrase boosting and post-processing via LLM domain context are essential.
  • Examples:
    • Hindi spoken in Devanagari script vs. Roman script leading to transcription and translation errors.
    • Phone number digit transcription errors like '415 e 6942 won 0'.
  • Terminology:
    • WER (Word Error Rate)
    • STT transcription
    • code-switching
    • dynamic keyword boosting
    • transliteration
    • post-processing
  • Why it matters: If transcription is wrong, the entire downstream agent response fails.

Failure Mode 3: Collecting Data

  • Explanation: Explores structured data extraction from unstructured voice conversations, drawing parallels to frontend form fields, Pydantic, Zod, and data classes.
  • Key claims:
    • Treat voice data collection like a typed schema UX problem rather than hoping an LLM parses transcripts correctly.
    • Constrained capture (defining required fields, types, validation rules, allowed values) increases extraction accuracy from 30% to 95%.
    • Relative date values (e.g., 'next week Wednesday at 8') require normalization and confirmation mechanisms.
  • Examples:
    • Capturing complex names like 'Aleksandr Szymonowicz' or relative date-time values.
    • Using unit tests at a field level to ensure extraction reliability.
  • Terminology:
    • structured data extraction
    • Pydantic
    • Zod
    • dataclass
    • constrained capture
    • unit testing
  • Why it matters: Open-ended prompt engineering for data collection results in hallucinated or malformed data.

Failure Mode 4: Input to TTS

  • Explanation: Examines issues when sending raw LLM output directly to Text-to-Speech (TTS) engines without a normalization layer.
  • Key claims:
    • LLM output should never go to TTS directly; a separate normalization layer is required.
    • Emojis and markdown must be stripped before synthesis.
    • Custom pronunciation dictionaries and speed control (e.g., slowing down to 0.8x or 0.7x for names and numbers) prevent pronunciation failures.
  • Examples:
    • An agent failing to pronounce difficult names like 'Balasubramanian' or company names like 'Plivo' correctly.
    • Normalizing messy stuff like emails, currencies, dates, and numbers.
  • Terminology:
    • TTS synthesis
    • normalization layer
    • pronunciation dictionary
    • prosody
    • speed scaling
  • Why it matters: Poor pronunciation ruins user trust immediately ('if your agent can't pronounce your name, you've already dropped the ball').

Failure Mode 5: Turn Detection and Barge-in

  • Explanation: Covers end-of-turn detection challenges (VAD vs semantic vs acoustic models) and handling interruptions/backchannels without breaking conversational flow.
  • Key claims:
    • VAD/silence-based turn detection fires on any pause, cutting users off mid-thought.
    • Semantic and acoustic models predict end-of-turn from words said and prosody (intonation, pitch, rhythm).
    • Backchanneling ('uh-huh', 'mhm') requires classifying interruptions to know whether to stop or keep going.
  • Examples:
    • VAD treating brief pauses or backchannels ('uh-huh', 'okay') as complete interruptions.
    • Managing walkie-talkie-style vs conversational backchanneling.
  • Terminology:
    • turn detection
    • VAD (Voice Activity Detection)
    • semantic turn detection
    • backchanneling
    • barge-in
    • prosody
  • Why it matters: Natural conversation pacing depends on accurate turn-taking and backchannel management.

Key Points

Latency Budget Management

  • Explanation: Time-to-first-audio (TTFA) must remain under 550ms for natural conversation; anything over 1.2s causes hang-ups.
  • Evidence: Customer telemetry across over a billion calls on Plivo's platform.
  • Practical implication: Engineers must carefully choose between frontier model intelligence and high-speed inference options like Groq, Cerebras, or smaller open-source models.

Transcription is Brittle

  • Explanation: Even the best STT engines have 4-6% WER on clean data, which degrades significantly with noise, accents, and code-switching.
  • Evidence: Real-world calls involve proper nouns, domain jargon, phone numbers, and multilingual code-switching.
  • Practical implication: Implement dynamic keyword boosting and intelligent LLM-based post-processing layers.

Constrained Data Collection

  • Explanation: Treating voice data collection like a typed schema UX problem drastically improves accuracy.
  • Evidence: Accuracy increases from 30% to 95% when moving from open-ended extraction to typed schema validation (Pydantic, Zod).
  • Practical implication: Define schemas, required fields, and allowed values before asking data from users.

TTS Normalization Layer

  • Explanation: LLM text output cannot be piped directly into TTS without normalization.
  • Evidence: Unfiltered output contains emojis, markdown, unformatted numbers, and mispronounced names.
  • Practical implication: Add a normalization layer to strip markdown/emojis, apply pronunciation dictionaries, and adjust speech rates for names and numbers.

Advanced Turn Detection

  • Explanation: Basic VAD fails by interrupting users on pauses or missing backchannels.
  • Evidence: Semantic and acoustic models are required to analyze prosody and distinguish backchannels from true interruptions.
  • Practical implication: Classify user interruptions to determine whether to stop speaking or continue with backchannels.

Frameworks, Models & Processes

Typical Voice AI Agent Pipeline

  • How it works: Audio passes from Telephony to Speech-to-Text (STT), then through Turn Detection to an LLM, and finally converted back to audio via Text-to-Speech (TTS).
  • Components:
    • Telephony
    • STT (Speech-to-Text)
    • Turn Detection
    • LLM
    • TTS (Text-to-Speech)
  • When to use: As the foundational architecture for any real-time conversational voice agent.

LLM Trilemma for Voice: Latency, Cost, and Intelligence

  • How it works: Balances the trade-offs between model intelligence, inference latency (TTFA), and hosting or API costs.
  • Components:
    • Latency
    • Cost
    • Intelligence
  • When to use: When selecting and benchmarking LLMs for voice agent pipelines.

Examples & Case Studies

An STT engine transcribes a phone number digit sequence incorrectly (e.g., '415 e 6942 won 0').

  • Illustrates: Failure Mode 2: Brittle transcription on numbers and proper nouns.
  • Lesson: Use intelligent post-processing and LLM domain context to fix transcription output before downstream logic.

A voice agent fails to correctly pronounce complex names like 'Aleksandr Szymonowicz' or the company name 'Plivo' ('Plee-voh').

  • Illustrates: Failure Mode 4: Input to TTS without normalization or pronunciation dictionaries.
  • Lesson: Use custom pronunciation dictionaries, slow down speech rates for specific entities, and implement a normalization layer.

A voice agent's VAD triggers on a brief conversational pause or backchannel ('uh-huh', 'mhm'), cutting off the user mid-thought.

  • Illustrates: Failure Mode 5a/5b: End-of-turn detection and backchanneling failures.
  • Lesson: Adopt semantic and acoustic turn detection models that analyze prosody and classify interruptions correctly.

Actionable Takeaways

  • Immediate:
    • Never pipe raw LLM output directly into a TTS engine; always use a normalization layer.
    • Strip emojis and markdown from text before TTS synthesis.
    • Implement dynamic keyword boosting in your STT engine for domain-specific vocabulary.
  • Strategic:
    • Treat voice data collection like a typed form schema (Pydantic, Zod) rather than relying on unconstrained prompts.
    • Balance the LLM trilemma (latency, cost, intelligence) by choosing appropriate model sizes or specialized hardware.
    • Move beyond basic VAD by incorporating semantic and acoustic prosody analysis for turn detection.
  • Questions to investigate:
    • How does your current voice agent pipeline handle TTFA latency under load?
    • Are you using typed schemas for data extraction in your voice conversations?
    • How does your system distinguish between a backchannel and a true barge-in interruption?

Claims Worth Verifying

  • Best STT engines achieve 4-6% Word Error Rate (WER) on clean, known evaluation sets. (Benchmark statistic)
  • Frontier models (OpenAI, Claude, Gemini) have TTFA times around 450-500ms. (Technical metric)
  • Structured schema-based data extraction increases accuracy from 30% to 95%. (Performance metric)

Notable Quotes

"If your agent can't pronounce my last name or my company's name, you've already dropped the ball." (at 40:37) ""Silence = done." Knowing when the user has actually finished is the hard part." (at 42:07)

Compressed Summary

  • Keep TTFA latency under 550ms for natural conversation flow.
  • Mitigate brittle STT transcription with dynamic keyword boosting and LLM post-processing.
  • Use typed schemas (Pydantic/Zod) for reliable structured data collection from voice.
  • Always normalize LLM text output (strip markdown/emojis, apply pronunciation dictionaries) before TTS.
  • Upgrade from basic VAD to semantic and acoustic turn detection to handle barge-ins and backchannels.
  • Keywords: voice agent, latency, transcription, turn detection, normalization
  • Core insight: Deploying production-ready voice AI agents requires solving five critical engineering failure modes: latency budgets, brittle STT transcription, unstructured data collection, TTS input normalization, and semantic turn detection.

Core insights

6
Architecturehigh noveltymoderate evidence

Voice data collection should be architected as a typed-schema UX problem rather than an open-ended prompting problem: define required fields, types, validation rules, and allowed values (Pydantic/Zod/dataclass-style), which reportedly lifts extraction accuracy from 30% to 95%.

Why it matters

Shifts responsibility for reliable capture from the LLM's free-form reasoning to an explicit schema contract, making extraction testable, predictable, and less prone to hallucinated or malformed fields.

Generalization

Applies to any agentic system extracting structured entities from unstructured input: constrain the output space and validate at field level rather than trusting the model to infer the schema.

Treat voice data collection like a typed schema UX problem rather than hoping an LLM parses transcripts correctly.
Open source video
Constrained capture (defining required fields, types, validation rules, allowed values) increases extraction accuracy from 30% to 95%.
Open source video
Empirical Resultmedium noveltymoderate evidence

Time-to-first-audio (TTFA) has concrete experiential thresholds — under 550ms feels natural, 750ms-1.2s produces conversational drag, and over 1.2s causes users to hang up — which turns latency into a hard cross-component design budget rather than a soft quality goal.

Why it matters

Gives engineers a measurable target to allocate across telephony, STT, turn detection, LLM, and TTS, and a decision boundary for choosing model/inference providers.

Generalization

In any real-time interactive AI pipeline, perceived quality thresholds should be defined in user-facing latency terms and used to budget each pipeline stage.

TTFA under 550ms feels natural/good; 750ms-1.2s causes conversation drag; over 1.2s causes users to hang up.
Open source video
Architecturemedium noveltymoderate evidence

LLM output must never be sent directly to TTS; a distinct normalization layer is required to strip emojis/markdown and to apply pronunciation dictionaries and speed control (e.g., 0.7x-0.8x for names and numbers).

Why it matters

Introduces an explicit boundary between reasoning output and speech synthesis, preventing the LLM from being responsible for surface-level rendering concerns it handles poorly.

Generalization

Generalizable as a 'presentation/normalization layer' pattern: model output should pass through a deterministic transforming layer before reaching any consumer with strict formatting constraints.

LLM output should never go to TTS directly; a separate normalization layer is required.
Open source video
Emojis and markdown must be stripped before synthesis.
Open source video
Custom pronunciation dictionaries and speed control (e.g., slowing down to 0.8x or 0.7x for names and numbers) prevent pronunciation failures.
Open source video
Mechanismhigh noveltymoderate evidence

End-of-turn detection should be a model-based classification problem (semantic and acoustic signals including prosody) rather than silence-based VAD, and must further classify interruptions to distinguish backchannels from true barge-ins.

Why it matters

Replaces a naive heuristic that cuts users off mid-thought with a prediction problem, and adds a second classification decision (stop vs. continue) that governs conversational flow.

Generalization

Applies to any streaming interaction system where 'is the input complete?' is probabilistic and where interruption semantics matter (e.g., streaming transcription, multi-agent handoff).

VAD/silence-based turn detection fires on any pause, cutting users off mid-thought.
Open source video
Semantic and acoustic models predict end-of-turn from words said and prosody (intonation, pitch, rhythm).
Open source video
Backchanneling ('uh-huh', 'mhm') requires classifying interruptions to know whether to stop or keep going.
Open source video
Failure Modemedium noveltymoderate evidence

Benchmark STT accuracy (4-6% WER on clean eval sets) is not predictive of production behavior; accents, noise, proper nouns, jargon, phone numbers, and code-switching degrade it, requiring dynamic keyword/phrase boosting and LLM post-processing with domain context.

Why it matters

Any downstream agent reasoning is bounded by transcription fidelity, so eval methodology must include realistic audio conditions and a corrective post-processing stage.

Generalization

Clean-set metrics routinely overstate production reliability; systems should assume a gap between benchmark and field conditions and add compensating layers.

Best STTs achieve 4-6% Word Error Rate (WER) on clean eval sets, but real-world calls perform much worse due to accent and noise.
Open source video
Dynamic keyword/phrase boosting and post-processing via LLM domain context are essential.
Open source video
Practicemedium noveltyweak evidence

Structured extraction reliability can be validated with field-level unit tests, treating each captured field as a testable unit rather than validating end-to-end conversation outcomes.

Why it matters

Enables deterministic, fast regression testing of extraction behavior across transcripts without full conversation replay, and localizes failures to specific fields.

Generalization

Field-level (or slot-level) unit testing is a generalizable evaluation pattern for any schema-constrained extraction agent.

Using unit tests at a field level to ensure extraction reliability.
Open source video

Deep dives

5

Per-stage TTFA budget allocation in voice agent pipelines

Research question

How should the end-to-end TTFA budget be allocated across telephony, STT, turn detection, LLM, and TTS stages, and how does optimal allocation shift when swapping inference backends (frontier models vs. dedicated accelerators vs. MoE)?

Why

End-to-end TTFA thresholds (under 550ms natural, over 1.2s hang-up) are actionable only if engineers can attribute latency to specific components and predict the effect of provider changes.

TTFA under 550ms feels natural/good; 750ms-1.2s causes conversation drag; over 1.2s causes users to hang up.
Open source video
Frontier models (OpenAI, Claude, Gemini) have high TTFA (450-500ms) unless using dedicated capacity (Cerebras, Groq).
Open source video
Source video

Schema-constrained extraction for voice data collection

Research question

To what extent does defining a typed schema (fields, types, validation rules, allowed values) with field-level unit tests improve extraction accuracy across domains and languages, compared to open-ended prompting?

Why

The claimed 30%→95% accuracy lift, if reproducible, redefines best practice for any agentic system extracting structured entities from unstructured input.

Treat voice data collection like a typed schema UX problem rather than hoping an LLM parses transcripts correctly.
Open source video
Constrained capture (defining required fields, types, validation rules, allowed values) increases extraction accuracy from 30% to 95%.
Open source video
Source video

Model-based turn detection and interruption classification

Research question

What classifier architecture and feature set (prosodic, lexical, temporal) best distinguishes backchannels from true barge-ins, and how does classification accuracy affect conversational flow in real-time voice agents?

Why

Replacing silence-based VAD with probabilistic end-of-turn prediction and interruption classification is necessary to avoid cutting users off and to handle backchanneling naturally.

VAD/silence-based turn detection fires on any pause, cutting users off mid-thought.
Open source video
Semantic and acoustic models predict end-of-turn from words said and prosody (intonation, pitch, rhythm).
Open source video
Backchanneling ('uh-huh', 'mhm') requires classifying interruptions to know whether to stop or keep going.
Open source video
Source video

Bridging the STT benchmark-to-production gap

Research question

How can evaluation corpora be constructed to predict production WER under accents, noise, code-switching, and domain vocabulary, and how much do keyword boosting plus LLM post-processing recover downstream task accuracy?

Why

Clean-set WER (4-6%) overstates reliability; without stratified evaluation and compensating layers, downstream agent reasoning is bounded by silent transcription failures.

Best STTs achieve 4-6% Word Error Rate (WER) on clean eval sets, but real-world calls perform much worse due to accent and noise.
Open source video
Dynamic keyword/phrase boosting and post-processing via LLM domain context are essential.
Open source video
Source video

Normalization layer between LLM and TTS

Research question

What deterministic normalization rules (emoji/markdown stripping, pronunciation dictionaries, speed scaling) are necessary before TTS, and how should this layer be tested independently of conversation quality?

Why

Treating LLM output as raw TTS input causes immediate trust loss; a dedicated normalization layer localizes failures and makes formatting concerns testable.

LLM output should never go to TTS directly; a separate normalization layer is required.
Open source video
Emojis and markdown must be stripped before synthesis.
Open source video
Custom pronunciation dictionaries and speed control (e.g., slowing down to 0.8x or 0.7x for names and numbers) prevent pronunciation failures.
Open source video
Source video

Article ideas

4

Your Voice Agent's Latency Budget Is a Lie

End-to-end TTFA thresholds are useless without per-stage instrumentation; teams should treat latency as a distributed systems problem, attributing milliseconds to telephony, STT, turn detection, LLM, and TTS, and re-budgeting when swapping inference backends.

Angle

Practical engineering guide to instrumenting and allocating a cross-component latency budget, with concrete numbers from the talk and a call to publish per-stage benchmarks.

Source video

Stop Prompting, Start Schematizing: The 30% to 95% Extraction Fix

Open-ended prompting for structured data extraction is a category error; treating voice data collection as a typed schema UX problem with field-level unit tests reliably lifts extraction accuracy from 30% to 95% and makes failures localizable.

Angle

Contrasts prompt-engineering folklore with type-system discipline, showing how Pydantic/Zod-style contracts and field-level tests change the game.

Source video

VAD Is Not a Turn Detector

Silence-based VAD is the wrong abstraction for conversational turn-taking; semantic and acoustic models plus interruption classification are required to avoid cutting users off and to handle backchannels correctly.

Angle

Explains why the simple heuristic fails, presents a model-based alternative, and argues for evaluating turn-taking as a probabilistic inference problem.

Source video

Your STT Benchmark Is Lying to You

Clean-set WER (4-6%) is not predictive of production performance; teams must build stratified eval corpora and add keyword boosting plus LLM post-processing to recover downstream task accuracy.

Angle

Exposes the benchmark-production gap with numbers from the talk, then outlines a concrete evaluation and mitigation strategy.

Source video

Project ideas

4

TTFA Stage Budget Analyzer

new

By instrumenting each pipeline stage (telephony, STT, turn detection, LLM, TTS) and swapping inference backends, teams can predict and reduce end-to-end TTFA below 550ms without changing model intelligence.

Proof of concept

Build a reference voice agent pipeline with per-stage timers; run the same conversation scripts across frontier model (OpenAI/Claude/Gemini), dedicated accelerators (Cerebras/Groq), and open-source MoE (Qwen/Gemma); report per-stage latency contributions and total TTFA.

Measurement

Per-stage latency distributions and total TTFA across configurations; correlation between stage reductions and simulated user hang-up rates (e.g., timeout logs).

Source video

Schema-Forge: Field-Level Extraction Test Harness

beyond-evals

Defining a typed schema with validation rules and running field-level unit tests on transcribed conversations will achieve >90% field extraction accuracy and localize failures to specific fields, outperforming end-to-end conversation-level evaluation.

Proof of concept

For appointment scheduling, define a Pydantic/Zod schema for required fields (name, date, time, phone); create a test set of 100 transcripts with ground truth; compare open-ended prompt extraction vs. schema-constrained extraction with field-level validation.

Measurement

Per-field precision/recall/F1, overall extraction accuracy, and number of fields requiring manual fallback.

Source video

Turn-Taking Classifier Playground

new

A classifier using prosodic features plus partial transcript tokens can distinguish backchannels from true barge-ins with >85% accuracy, and using it to gate agent stopping behavior reduces perceived interruptions in simulated conversations.

Proof of concept

Collect or simulate call audio with labeled interruptions (backchannel vs barge-in); extract prosody (pitch, energy, rhythm) and lexical features; train a classifier; integrate into a turn-detection module and measure stop/continue decisions against human labels.

Measurement

Classification accuracy, F1, and false-positive/negative rates; user-perceived interruption score in a Wizard-of-Oz study or simulated user hang-up rate.

Source video

STT Stress-Test Harness

beyond-evals

A stratified evaluation corpus with accents, noise, code-switching, and domain vocabulary will show WER 3-5x higher than clean-set benchmarks, and dynamic keyword boosting plus LLM post-processing will recover a significant portion of downstream task accuracy.

Proof of concept

Assemble a multilingual, noise-augmented call corpus; measure WER for baseline STT vs keyword-boosted vs LLM post-processed; measure downstream extraction accuracy from resulting transcripts.

Measurement

WER per stratum (accent, noise, code-switch), correlation between WER and downstream field extraction accuracy, and improvement from mitigations.

Source video

Architectural implications

3

LLM output has different formatting requirements than TTS input.

Before

Raw LLM text streams directly into the TTS synthesis call.

After

A dedicated normalization layer sits between the LLM and TTS, handling emoji/markdown stripping, pronunciation dictionaries, and speed scaling.

Consequence

Formatting and prosody concerns are removed from the model; failures become localized and the normalization logic becomes independently testable.

Source video

Data collection accuracy depends on constraining the extraction target.

Before

Open-ended prompts ask the LLM to extract fields from conversation transcripts.

After

A typed schema (fields, types, validation rules, allowed values) defines capture as an explicit contract, backed by field-level tests.

Consequence

Extraction accuracy becomes measurable and improvable per field instead of relying on prompt tuning.

Source video

Silence detection is an inadequate proxy for conversational completeness.

Before

VAD/silence thresholds decide when the user has finished speaking.

After

Semantic and acoustic models predict end-of-turn; interruption classification decides whether to stop or continue for backchannels.

Consequence

Turn-taking becomes a probabilistic inference component that must be evaluated and tuned, not a fixed timeout parameter.

Source video

Tradeoffs and failure modes

4

Frontier model intelligence vs. latency

Benefit

Frontier models (OpenAI, Claude, Gemini) provide higher reasoning quality.

Cost or risk

They carry high TTFA (450-500ms) that can push the pipeline past the natural-conversation threshold unless dedicated capacity is used.

Frontier models (OpenAI, Claude, Gemini) have high TTFA (450-500ms) unless using dedicated capacity (Cerebras, Groq).
Open source video
Source video

Mixture-of-experts sizing

Benefit

MoE open-source models (e.g., Qwen 3.5, Gemma 4) can offer good speed-intelligence tradeoffs.

Cost or risk

MoE models require careful sizing to stay within the latency budget.

mixture-of-experts (MoE) models need careful sizing
Open source video
Source video

VAD-based turn detection

Benefit

Simple silence-based VAD is easy to implement.

Cost or risk

It fires on any pause, cutting users off mid-thought and mishandling backchannels.

VAD/silence-based turn detection fires on any pause, cutting users off mid-thought.
Open source video
Source video

Pronunciation and name handling in TTS

Benefit

Custom pronunciation dictionaries plus speed control (0.8x/0.7x) can fix difficult names and numbers.

Cost or risk

Failing to normalize before synthesis causes immediate trust loss with users.

if your agent can't pronounce your name, you've already dropped the ball
Open source video
Source video

Open questions

4

How should STT evaluation sets be constructed so that reported WER predicts production performance under accents, noise, and code-switching?

Why unresolved

The summary reports a large clean-vs-real-world WER gap but does not describe an evaluation methodology that closes it.

Research direction

Build accent/noise/code-switch-stratified evaluation corpora and measure correlation between eval WER and downstream task success.

Source video

How should interruptions be classified as backchannel vs. true barge-in, and what signals (lexical, prosodic, temporal) drive that decision?

Why unresolved

The summary states that classification is required but does not specify features, thresholds, or a model approach.

Research direction

Prototype an interruption classifier on prosody plus transcript tokens and measure stop/continue decision accuracy against human labels.

Source video

What confirmation mechanisms reliably resolve relative date-time expressions ('next week Wednesday at 8') into normalized values in a voice conversation?

Why unresolved

The summary says relative dates require normalization and confirmation but does not define the interaction or validation pattern.

Research direction

Design and test a multi-turn confirmation loop measuring extraction accuracy versus single-shot parsing.

Source video

How much of the TTFA budget should be allocated to each pipeline stage (STT, turn detection, LLM, TTS), and how does allocation change when swapping inference backends?

Why unresolved

The summary gives end-to-end TTFA thresholds but no per-stage budget breakdown.

Research direction

Instrument each stage in a reference pipeline and measure TTFA contributions across model/inference-provider configurations.

Source video

Key claims

8
factualVerification needed

Time-to-first-audio under 550ms feels natural, 750ms-1.2s causes conversational drag, and over 1.2s causes users to hang up.

Evidence

TTFA under 550ms feels natural/good; 750ms-1.2s causes conversation drag; over 1.2s causes users to hang up.

Question

Across what call volumes and demographics were these TTFA thresholds measured, and how consistent are they?

Source video
comparativeVerification needed

Frontier models have high TTFA (450-500ms) unless dedicated capacity such as Cerebras or Groq is used.

Evidence

Frontier models (OpenAI, Claude, Gemini) have high TTFA (450-500ms) unless using dedicated capacity (Cerebras, Groq).

Question

Are the 450-500ms figures measured in a full voice pipeline or in isolation, and do they hold across regions and providers?

Source video
factualVerification needed

Best STT engines reach 4-6% WER on clean eval sets but degrade significantly in production due to accent and noise.

Evidence

Best STTs achieve 4-6% Word Error Rate (WER) on clean eval sets, but real-world calls perform much worse due to accent and noise.

Question

What is the measured WER distribution on real production calls versus clean benchmarks?

Source video
causalVerification needed

Constrained capture with defined fields, types, validation rules, and allowed values raises extraction accuracy from 30% to 95%.

Evidence

Constrained capture (defining required fields, types, validation rules, allowed values) increases extraction accuracy from 30% to 95%.

Question

How was extraction accuracy measured, on what field types, and is the improvement sustained across domains and languages?

Source video
opinionVerification not requested

LLM output should never be sent directly to TTS; a separate normalization layer is required.

Evidence

LLM output should never go to TTS directly; a separate normalization layer is required.

Source video
comparativeVerification needed

VAD/silence-based turn detection fires on any pause and cuts users off mid-thought, whereas semantic and acoustic models use words and prosody to predict end-of-turn.

Evidence

VAD/silence-based turn detection fires on any pause, cutting users off mid-thought.

Question

What is the measured false-end-of-turn rate for VAD versus semantic/acoustic turn detection on real calls?

Source video
causalVerification needed

Interruption classification is required to decide whether an agent should stop or keep going during backchanneling.

Evidence

Backchanneling ('uh-huh', 'mhm') requires classifying interruptions to know whether to stop or keep going.

Question

What classifier accuracy is achievable for distinguishing backchannels from true barge-ins, and how does error affect user-perceived flow?

Source video
opinionVerification not requested

Field-level unit tests ensure extraction reliability.

Evidence

Using unit tests at a field level to ensure extraction reliability.

Source video

Connections

5