AI Engineer · Published 2026-07-24

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

privateReview: unreviewedConfidence: high
multimodal agentevaluationsclosed-loopimage enhancementuber scaleclosed-loop-evalsmultimodal-agentagent-observabilityhuman-alignmentauto-tuningroutingpairwise-evaluationproduction-ml
Open on YouTube ↗

Summary

Overview

  • Speaker: Soumya Gupta, Jai Chopra
  • Channel: AI Engineer
  • Main topic: Building closed-loop evaluations for multimodal agents at Uber scale
  • Purpose: To share production-grade engineering practices for building, evaluating, and self-tuning multimodal AI agents at scale using closed-loop eval pipelines. Soumya Gupta and Jai Chopra from Uber explain how they designed real-world production evaluations and closed-loop feedback systems for multimodal agents handling food photography enhancement on Uber Eats. They cover agent architecture (image understanding, routing, generation, QA), offline human alignment, routing and generation evaluations, failure modes, and three feedback loops (model loop, dogfooding loop, marketplace loop) to maintain high agent performance at scale.

Topic Map

Introduction & Uber Eats Scale

  • Explanation: Introduction of the speakers from Uber computer vision team and context on Uber Eats scale ($90B run rate, millions of items added monthly, 20% YoY growth, 10,000+ cities globally).
  • Key claims:
    • Uber Eats delivery marketplace is just as big as the mobility side.
    • Visual content plays a critical role in user experience and initial impressions.
    • Smaller, independent merchants lack time, know-how, and budget for professional food photography.
  • Examples:
    • Merchants updating menus frequently face high costs for professional photo shoots.
  • Terminology:
    • Uber Eats
    • multimodal agent
    • food photography
  • Why it matters: Establishes the massive business scale and real-world stakes of automated image enhancement.

Agent Goals & Spectrum of Agency

  • Explanation: Defining the core goals for the image-editing agent (stay faithful, preserve brand, avoid sameness) and balancing determinism versus creativity on the agency spectrum.
  • Key claims:
    • Consumers want authentic photos and actively distrust AI-generated food photography.
    • Must stay faithful to original image, preserve merchant brand identity, and avoid sameness across marketplace images.
    • Agents provide the right balance between deterministic rules and high agency creativity.
  • Examples:
    • Having the same prompt for every photo would collapse marketplace diversity.
  • Terminology:
    • stay faithful
    • preserve brand
    • avoid sameness
    • agency spectrum
  • Why it matters: Sets the design constraints and guiding principles required for consumer trust in a marketplace.

Agent Orchestration & Logging

  • Explanation: Detailed walkthrough of the end-to-end orchestration pipeline: image quality understanding, routing, prompting, generation, LLM QA, post-processing, and logging.
  • Key claims:
    • End-to-end logging is critical to set up first before building self-learning loops.
    • All agent steps are logged in a flat JSON structure using Arize for observability and debugging.
    • Routing agent decides whether to enhance or skip based on rubrics (technical, composition, content, policy, dish match).
  • Examples:
    • Routing agent evaluates technical quality; if below par, sends to image editing agent.
  • Terminology:
    • orchestration
    • routing agent
    • LLM QA
    • Arize
    • logging
  • Why it matters: Provides a reproducible, observable architecture for multi-agent workflows in production.

Routing Evals: Offline Human Alignment

  • Explanation: How to evaluate and align the routing agent using human labels, precision/recall metrics, and handling drift online.
  • Key claims:
    • Human labelers serve as the golden source of truth to remove subjective biases.
    • Routing guardrail metric is recall to prevent bad images from slipping into the system.
    • Online tuning pipeline samples production data regularly, compares against human labels, and triggers auto-tuning via a diagnoser agent.
  • Examples:
    • Routing failure precision miss: high-quality cheeseburger image incorrectly flagged as low quality.
  • Terminology:
    • human alignment
    • recall
    • precision
    • confusion matrix
    • diagnoser agent
  • Why it matters: Ensures that automated routing aligns with human standards without requiring manual intervention at scale.

Generation Evals & Pairwise Comparison

  • Explanation: Evaluating image generation and enhancement using prompt generation, multi-dimensional QA gates, pass@k metrics, and pairwise comparisons.
  • Key claims:
    • Generation pipeline uses a 3-step process: generate prompt, enhance image, and pass through QA gate.
    • QA gate checks faithfulness, portion size, natural plating, colors/textures, and sharpness.
    • Pairwise comparison (pass@k) assesses whether the generated image is a better version of the original.
  • Examples:
    • Failure example: added shrimp to crab legs, failing faithfulness.
  • Terminology:
    • QA gate
    • pass@k
    • pairwise comparison
    • reward hacking
  • Why it matters: Prevents hallucinated food items, reward hacking, and unrealistic edits from reaching production menus.

Closed-Loop Feedback Systems

  • Explanation: Overview of the three feedback loops: Model Loop, Dogfooding Loop, and Marketplace Loop.
  • Key claims:
    • Model loop addresses drift against offline human labels.
    • Dogfooding loop collects internal thumbs up/down and freeform feedback from merchants and designers.
    • Marketplace loop measures business metrics like conversion rate via A/B experimentation.
  • Examples:
    • Slicing marketplace conversion data by geography, device type, and dish type to tune specific segments.
  • Terminology:
    • model loop
    • dogfooding loop
    • marketplace loop
    • diagnoser
    • conversion rate
  • Why it matters: Enables continuous self-improvement and marketplace optimization without human-in-the-loop bottlenecks.

Key Points

End-to-end logging must be established first

  • Explanation: Without comprehensive, flat JSON logging across all agent steps from the start, it is impossible to optimize or build self-learning loops.
  • Evidence: Mentioned during agent orchestration discussion and code overview using Arize.
  • Practical implication: Engineers should prioritize telemetry and structured logging before building complex agent feedback mechanisms.

Human labels as the golden source of truth

  • Explanation: Using objective guidelines for human labelers removes subjective bias and provides a reliable baseline for offline alignment.
  • Evidence: Offline human alignment workflow for routing and generation evals.
  • Practical implication: Invest in clean, unbiased human-labeled evaluation datasets before training or tuning agent prompts.

Config-driven automated tuning pipelines

  • Explanation: Using a diagnoser agent to analyze production drift, rewrite agent configurations, and trigger auto-tuning pipelines keeps systems sharp over time.
  • Evidence: Detailed explanation of online tuning focused on drift without human-in-the-loop.
  • Practical implication: Automate closed-loop tuning to reduce maintenance overhead for production AI agents.

Frameworks, Models & Processes

Multimodal Agent Orchestration Pipeline

  • How it works: Inputs pass through image quality understanding, routing, prompting, generation, LLM QA, and post-processing gates.
  • Components:
    • Image Quality Understanding
    • Routing Agent
    • Prompting Agent
    • Generation / Editing Agent
    • LLM QA Gate
    • Publish Ready QA
  • When to use: When building production multimodal AI workflows requiring strict quality, safety, and brand alignment.

Closed-Loop Feedback Framework

  • How it works: Combines production data, human labels, and feedback loops into a diagnoser agent that auto-tunes agent configurations and benchmarks them against gold standards.
  • Components:
    • Diagnoser Agent
    • Reflect Sub-agent
    • Synthesize Sub-agent
    • Agent Config Store
    • Evaluator & Benchmark Engine
  • When to use: When maintaining agent performance against model drift and data changes in production.

Examples & Case Studies

An image of 6 chicken wings passed routing with a dish name of 8 pieces chicken wings.

  • Illustrates: Recall miss in routing where the agent approved an incorrect count.
  • Lesson: Static offline models fail on edge cases; prompt and system alignment must evolve continuously.

An image enhancement agent added shrimp to a crab leg dish and removed sauce from sushi.

  • Illustrates: Faithfulness and completeness failure modes in generation.
  • Lesson: Rigorous QA gates and pairwise comparison evals are needed to catch hallucinations and reward hacking.

Actionable Takeaways

  • Immediate:
    • Implement structured end-to-end logging before building agent optimization loops.
    • Define clear rubric-based evaluation criteria (technical, composition, policy, dish match).
  • Strategic:
    • Balance agent agency and determinism to maintain safety without sacrificing creativity.
    • Build config-driven automated tuning pipelines to combat production data drift.
  • Questions to investigate:
    • How to scale human labeling cost-effectively for high-throughput multimodal applications?
    • What are the best automated metrics to detect subtle reward hacking in image generation?

Claims Worth Verifying

  • Uber Eats delivery marketplace is comparable in scale to the mobility side. (business scale)
  • End-to-end logging in a flat JSON structure significantly simplifies root-cause diagnosis for non-technical team members. (engineering practice)

Notable Quotes

"Building Closed-Loop Evals for a Multimodal Agent at Uber Scale" "End to end logging is important to setup first" (at 10:23)

Compressed Summary

  • Uber Eats scales food photography enhancement using multimodal agents.
  • Key goals: stay faithful, preserve brand, avoid sameness.
  • Architecture includes image understanding, routing, generation, LLM QA, and publishing gates.
  • Three feedback loops: model loop, dogfooding loop, and marketplace loop.
  • Diagnoser agent automates config tuning using production feedback without human intervention.
  • Keywords: multimodal agent, evaluations, closed-loop, image enhancement, uber scale
  • Core insight:

Core insights

7
Practicemedium noveltystrong evidence

Establish end-to-end structured logging (flat JSON per agent step, via Arize) as a prerequisite to any self-learning loop; the logging schema is effectively the interface that makes closed-loop tuning possible at all.

Why it matters

It inverts the usual build order: teams typically build agents, then bolt on observability. Here telemetry is the foundation that later eval/tuning loops consume, so retrofitting it is expensive.

Generalization

For any multi-agent pipeline intended to be auto-tuned, define the per-step record contract before the agent behavior, not after.

End-to-end logging is critical to set up first before building self-learning loops.
Open source video
All agent steps are logged in a flat JSON structure using Arize for observability and debugging.
Open source video
Without comprehensive, flat JSON logging across all agent steps from the start, it is impossible to optimize or build self-learning loops.
Open source video
Tradeoffmedium noveltystrong evidence

The routing guardrail metric is deliberately asymmetric: recall is the optimization target so bad images cannot slip into the system, accepting over-enhancement (precision loss) as the cheaper error.

Why it matters

Picking recall over precision is a value judgment encoded as a metric, and it directly determines which failure examples the team treats as bugs.

Generalization

In gated pipelines, choose the guardrail metric by the asymmetry of downstream cost, not by generic F1-style balance.

Routing guardrail metric is recall to prevent bad images from slipping into the system.
Open source video
Routing failure precision miss: high-quality cheeseburger image incorrectly flagged as low quality.
Open source video
Architecturehigh noveltymoderate evidence

A diagnoser agent closes the loop by sampling production data, comparing against human labels, rewriting agent configurations, and triggering auto-tuning pipelines — shifting human effort from labeling decisions to authoring guidelines and letting config rewrite be automated.

Why it matters

This puts an LLM inside the tuning layer of the system, not just the inference layer, which changes where drift remediation responsibility lives.

Generalization

Closed-loop self-tuning can be structured as: sample -> label-compare -> diagnose -> rewrite config -> redeploy, with an agent owning the diagnose/rewrite step.

Online tuning pipeline samples production data regularly, compares against human labels, and triggers auto-tuning via a diagnoser agent.
Open source video
Using a diagnoser agent to analyze production drift, rewrite agent configurations, and trigger auto-tuning pipelines keeps systems sharp over time.
Open source video
Mental Modelmedium noveltystrong evidence

Three nested feedback loops operate at different signal scopes — model loop (drift vs offline human labels), dogfooding loop (internal thumbs up/down and freeform merchant/designer feedback), marketplace loop (business conversion via A/B testing) — with per-segment slicing by geography, device type, and dish type.

Why it matters

Treating eval as a layered stack with distinct signal sources and scopes prevents conflating model-quality drift with business-metric drift, which require different remediation.

Generalization

Design eval as a hierarchy from model-level regression signals up to end-business outcome metrics, and slice the top layer by segment to localize tuning.

Model loop addresses drift against offline human labels.
Open source video
Dogfooding loop collects internal thumbs up/down and freeform feedback from merchants and designers.
Open source video
Marketplace loop measures business metrics like conversion rate via A/B experimentation.
Open source video
Slicing marketplace conversion data by geography, device type, and dish type to tune specific segments.
Open source video
Mechanismmedium noveltystrong evidence

Generation quality is enforced by a composition of a multi-dimension QA gate (faithfulness, portion size, natural plating, colors/textures, sharpness) plus pairwise pass@k comparison against the original, rather than a single scalar score.

Why it matters

Pairwise-vs-original framing directly targets regressions (edits that are worse than the input), which absolute quality scores cannot detect; the multi-dimension gate separately catches specific hallucination classes like added ingredients.

Generalization

For enhancement/edit agents, evaluate each output against its own input as a pairwise baseline, not only against an absolute quality rubric.

QA gate checks faithfulness, portion size, natural plating, colors/textures, and sharpness.
Open source video
Pairwise comparison (pass@k) assesses whether the generated image is a better version of the original.
Open source video
Failure example: added shrimp to crab legs, failing faithfulness.
Open source video
Mental Modelhigh noveltymoderate evidence

Agent goals are encoded as a constrained triple — stay faithful, preserve brand, avoid sameness — making diversity across the marketplace an explicit objective rather than a byproduct; a uniform prompt would collapse it.

Why it matters

Introces a global, cross-item metric (sameness) that a per-item eval harness cannot measure, forcing a marketplace-level view into the eval design.

Generalization

When an agent acts on many items in a shared surface, add cross-item distributional constraints to the objective, not just per-item quality gates.

Must stay faithful to original image, preserve merchant brand identity, and avoid sameness across marketplace images.
Open source video
Having the same prompt for every photo would collapse marketplace diversity.
Open source video
Architecturemedium noveltymoderate evidence

Routing is rubric-based and selective (technical, composition, content, policy, dish match), so most items can skip the expensive generative path — evaluation of the router therefore has direct cost implications for the whole pipeline.

Why it matters

The router is both a quality gate and a cost-control valve; its recall target means it must bias toward routing more items into generation.

Generalization

Where a generative step is expensive, the classifier in front of it should be evaluated on the cost-weighted confusion matrix, not accuracy.

Routing agent decides whether to enhance or skip based on rubrics (technical, composition, content, policy, dish match).
Open source video
Routing agent evaluates technical quality; if below par, sends to image editing agent.
Open source video

Deep dives

5

Guardrails for agents that rewrite production configuration

Research question

What validation, canary, and rollback envelope is required before an LLM diagnoser's rewritten agent configuration is promoted to production, and does shadow evaluation on held-out human-labeled data reduce blast radius?

Why

The talk places an LLM in the tuning layer, not just the inference layer. That creates a new class of production change whose blast radius spans every routed item, yet no promotion gate is described. Without a promotion contract, drift remediation and drift causation become indistinguishable.

Online tuning pipeline samples production data regularly, compares against human labels, and triggers auto-tuning via a diagnoser agent.
Open source video
Using a diagnoser agent to analyze production drift, rewrite agent configurations, and trigger auto-tuning pipelines keeps systems sharp over time.
Open source video
Source video

Quantifying 'avoid sameness' as a marketplace-level eval metric

Research question

Which distributional diversity metrics over generated image sets (embedding spread, prompt/style entropy, pairwise perceptual distance) correlate with marketplace conversion, and can a per-item eval harness detect a collapse toward sameness?

Why

The objective triple explicitly includes a cross-item constraint that no per-item quality gate can observe. If sameness is not measured, a locally improving agent can monotonically degrade the shared marketplace surface while every item-level metric stays green.

Must stay faithful to original image, preserve merchant brand identity, and avoid sameness across marketplace images.
Open source video
Having the same prompt for every photo would collapse marketplace diversity.
Open source video
Source video

Representativeness and staleness control in production sampling for drift detection

Research question

How should production samples be stratified (by geography, device type, dish type, merchant tier) so that label-distribution drift is detected before it moves marketplace metrics, and what sampling cadence bounds detection latency?

Why

The entire closed loop is only as good as the sample it diagnoses. If sampling is unstratified or silently stale, the diagnoser rewrites configs in response to a distribution that no longer exists, and auto-tuning becomes a drift amplifier rather than a corrector.

Slicing marketplace conversion data by geography, device type, and dish type to tune specific segments.
Open source video
Model loop addresses drift against offline human labels.
Open source video
Source video

Cost-weighted evaluation of the router as a compute valve

Research question

Does optimizing the router's recall-first guardrail on a cost-weighted confusion matrix rather than on F1 change the routing threshold enough to materially alter generation spend per thousand items at constant quality?

Why

The router is simultaneously the quality gate and the spend control for the whole pipeline, so its threshold is an economic parameter. A precision miss documented in the talk (a good cheeseburger flagged as low quality) is not a quality bug but a budget line item, and should be evaluated as one.

Routing guardrail metric is recall to prevent bad images from slipping into the system.
Open source video
Routing failure precision miss: high-quality cheeseburger image incorrectly flagged as low quality.
Open source video
Routing agent decides whether to enhance or skip based on rubrics (technical, composition, content, policy, dish match).
Open source video
Source video

Calibrating pairwise pass@k against human pairwise judgment

Research question

How much do automated pairwise-vs-original pass@k judgments agree with human pairwise labels, and does agreement hold across dish types where portion size and plating expectations differ?

Why

The QA stack combines a multi-dimension rubric, pairwise comparison, and human labels, but if pass@k diverges from human judgment per segment it will silently validate edits that the golden source rejects, making the two layers of the eval stack contradictory rather than layered.

Pairwise comparison (pass@k) assesses whether the generated image is a better version of the original.
Open source video
Human labelers serve as the golden source of truth to remove subjective biases.
Open source video
QA gate checks faithfulness, portion size, natural plating, colors/textures, and sharpness.
Open source video
Source video

Article ideas

4

Your Trace Schema Is the Real Agent Interface

Teams that build agents and then bolt on observability cannot build self-learning loops; the flat per-step record contract must be designed first because it is the substrate that every tuning loop consumes, and it therefore deserves versioning discipline like any other public interface.

Angle

Invert the standard build order using a single production case: show which tuning capabilities became possible only because the logging contract existed up front, and what breaks when the schema drifts.

Source video

Recall Is a Budget Decision, Not a Quality Preference

Choosing recall for a gating classifier is not a generic best practice but a statement about asymmetric downstream cost; teams should derive the guardrail metric from the price of each error and evaluate the router on a cost-weighted confusion matrix instead of F1.

Angle

Walk through the recall-first routing decision and show how the accepted precision loss (routing high-quality images needlessly) is a compute line item, not a quality bug.

Source video

Putting an LLM in the Tuning Layer Changes Who Owns Drift

Moving drift remediation from a human reading dashboards to a diagnoser agent that rewrites configuration relocates responsibility for correctness: the config becomes generated output, so it needs its own canary, shadow evaluation, and rollback path before any loop closes automatically.

Angle

Reframe the sample, compare, diagnose, rewrite, redeploy loop as a code-generation pipeline and apply the same release engineering rigor to it.

Source video

Per-Item Gates Cannot Measure Marketplace Sameness

When an agent acts across many items on a shared surface, per-item quality gates miss the failure mode that matters most: the gradual collapse of diversity; distributional constraints must be promoted to first-class objectives with their own metrics and owners.

Angle

Use the uniform-prompt collapse as the concrete illustration of a metric that is invisible to item-level evaluation but decisive for marketplace trust.

Source video

Project ideas

4

Config-Diff Shadow Gate

beyond-evals

A diagnoser-generated configuration change that is shadow-evaluated against a held-out human-labeled set before promotion will show lower regression rates on production metrics than direct promotion of the same rewrite, at a bounded evaluation cost.

Proof of concept

Wrap the diagnoser's rewritten configs in a promotion service that replays both old and new configs against a frozen stratified human-labeled sample, computes per-dimension deltas on the existing QA rubric, and requires a positive pairwise delta before canarying to a small traffic slice with automatic rollback.

Measurement

Regression rate of promoted configs (fraction that must be rolled back), per-dimension QA delta between shadow and production outcome, and added evaluation cost per promotion.

Source video

Sameness Meter

movement-lab

Embedding spread and prompt/style entropy computed over a generated image batch will detect a diversity collapse at least one tuning cycle earlier than marketplace conversion metrics, and will correlate with conversion when conversion eventually moves.

Proof of concept

Compute per-batch distributional diversity metrics over generated images for a fixed menu cohort, replay historical batches where a prompt change occurred, and compare the metric trajectory against the corresponding A/B conversion series.

Measurement

Lead time in days between diversity-metric deviation and conversion deviation, plus rank correlation between the two series across cohorts.

Source video

Cost-Weighted Router Bench

gatehouse

Tuning the routing threshold on a cost-weighted confusion matrix (where a precision miss is priced as wasted generation spend) will find a threshold that reduces generation spend per thousand items with no measurable drop in downstream QA pass rates, compared to a recall-only threshold.

Proof of concept

Build an offline replay harness over logged routing decisions with known downstream labels, sweep the threshold, and plot spend per thousand items against QA pass rate and faithfulness-gate failure rate to locate the Pareto frontier.

Measurement

Generation spend per thousand items at matched QA pass rate; change in false-positive routing rate; faithfulness failure rate at the chosen threshold.

Source video

Drift Sampling Simulator

new

Stratified production sampling by geography, device type, and dish type detects injected label-distribution drift earlier than uniform random sampling at equal sample volume, and the detection advantage grows as drift concentrates in one segment.

Proof of concept

Simulate a production stream with synthetic segment-level drift injections, run stratified versus uniform samplers at matched budget, and measure detection latency and false-alarm rate for each sampling policy as drift concentration varies.

Measurement

Detection latency in sample cycles at fixed false-alarm rate; segment-level recall of drift detection; samples required per true drift event.

Source video

Architectural implications

3

Observability is a prerequisite layer that the tuning loop depends on, rather than a downstream add-on.

Before

Build agent, then add logging/dashboards when something breaks.

After

Define a flat per-step JSON record schema consumed by both debugging and automated tuning before agent behavior is finalized.

Consequence

The trace schema becomes a versioned interface; changing agent steps requires changing the record contract, and tuning pipelines break if it drifts.

Source video

An agent (the diagnoser) now sits in the tuning/configuration layer, not only in the inference path.

Before

Humans read eval dashboards and edit prompts/configs manually.

After

A diagnoser agent analyzes drift, rewrites configurations, and triggers auto-tuning pipelines without human-in-the-loop.

Consequence

New safety surface: config-rewriting agents need their own eval and rollback, since their output changes production behavior broadly.

Source video

Evaluation is stratified into three loops with distinct signal types (human labels, internal feedback, business A/B metrics).

Before

One offline eval suite used as the proxy for all quality questions.

After

Separate loops per signal scope, with the marketplace loop sliced by geography, device type, and dish type.

Consequence

Teams must reconcile conflicting signals across loops and decide which layer owns a given regression.

Source video

Tradeoffs and failure modes

5

Routing guardrail metric selection

Benefit

Optimizing recall ensures bad images do not slip into the enhancement system.

Cost or risk

Loses precision, e.g., a high-quality image is flagged as low quality and unnecessarily routed, spending generation cost.

Routing guardrail metric is recall to prevent bad images from slipping into the system.
Open source video
Source video

Agency vs determinism in an image agent

Benefit

Agents balance deterministic rules with creative agency, avoiding uniform outputs across the marketplace.

Cost or risk

Higher agency raises the risk of unfaithful or hallucinated edits that break consumer trust.

Agents provide the right balance between deterministic rules and high agency creativity.
Open source video
Source video

Automated generation QA gates

Benefit

Prevents hallucinated food items, reward hacking, and unrealistic edits from reaching production menus.

Cost or risk

Multi-dimension QA plus pairwise pass@k adds evaluation cost and can still be gamed structurally (reward hacking).

Prevents hallucinated food items, reward hacking, and unrealistic edits from reaching production menus.
Open source video
Source video

AI-generated food photography

Benefit

Automated enhancement serves merchants who lack budget for professional photo shoots.

Cost or risk

Consumers actively distrust AI-generated food photography, so authenticity must be preserved for trust to hold.

Consumers want authentic photos and actively distrust AI-generated food photography.
Open source video
Source video

Automated closed-loop tuning

Benefit

Reduces maintenance overhead for production AI agents by removing human-in-the-loop drift correction.

Cost or risk

Removing humans from tuning increases the blast radius of an incorrect diagnoser decision.

Automate closed-loop tuning to reduce maintenance overhead for production AI agents.
Open source video
Source video

Open questions

5

How does the pipeline guarantee that production samples used for drift detection remain representative of the live distribution as the marketplace shifts?

Why unresolved

The summary states sampling happens regularly but gives no sampling strategy, stratification, or staleness control.

Research direction

Design and benchmark stratified production sampling schemes against label-distribution drift metrics.

Source video

What guardrails constrain the diagnoser agent when it rewrites configurations, given no human is in the loop?

Why unresolved

Auto-tuning via config rewrite is described but no validation, canary, or rollback mechanism is mentioned.

Research direction

Prototype config-diff validation: shadow evaluation of rewritten configs on held-out human-labeled sets before promotion.

Source video

How is 'avoid sameness' quantified and measured across a marketplace of millions of items?

Why unresolved

The objective is stated as a constraint but no metric or measurement method is provided.

Research direction

Define distributional diversity metrics (embedding spread, prompt/style entropy) over generated image sets and test their correlation with conversion.

Source video

How does the marketplace loop attribute a conversion-rate change to the image agent rather than to other marketplace variables in the A/B test?

Why unresolved

The summary describes commerce slicing by geography, device type, and dish type but not the causal identification strategy.

Research direction

Apply segmented A/B analysis with variance controls and sensitivity checks to isolate image-driven conversion effects.

Source video

How much do pairwise pass@k scores agree with the human-labeled golden source, and does that agreement hold across dish types?

Why unresolved

Both human labels and pass@k are described, but their correlation or agreement is never reported.

Research direction

Build a calibration study comparing automated pairwise judgments against human pairwise labels per dish segment.

Source video

Key claims

8
causalVerification needed

End-to-end logging must be established before self-learning loops can be built.

Evidence

Without comprehensive, flat JSON logging across all agent steps from the start, it is impossible to optimize or build self-learning loops.

Question

Does partial or retrofitted logging measurably degrade the effectiveness of automated tuning loops?

Source video
factualVerification needed

All agent steps are logged in a flat JSON structure using Arize for observability and debugging.

Evidence

All agent steps are logged in a flat JSON structure using Arize for observability and debugging.

Question

Does a flat schema scale for nested multi-agent traces without losing step-level attribution?

Source video
causalVerification needed

The routing guardrail metric is recall, so bad images do not slip through.

Evidence

Routing guardrail metric is recall to prevent bad images from slipping into the system.

Question

What is the measured cost penalty of the precision loss incurred by recall-first routing?

Source video
causalVerification needed

Human labelers using objective guidelines remove subjective bias and serve as the golden source of truth.

Evidence

Human labelers serve as the golden source of truth to remove subjective biases.

Question

How much inter-labeler disagreement remains after guidelines, and how does that bound eval noise?

Source video
factualVerification needed

Online tuning compares production samples against human labels and triggers auto-tuning via a diagnoser agent.

Evidence

Online tuning pipeline samples production data regularly, compares against human labels, and triggers auto-tuning via a diagnoser agent.

Question

How frequently must sampling occur to detect drift before it affects marketplace metrics?

Source video
factualVerification needed

Pairwise comparison via pass@k determines whether a generated image is better than the original.

Evidence

Pairwise comparison (pass@k) assesses whether the generated image is a better version of the original.

Question

What pass@k threshold is used, and how does it correlate with human preference?

Source video
opinionVerification needed

AI-generated food photography is actively distrusted by consumers.

Evidence

Consumers want authentic photos and actively distrust AI-generated food photography.

Question

Is the distrust measurable in conversion or trust surveys, and does it shift as enhancement subtlety changes?

Source video
causalVerification needed

No single prompt should be used across all photos because it would collapse marketplace diversity.

Evidence

Having the same prompt for every photo would collapse marketplace diversity.

Question

Does prompt/style diversity measurably increase engagement versus uniform output?

Source video

Connections

5