AI Engineer · Published 2026-08-30

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

privateReview: unreviewedConfidence: high
generative mediaimagengeminimultimodalvideo modelslatencymultimodal-modelsworld-modelsgenerative-mediavideo-modelshuman-evaluationreward-hackingevaluation
Open on YouTube ↗

Summary

Overview

  • Speaker: Dumitru Erhan, Shane Gu, Nicole Brichtova
  • Channel: AI Engineer
  • Main topic: State of the art in generative media models, focusing on video and image generation at Google DeepMind.
  • Purpose: Provide engineering insights and practical reflections on building, evaluating, and scaling state-of-the-art generative video and image models. A panel discussion featuring Dumitru Erhan, Shane Gu, and Nicole Brichtova from Google DeepMind discussing the recent launches, scaling challenges, architectural trade-offs, and future directions of generative media models like Imagen and Gemini Omni.

Topic Map

Recent Model Launches and Updates

  • Explanation: Discussion of recent model releases including Imagen 3 Light and Gemini Omni Flash APIs.
  • Key claims:
    • Imagen 3 Light is the fastest and cheapest model in the Imagen family with frontier quality.
    • Gemini Omni Flash APIs unlock low-latency video generation and editing at competitive pricing.
  • Examples:
    • Using Imagen 3 Light for rapid ideation and iteration with a 3-second latency window.
  • Terminology:
    • Imagen 3 Light
    • Gemini Omni Flash
    • API
  • Why it matters: Demonstrates the democratization of frontier-quality generative media for developers at lower cost and latency.

Modality Fusion and World Models

  • Explanation: Exploring the intersection of language, vision, and video models as foundation models for space and time.
  • Key claims:
    • Video models act as foundational models for space and time understanding.
    • Language models serve as a universal representation layer, though multimodal integration is critical for AGI.
  • Examples:
    • Translating gadget instructions from English to Romanian while preserving formatting and diagram details.
  • Terminology:
    • Foundation Models
    • Modality Fusion
    • World Models
    • Causal Factor
  • Why it matters: Highlights the shift from single-modality models toward unified multi-modal agent architectures.

Evaluation and Human Preference in Generative Media

  • Explanation: The challenges of evaluating generative video and image models using human feedback vs automated metrics.
  • Key claims:
    • Human evaluation remains essential due to the subjective nature of aesthetics and style.
    • Scaling evaluation with thousands of human evaluators helps capture nuanced preferences that models miss.
  • Examples:
    • Detecting subtle AI artifacts like wedding rings on every hand through trusted tester feedback.
  • Terminology:
    • Evals
    • Human Evaluation
    • Auto-raters
    • Reward Hacking
  • Why it matters: Good evaluation loops are critical for aligning generative media models with human aesthetic and practical standards.

Key Points

Low Latency Unlocks Creativity

  • Explanation: Achieving low latency (e.g., 3-second generation) fundamentally changes how creators iterate and ideate.
  • Evidence: Feedback from early users experimenting with Imagen 3 Light.
  • Practical implication: Developers should prioritize latency reduction alongside quality improvements in media pipelines.

Video as a Foundation Model

  • Explanation: Video generation models capture complex spatial and temporal dynamics that text-only models struggle to represent.
  • Evidence: Improvements in video understanding tasks when scaled alongside language models.
  • Practical implication: Investing in joint audio-video-text architectures provides stronger reasoning foundations for robotics and simulation.

The Limits of Language Representation

  • Explanation: While language is a powerful universal interface, it acts as a lossy compression format for sensory and aesthetic data like taste, smell, and visual texture.
  • Evidence: Difficulty in describing nuanced wine tastes or visual artifacts purely through text prompts.
  • Practical implication: Multimodal conditioning is necessary to bridge the gap between human intent and model output.

Frameworks, Models & Processes

Pre-training vs. Post-training Evaluation Framework

  • How it works: Splits model development into foundational pre-training for broad capabilities and targeted post-training for specific human preferences and safety.
  • Components:
    • Pre-training scaling
    • Instruction tuning
    • Human preference evaluation (Evals)
  • When to use: When developing and refining multi-modal generative models for production release.

Examples & Case Studies

A model consistently generated wedding rings on hands in generated images due to biased training data.

  • Illustrates: Unintended biases and reward hacking in generative models caught by human evaluators.
  • Lesson: Rigorous post-training feedback loops are required to catch subtle artifacts before public release.

Actionable Takeaways

  • Immediate:
    • Explore Imagen 3 Light for fast, low-latency image generation workflows.
    • Leverage Gemini Omni Flash APIs for video generation tasks.
  • Strategic:
    • Integrate multimodal foundation models (video, audio, text) rather than relying solely on text interfaces.
    • Build robust human evaluation loops alongside automated metrics.
  • Questions to investigate:
    • How do we bridge the sensory gap (taste, smell, touch) in generative models?
    • What is the optimal intermediate representation for joint video-language reasoning?

Claims Worth Verifying

  • Imagen 3 Light matches frontier model quality at lower cost and latency. (product performance claim)
  • Video models serve as foundational models for space and time. (architectural claim)

Notable Quotes

"And then second launch we finally launched the Gemini Omni Flash APIs that we pre announced at IO." (at 4:58) "The video model is an extremely good foundation model for space and time information." (at 16:50)

Compressed Summary

  • Imagen 3 Light offers frontier image generation quality with reduced latency and cost.
  • Gemini Omni Flash APIs bring video generation and editing capabilities to developers.
  • Video and spatial reasoning models are becoming foundational for broader multimodal AI systems.
  • Human evaluation and trusted tester feedback remain critical for catching subtle generation artifacts.
  • Keywords: generative media, imagen, gemini, multimodal, video models
  • Core insight: State-of-the-art generative media is evolving from single-modality models toward unified multimodal foundation models where low latency and rigorous human evaluation are key to practical deployment.

Core insights

5
Empirical Resultmedium noveltymoderate evidence

Generation latency is a capability threshold, not a cost metric: a ~3-second image-generation window changes generative media from one-shot batch requests into interactive ideation loops, fundamentally altering how creators explore alternatives.

Why it matters

Engineers selecting or building media models should optimize end-to-end interactive latency, not just throughput and price. Fast generation lets agents and users run many cheap iterations, which improves outcome quality in ways that a single high-latency call cannot.

Generalization

Any agentic system that can return a useful draft inside a few seconds supports trial-and-error search, user-in-the-loop refinement, and branch exploration; systems that take a minute or more force one-shot, overspecified requests. Latency budgets determine interaction paradigm.

Achieving low latency (e.g., 3-second generation) fundamentally changes how creators iterate and ideate.
Open source video
Using Imagen 3 Light for rapid ideation and iteration with a 3-second latency window.
Open source video
Mental Modelmedium noveltymoderate evidence

Language is a lossy compression for sensory and aesthetic data; prompt-only generative interfaces impose an irreducible bottleneck for visual texture, taste, smell, and exact reference details.

Why it matters

For creative or physical-generation tasks, the text prompt is the weakest link. Systems that accept and propagate reference images, video, or audio as first-class conditioning signals can represent user intent far more faithfully than text-to-media pipelines.

Generalization

In multimodal agent design, converting media to a few words is a lossy serialization. Preserve the original reference in context whenever the user supplies it, especially when the goal is visual fidelity rather than abstract description.

it acts as a lossy compression format for sensory and aesthetic data like taste, smell, and visual texture.
Open source video
Multimodal conditioning is necessary to bridge the gap between human intent and model output.
Open source video
Architecturehigh noveltymoderate evidence

Video generation models should be regarded as foundation models for space and time, not simply pixel predictors; training on video can produce spatiotemporal understanding that text-only models lack.

Why it matters

If video models learn a space-time prior, they become useful substrates for planning, simulation, robotics, and causal reasoning, not just for rendering clips. Architectures should treat video-native features as a first-class reasoning channel alongside language.

Generalization

Base-model selection should consider not only generation quality but what latent structure a model can expose for downstream reasoning; video-trained foundations may outperform text-only models on spatial-dynamics tasks even outside generation.

The video model is an extremely good foundation model for space and time information.
Open source video
Video models act as foundational models for space and time understanding.
Open source video
Improvements in video understanding tasks when scaled alongside language models.
Open source video
Practicemedium noveltymoderate evidence

Scaled human evaluation remains essential for subjective and artifact-heavy generative media; trusted tester feedback catches subtle systematic issues that auto-raters and automated metrics miss, such as wedding rings appearing on every generated hand.

Why it matters

Production releases must include large-scale human eval and trusted-tester loops as a post-training gate. Relying solely on automated evals risks shipping aesthetically fluent but biased or artifact-laden generations.

Generalization

Quality signals for generative systems are not fully reducible to scalar automated metrics. Where subjective preference or rare but systematic artifacts matter, design a human-feedback acquisition pipeline and feed its judgments back into the model.

Scaling evaluation with thousands of human evaluators helps capture nuanced preferences that models miss.
Open source video
Detecting subtle AI artifacts like wedding rings on every hand through trusted tester feedback.
Open source video
Failure Modehigh noveltymoderate evidence

A consistent, undesirable generation pattern across many images (e.g., a wedding ring on every hand) is a failure mode of biased training data or reward hacking, and it must be caught by targeted artifact probes, not only by average-preference scoring.

Why it matters

Distribution-level artifacts evade pointwise quality ratings and can corrode trust if they leak into real products. Post-training pipelines need explicit roll-call sampling and root-cause analysis for spurious correlations.

Generalization

AI reliability testing should include invariance checks: generate a diverse set of samples along a controlled variable and look for unwanted constant attributes. This is analogous to coverage testing in agent harnesses and catches reward exploitation more reliably than mean metrics.

A model consistently generated wedding rings on hands in generated images due to biased training data.
Open source video
Rigorous post-training feedback loops are required to catch subtle artifacts before public release.
Open source video

Deep dives

5

Latency thresholds as interactive creativity operators

Research question

How does end-to-end generation latency below a few seconds change user and agent exploration behavior, and does it materially improve final output quality compared with one-shot generation?

Why

If latency is a capability threshold rather than a cost metric, product and agent designs should optimize round-trip latency, support partial results, and expose edit/refine verbs so generative models can be used as search operators in creative workflows.

Achieving low latency (e.g., 3-second generation) fundamentally changes how creators iterate and ideate.
Open source video
Using Imagen 3 Light for rapid ideation and iteration with a 3-second latency window.
Open source video
Source video

Do video models learn a causal space-time prior or only high-fidelity pixel statistics?

Research question

Can video pretraining be distinguished from language pretraining with physical counterfactuals, occlusion, and object-permanence probes, and does that determine whether video models should be used as planners and simulators?

Why

Treating video models as foundation models for space and time has major architectural consequences; if they only mimic pixels statistically, relying on them for planning, robotics, or causal reasoning would be unsafe.

Video models act as foundational models for space and time understanding.
Open source video
Improvements in video understanding tasks when scaled alongside language models.
Open source video
Source video

Systematic artifact detection through controlled-variable sampling

Research question

Can a roll-call sampling method that varies controlled inputs catch systematic artifacts (e.g., rings on every hand) better than average-preference scoring, and what root causes does it expose?

Why

Spurious correlations and reward hacking can produce fluent but unacceptable outputs that evade pointwise quality metrics. A targeted invariance check is a cheap, automated way to catch distribution-level failures before release.

A model consistently generated wedding rings on hands in generated images due to biased training data.
Open source video
Detecting subtle AI artifacts like wedding rings on every hand through trusted tester feedback.
Open source video
Source video

Lossy text serialization in multimodal agent memory

Research question

For which visual-fidelity and editing tasks does preserving original image/video/audio payloads in agent context outperform text-summarized memory, and by how much?

Why

Language is a lossy compression format for sensory and aesthetic data, so prompt-only interfaces and text summaries can erase exactly the details generative and spatial tasks depend on. Agent runtimes need native multimodal message primitives.

it acts as a lossy compression format for sensory and aesthetic data like taste, smell, and visual texture.
Open source video
Language models serve as a universal representation layer, though multimodal integration is critical for AGI.
Open source video
Source video

Scaling human evaluation for subjective generative quality

Research question

At what evaluator scale and diversity do marginal preference insights saturate, and how should auto-raters be used to triage what needs human judgment?

Why

Human evaluation remains essential for aesthetics and style, but it is expensive and difficult to scale. Knowing where automated metrics fail and how to build human-in-the-loop data flywheels is critical for production alignment.

Scaling evaluation with thousands of human evaluators helps capture nuanced preferences that models miss.
Open source video
Human evaluation remains essential due to the subjective nature of aesthetics and style.
Open source video
Source video

Article ideas

4

The 3-Second Creativity Threshold

Generative media systems should be engineered as interactive search tools, not batch endpoints, because once generation returns in about three seconds, users and agents switch from one-shot prompting to iterative exploration and produce qualitatively better work.

Angle

Argue that latency is a capability and product design variable, not just an infrastructure cost; derive API, streaming, and agent-loop requirements from the interaction dynamics it unlocks.

Source video

Your Agent Doesn't Need a Memo, It Needs the Image

Agent memory that flattens visual and sensory inputs to text destroys the exact information generative and spatial tasks need to preserve; first-class multimodal context is an engineering requirement, not an optional extension.

Angle

Frame text summaries as lossy serialization and argue for native media payloads in message buses, memory, and tool protocols.

Source video

Video Models Are Claiming a Physical Intuition — Let's Probe It

If video generators actually learn space-time structure, we should benchmark them with physical counterfactuals and occlusion before wiring them into planners; generation fidelity alone cannot prove an internal causal world model.

Angle

Make the argument that video models may be physical-intuition engines, but standard fidelity evals are insufficient evidence; propose probing them as simulators.

Source video

The Wedding Ring Test: Why Scaled Human Eval Still Belongs in the Release Gate

Scaled human evaluators and trusted testers catch systematic artifacts and aesthetic blind spots that automated metrics and auto-raters miss, so no generative media release should gate on auto-raters alone.

Angle

Use the wedding-ring artifact as a concrete argument for adding human-feedback acquisition pipelines and artifact probes to standard evaluation stacks.

Source video

Project ideas

4

Roll-Call Artifact Scanner

beyond-evals

A controlled-variable roll-call sampler can detect systematic generation artifacts (e.g., a ring on every hand) with higher recall at equal generation budget than average-preference auto-raters.

Proof of concept

Build a service that generates images from parameterized prompts while holding an artifact-relevant subject constant and varying pose, lighting, and style; run a weak object detector for the suspected artifact and compare the artifact rate against auto-rater preference scores. Validate by injecting a known spurious correlation into a fine-tuned image model and probing the subset where the artifact should appear.

Measurement

Artifact recall, precision, false-positive rate, and number of generations needed to reach confident detection.

Source video

Latency-Loop Ideation Harness

movement-lab

When image generation latency is about three seconds, an agentic propose-critique-edit loop improves mean human-rated output quality and idea diversity compared with a single one-shot high-latency call under the same wall-clock or API budget.

Proof of concept

Use a low-latency image generation API with an edit/refine endpoint; implement two pipelines: a one-shot prompt-to-image pipeline and an iterative loop that scores drafts, proposes edits, and refines. Run the same creative tasks through both and collect human preference judgments on final outputs.

Measurement

Final-output win rate, number of distinct drafts explored, cost per accepted output, and average wall-clock time to a high-rated result.

Source video

Media-Native Memory Bus

new

In multi-turn visual editing tasks, preserving original image/video payloads as first-class context yields higher human-rated fidelity to subtle visual details than storing text summaries of each message.

Proof of concept

Build a simple agent harness with two memory backends: one stores native media payloads, the other summarizes every visual message into text. Ask users to issue reference-dependent edit commands and compare outputs on attribute-level correctness and number of clarification turns.

Measurement

Human-rated detail fidelity (e.g., correct reference objects, textures, and spatial layout) and clarification rates across turns.

Source video

Auto-Rater/Human Triage Flywheel

gatehouse

Routing only low-confidence auto-rater examples to human evaluators finds systematic artifacts and nuanced failures with higher per-human-label recall than random sampling at the same human budget.

Proof of concept

Simulate a task with a generative model plus an auto-rater; inject a subtle systematic artifact. Compare uncertainty-based triage (send low-confidence items to humans) vs random sampling. Track artifact discovery per human label and agreement between auto-rater, trusted tester, and model output.

Measurement

Artifact recall per human annotation, human-evaluator agreement, and steady-state improvement of the final preference model.

Source video

Architectural implications

4

Video models are described as foundation models for space and time, while language is described as a universal but lossy representation layer.

Before

The dominant architecture treats language as the core reasoning substrate, with vision and video models as peripheral encoders/decoders or special-purpose generators.

After

A joint audio-video-text base model becomes the substrate for both generation and understanding; planning and control applications can consume spatiotemporal representations directly instead of going through text captions.

Consequence

Agent frameworks need a representation strategy and serialization format for continuous multimodal latents, not only text messages; context must carry video/frame references and spatial-time tokens to preserve model capabilities.

Source video

Low-latency generative APIs (Imagen 3 Light, Gemini Omni Flash) are making media generation an interactive primitive rather than a delayed side effect.

Before

Generative media calls are usually asynchronous and expensive, so applications use one-shot static generation and try to improve prompts offline.

After

An agent can loop generate -> critique -> edit within a few seconds, using generation as a search operator in creative workflows and as cheap synthetic proposal sampling.

Consequence

System design should optimize round-trip latency, support partial/streamed results, and expose edit/refine endpoints as separate verbs; queue-based batching may be the wrong abstraction for interactive creative loops.

Source video

Language-only conditioning is lossy for sensory details and visual specifics, and multimodal conditioning was called out as necessary.

Before

Multi-agent communication and memory are text-centric: users or tools describe images, audio, or video in natural language, and the system stores only the text description.

After

First-class multimodal objects are passed between agents and stored as references in conversation memory, preserving fidelity across turns and between collaborating models.

Consequence

Harness/runtime APIs must support images, video, and audio as native message payloads, with explicit handling of size, provenance, and persistence so data is not unnecessarily flattened into lossy text.

Source video

Human evaluation at scale was presented as a core component of model post-training, specifically for aesthetics and style, alongside automated evals.

Before

Evals are commonly an offline, one-time gate after training, with metrics computed on fixed gold set and no ongoing feedback loop into the model.

After

Evaluation is an integrated post-training loop: run large-scale human preference collection, combine with auto-raters and trusted testers, and feed the filtered signal into the model's instruction-tuning or reward stage.

Consequence

Engineering work for evaluation shifts from writing static benchmark suites to building data flywheels, human-routing infrastructure, and bias/artifact scanners that continuously generate training and validation data.

Source video

Tradeoffs and failure modes

3

Automated metrics and auto-raters vs. scaled human evaluation

Benefit

Auto-raters can be evaluated continuously at scale, are cheap, and can guide model iterations quickly.

Cost or risk

They miss subjective aesthetic nuances and systematic artifacts such as rings on every hand; they can also be reward-hacked by the model.

Human evaluation remains essential due to the subjective nature of aesthetics and style. Scaling evaluation with thousands of human evaluators helps capture nuanced preferences that models miss.
Open source video
Source video

Language-only prompt interface vs. multimodal conditioning

Benefit

Text is a compact, flexible universal interface that works across many modalities and is easy to embed, store, and reason over.

Cost or risk

Text is a lossy compression of the actual desired output details (visual texture, spatial layout, sensory qualities), so prompt-only systems cannot fully convey intent.

Language ... acts as a lossy compression format for sensory and aesthetic data like taste, smell, and visual texture. Multimodal conditioning is necessary to bridge the gap between human intent and model output.
Open source video
Source video

Pre-training breadth vs. post-training human preference alignment

Benefit

Broad pre-training captures general capabilities, while targeted post-training aligns a model to human preferences and safety requirements.

Cost or risk

If the two phases are not tightly connected, post-training can distort the broad capabilities of pre-training or inherit hidden biases that only show up in the generated distribution.

Splits model development into foundational pre-training for broad capabilities and targeted post-training for specific human preferences and safety.
Open source video
Source video

Open questions

3

What is the optimal intermediate representation for joint video-language reasoning?

Why unresolved

The summary claims video models are good space-time foundation models but does not specify whether to bridge video and language via discrete tokens, continuous embeddings, or separate aligned encoders.

Research direction

Benchmark architectures on spatiotemporal reasoning tasks, not just generation fidelity, and measure transfer to robotics/simulation benchmarks.

Source video

How do we bridge the sensory gap (taste, smell, touch) in generative models?

Why unresolved

These modalities lack rich textual descriptions and simple machine-readable labels; language is a particularly lossy representation here and there is no established conditioning channel.

Research direction

Develop synthetic data pipelines and sensor/actuator feedback loops so that models can be conditioned directly on non-visual/non-textual signals.

Source video

Does learning to generate video actually produce an internal causal model of space and time, or can it be satisfied by sophisticated statistical imitation of pixels?

Why unresolved

The panel asserts that video is a strong foundation model for space and time, but does not define a causal measure or evaluation that distinguishes genuine world models from high-fidelity predictive patterns.

Research direction

Probe video models with physical counterfactuals, object permanence, and occlusions; use those evals as gates before treating video models as planners/simulators.

Source video

Key claims

7
comparativeVerification needed

Imagen 3 Light is the fastest and cheapest model in the Imagen family with frontier quality.

Evidence

Imagen 3 Light is the fastest and cheapest model in the Imagen family with frontier quality.

Question

Run an independent benchmark of Imagen 3 Light vs. other Imagen models for speed, cost, and human-rated quality under the same serving configuration.

Source video
comparativeVerification needed

Gemini Omni Flash APIs unlock low-latency video generation and editing at competitive pricing.

Evidence

Gemini Omni Flash APIs unlock low-latency video generation and editing at competitive pricing.

Question

Measure the API's end-to-end latency and per-minute cost across video generation and editing workloads, and compare with current market alternatives.

Source video
opinionVerification needed

Video models act as foundational models for space and time understanding.

Evidence

Video models act as foundational models for space and time understanding.

Question

Does video pretraining consistently improve downstream spatial-reasoning benchmarks (e.g., navigation, physics prediction) in a way that language-only pretraining does not?

Source video
causalVerification needed

Improvements in video understanding tasks are observed when models are scaled alongside language models.

Evidence

Improvements in video understanding tasks when scaled alongside language models.

Question

Controlled ablation isolating video-scale from language-scale: do video-understanding gains come from video data, language data, or compute scale?

Source video
causalVerification needed

A generation latency of about three seconds fundamentally changes how creators iterate and ideate.

Evidence

Achieving low latency (e.g., 3-second generation) fundamentally changes how creators iterate and ideate.

Question

Run a controlled user study comparing number of creations, exploration diversity, and final output quality across 3s vs. 30s generative feedback.

Source video
causalVerification needed

Scaling evaluation with thousands of human evaluators captures nuanced preferences that automated models miss.

Evidence

Scaling evaluation with thousands of human evaluators helps capture nuanced preferences that models miss.

Question

Analyze rating curves: at what evaluator count and diversity do marginal preference insights saturate for aesthetic/generative media tasks?

Source video
causalVerification needed

A model consistently generated wedding rings on hands due to biased training data.

Evidence

A model consistently generated wedding rings on hands in generated images due to biased training data.

Question

Identify the dataset spurious correlation causing persistent ring generation and reproduce it as a red-team artifact test.

Source video

Connections

5