AI Engineer · Published 2026-09-10

Training Taste — Thais Castello Branco, Taste Labs

privateReview: unreviewedConfidence: high
sloptastedesignprobesbrand-apislop-detectioninference-timeclassifier-evaldesign-systemsmode-collapsequality-gates
Open on YouTube ↗

Summary

Overview

  • Speaker: Thais Castello Branco
  • Channel: AI Engineer
  • Main topic: Measuring and solving AI slop through design engineering, taste, and agent app-layer tools
  • Purpose: To explain how to measure, understand, and fix AI slop in design and applications by raising the bar of judgment and introducing tools like Brand API. Thais Castello Branco, founder of Taste Labs, discusses how to decode subjective domains like design and end AI slop. She explores the shift from coding and math capability in AI models to design and aesthetics, identifying three key characteristics of AI slop: repetition, lack of fit, and low intent. She presents quantitative probing methods for slop identification and introduces practical tools like the Brand API to ensure AI agents adhere to brand standards and human judgment at interference time.

Topic Map

The Problem of AI Slop in Design

  • Explanation: Discusses how while AI models have achieved incredible capability in coding and math, they struggle with design, writing, and aesthetics, leading to widespread AI slop.
  • Key claims:
    • AI slop is driven by repetition, lack of fit, and low intent.
    • The cost of generation is going to zero, making judgment more valuable than ever.
  • Examples:
    • AI-generated websites and pitch decks that look repetitive and soulless.
  • Terminology:
    • AI slop
    • mode collapse
    • subjective domains
  • Why it matters: Understanding slop is the first step toward building systems that enforce taste and prevent generic, low-effort AI outputs.

Measuring Slop Quantitatively

  • Explanation: Explains how Taste Labs analyzed 2.4 million websites from the past 10 years and synthetically generated AI sites to build probes (baby classifiers) to measure and detect slop.
  • Key claims:
    • Patterns are an effective way to identify slop quantitatively.
    • Slop probes separate designer-approved from designer-rejected sites at AUC 0.74, outperforming many LLM judge methodologies.
  • Examples:
    • Probes checking for weight-only type pairing, neutral color palettes, Bootstrap/jQuery, and single font families.
  • Terminology:
    • probes
    • baby classifiers
    • AUC
    • pattern mining
  • Why it matters: In order to fix slop, you first have to measure it and understand it quantitatively.

Interference Time Solutions & Brand API

  • Explanation: Introduces the Brand API as an agent tool that extracts structured brand components from a URL and helps agents produce high-fidelity, on-brand outputs.
  • Key claims:
    • Solving slop at inference time is crucial because that is when user interaction and intent happen.
    • Brand API extracts brand guidelines into structured context for agents to score and verify outputs.
  • Examples:
    • The General Intelligence Company of New York website generated via Claude Design vs Claude Design + Taste API.
  • Terminology:
    • Brand API
    • inference time
    • agent tools
    • brand adherence
  • Why it matters: Provides practical engineering infrastructure to ensure AI agents maintain high aesthetic standards and brand consistency.

Key Points

Defining AI Slop Characteristics

  • Explanation: Slop is characterized by repetition (seeing the same thing many times), lack of fit (failing to match specific context or user intent), and low intent.
  • Evidence: Analysis of millions of web designs and AI-generated outputs.
  • Practical implication: Developers and designers must build constraints and judgment layers to prevent low-intent, repetitive AI output.

Measuring via Probes

  • Explanation: Using pattern mining on design features allows the creation of lightweight classifiers ('probes') that detect slop with high accuracy.
  • Evidence: Slop probe achieving AUC 0.74 in separating designer-approved from designer-rejected sites.
  • Practical implication: Teams can implement automated verification gates in production pipelines to block slop.

The Increasing Value of Judgment

  • Explanation: As the cost of generation drops to zero due to AI, human judgment and design curation become the most expensive and valuable assets.
  • Evidence: Observations on modern web generation capabilities available to non-designers.
  • Practical implication: Focus must shift from raw generation capability to curation, taste, and verification systems.

Frameworks, Models & Processes

Taste Labs Slop Mitigation Framework

  • How it works: Breaks down subjective design problems into measurable pillars: Repetition, Fit, Intent, and Judgement.
  • Components:
    • Repetition -> Creativity API
    • Fit -> Brand API
    • Intent -> Prompt enhancement
    • Judgement -> Baby classifier for slop
  • When to use: When designing agent applications and training frontier models to avoid generic, uninspired outputs.

Examples & Case Studies

Analyzing 2.4 million websites from the past 10 years and comparing them against thousands of AI-generated sites.

  • Illustrates: The homogeneity and pattern repetition introduced by AI generation.
  • Lesson: AI significantly accelerates pattern repetition and lack of fit across digital domains.

Using Brand API with The General Intelligence Company website prompt.

  • Illustrates: Extracting structured brand components from a URL to guide an AI agent.
  • Lesson: Providing structured brand context dramatically improves the fidelity and aesthetic quality of agent outputs.

Actionable Takeaways

  • Immediate:
    • Define quantitative probes to catch common design and text patterns in AI outputs.
    • Implement inference-time validation gates for AI-generated assets.
  • Strategic:
    • Shift focus from generation cost reduction to judgment and curation capability.
    • Build structured brand repositories and APIs to guide AI agents.
  • Questions to investigate:
    • How can we scale human judgment in automated agent pipelines?
    • What other subjective domains beyond design can be decoded into structured APIs?

Claims Worth Verifying

  • The slop probe separates designer-approved from designer-rejected sites at AUC 0.74. (metric)
  • Over 2.4 million websites from the past 10 years were analyzed. (empirical)

Notable Quotes

"To reach 'great', we first need to raise the floor. To raise the floor, we need to know how to measure it, decompose it and affect it." (at 23:18)

Compressed Summary

  • AI generation has eliminated production costs, making human judgment and taste paramount.
  • AI slop is defined by repetition, lack of fit, and low intent.
  • Taste Labs uses quantitative probes (AUC 0.74) to detect slop in design.
  • Brand API structures brand guidelines so AI agents can produce on-brand, high-fidelity outputs.
  • Keywords: slop, taste, design, probes, brand-api
  • Core insight: To eliminate AI slop, we must measure and decompose subjective design patterns into structured APIs and judgment gates at inference time.

Core insights

6
Empirical Resulthigh noveltymoderate evidence

Cheap, pattern-based 'baby classifier' probes can outperform LLM-as-judge for detecting low-quality output in subjective domains (design), reaching AUC 0.74 separating designer-approved from designer-rejected sites.

Why it matters

Suggests that for some subjective-quality evals, mining concrete surface patterns and training small discriminative classifiers may be a better cost/accuracy tradeoff than prompting a large LLM to judge aesthetics, which is expensive and known to be noisy.

Generalization

Any subjective or 'taste-driven' acceptance criterion may be approximated by a lightweight feature/pattern classifier extracted from large corpora, used as an automated gate rather than an LLM rubric.

Slop probes separate designer-approved from designer-rejected sites at AUC 0.74, outperforming many LLM judge methodologies.
Open source video
Patterns are an effective way to identify slop quantitatively.
Open source video
Probes checking for weight-only type pairing, neutral color palettes, Bootstrap/jQuery, and single font families.
Open source video
Architecturehigh noveltymoderate evidence

A subjective domain is decomposed into four measurable pillars each mapped to a distinct mechanism: Repetition→Creativity API, Fit→Brand API, Intent→prompt enhancement, Judgement→baby classifier.

Why it matters

Gives a reusable decomposition pattern: instead of one monolithic 'quality model', split the subjective objective into separable failure axes, each served by its own specialized tool so failures are attributable and independently fixable.

Generalization

Subjective objectives in agentic systems should be factored into orthogonal, individually measurable dimensions with a dedicated mechanism per dimension rather than a single opaque judge.

Breaks down subjective design problems into measurable pillars: Repetition, Fit, Intent, and Judgement.
Open source video
Repetition -> Creativity API
Open source video
Fit -> Brand API
Open source video
Intent -> Prompt enhancement
Open source video
Judgement -> Baby classifier for slop
Open source video
Mechanismmedium noveltymoderate evidence

The Brand API extracts structured brand components from a URL and supplies them as structured context that agents use to score and verify their own outputs, rather than relying on prose style instructions.

Why it matters

Turns tacit brand/design knowledge into a machine-readable artifact that can be fetched, diffed, and used for verification, making brand adherence an inspectable interface rather than an implicit prompt hint.

Generalization

Tacit human standards (brand, tone, conventions) are best encoded as structured, retrievable APIs that agents can query and self-verify against, not as long natural-language instructions.

Brand API extracts brand guidelines into structured context for agents to score and verify outputs.
Open source video
Extracting structured brand components from a URL to guide an AI agent.
Open source video
Providing structured brand context dramatically improves the fidelity and aesthetic quality of agent outputs.
Open source video
Architecturemedium noveltymoderate evidence

Quality enforcement is placed at inference time rather than training time, because 'that is when user interaction and intent happen.'

Why it matters

Locates the responsibility for constraint enforcement in the agent runtime/tool layer, implying quality gates should be composable runtime tools that intercept generation, not properties baked into a base model.

Generalization

For subjective constraints that are context-dependent, enforcement belongs at inference/run time where intent is available, not in static model training.

Solving slop at inference time is crucial because that is when user interaction and intent happen.
Open source video
Predictionmedium noveltymoderate evidence

As generation cost approaches zero, the scarce and valuable resource shifts from generation capability to judgment and curation, so engineering investment should move toward verification systems.

Why it matters

Reframes roadmap priorities: the differentiator is no longer the generator but the verification/curation layer around it, matching the broader pattern of evaluation and gating infrastructure becoming the bottleneck.

Generalization

When a capability becomes commoditized/cheap, value migrates to the adjacent scarce resource (judgment, verification, selection) — build the constraint layer, not another generator.

The cost of generation is going to zero, making judgment more valuable than ever.
Open source video
Focus must shift from raw generation capability to curation, taste, and verification systems.
Open source video
Mental Modelmedium noveltymoderate evidence

AI slop is definable as three concrete failure modes — repetition, lack of fit, and low intent — which function as an operational eval taxonomy rather than a vague aesthetic complaint.

Why it matters

Converts a fuzzy, subjective complaint into named axes that can each be instrumented (frequency/duplication measures, context-fit checks, intent-capture checks), enabling targeted diagnostics.

Generalization

Naming the discrete failure modes of a quality problem is a prerequisite to measuring and remediating it in automated pipelines.

AI slop is driven by repetition, lack of fit, and low intent.
Open source video
Slop is characterized by repetition (seeing the same thing many times), lack of fit (failing to match specific context or user intent), and low intent.
Open source video
To reach 'great', we first need to raise the floor. To raise the floor, we need to know how to measure it, decompose it and affect it.
Open source video

Deep dives

4

Probe-based subjective quality gates vs LLM judges

Research question

Under what labeling conditions, feature sets, and distribution shifts do small pattern-based probes outperform LLM-as-judge on subjective quality tasks, and where do they fail?

Why

If cheap probes are as or more discriminative than LLM judges for design slop, eval and guardrail stacks should default to probe-first architecture with LLM judges reserved for disagreement cases.

Slop probes separate designer-approved from designer-rejected sites at AUC 0.74, outperforming many LLM judge methodologies.
Open source video
Patterns are an effective way to identify slop quantitatively.
Open source video
Probes checking for weight-only type pairing, neutral color palettes, Bootstrap/jQuery, and single font families.
Open source video
Source video

Factoring subjective quality into orthogonal measurable pillars

Research question

Can subjective objectives be reliably decomposed into orthogonal axes (repetition, fit, intent, judgment) each served by a dedicated mechanism, and does attributing failures per axis improve remediation?

Why

A monolithic judge makes failures opaque; separable pillars make them attributable, independently testable, and independently fixable in agent pipelines.

AI slop is driven by repetition, lack of fit, and low intent.
Open source video
Slop is characterized by repetition (seeing the same thing many times), lack of fit (failing to match specific context or user intent), and low intent.
Open source video
Source video

Inference-time constraint enforcement as app-layer tooling

Research question

How do inference-time quality gates compare with training-time or fine-tuning-based approaches on output quality, cost, latency, and context sensitivity?

Why

If intent is only available at inference time, quality enforcement must be a runtime tool; understanding its cost profile determines architectural placement of gates.

Solving slop at inference time is crucial because that is when user interaction and intent happen.
Open source video
Brand API extracts brand guidelines into structured context for agents to score and verify outputs.
Open source video
Source video

Structured tacit standards as retrievable APIs

Research question

What is the fidelity ceiling of encoding brand/taste into structured schemas, and what tacit judgment is lost versus prose or human review?

Why

Brand API success depends on how much of tacit design judgment can be operationalized; knowing the lossy boundary determines which decisions still need humans.

Brand API extracts brand guidelines into structured context for agents to score and verify outputs.
Open source video
Source video

Article ideas

4

Your Aesthetic Judge Should Be a Baby Classifier (Not a Prompt)

For subjective quality gates like design slop, lightweight pattern probes are a better default evaluator than LLM-as-judge because they are cheaper, faster, and—at least in the reported design case—more discriminative, so teams should adopt probe-first pipelines and reserve LLM judges for edge cases.

Angle

Contrarian eval-engineering argument grounded in AUC/cost tradeoff.

Source video

Stop Prompting for Brand, Ship a Brand API

Brand adherence should not be a prose instruction; it should be an inspectable runtime API that extracts structured brand components and lets agents score their own outputs, converting taste from implicit prompt hint to verifiable interface.

Angle

Architecture argument for structured context over long system prompts.

Source video

The Four Pillars of Taste Engineering

Subjective quality problems are tractable once decomposed into repetition, fit, intent, and judgment, each with a distinct mechanism; teams should instrument these axes separately instead of asking one model 'is this good?'

Angle

Framework article with concrete failure-axis instrumentation.

Source video

When Generation Is Free, Build the Gatehouse

As generation cost approaches zero, the scarce resource is judgment and verification; the strategic investment shifts from better generators to runtime constraint layers, probes, and brand APIs that gate outputs.

Angle

Strategic/architectural argument about value migration to verification.

Source video

Project ideas

4

ProbeGate

beyond-evals

A lightweight pattern-based slop probe can match or exceed a tuned LLM judge on held-out designer-approved/rejected web designs at a fraction of cost and latency.

Proof of concept

Collect labeled design samples, extract surface features (type pairing, palette, framework traces), train baby classifier, compare AUC against LLM judge on same split; instrument false positives on unusual-but-good designs.

Measurement

AUC, precision/recall at operating threshold, cost per 1k evaluations, latency, and false-positive rate on novelty set.

Source video

BrandAPI Runtime Guard

gatehouse

Providing agents with structured brand context fetched at inference time increases brand adherence and reduces generic/slop outputs more than an equivalent prose style prompt.

Proof of concept

Build URL→structured brand schema extractor; agent generates landing pages/pitch decks with prose prompt vs Brand API context; score outputs with probes and human raters.

Measurement

Brand adherence rubric score, slop probe score, human preference, latency overhead per request.

Source video

Pillar Transfer Test

movement-lab

The repetition/fit/intent/judgment decomposition transfers to at least one non-design subjective domain (e.g., UX copy or code style), with probes achieving AUC > 0.70 against human labels.

Proof of concept

Define domain-specific probes for each pillar, collect small labeled dataset, train/evaluate classifiers, compare with LLM judge baseline.

Measurement

AUC per pillar, inter-pillar correlation, failure attribution accuracy, cost.

Source video

ProbeReward Loop

beyond-evals

Using slop probes as a rejection-sampling or reward signal during generation improves judged aesthetic quality without collapsing diversity.

Proof of concept

Generate N candidate outputs, rank/filter by probe score, compare baseline vs filtered outputs on quality and diversity metrics.

Measurement

Probe score, human preference, output diversity (distinct n-grams/embeddings), mode-collapse indicators.

Source video

Architectural implications

3

Quality/brand constraints are implemented as runtime agent tools rather than baked into the model or prompt.

Before

Brand and style guidance lives in system prompts as prose, with no structured verification.

After

Brand guidelines are exposed as a structured API that agents query and score outputs against at inference time.

Consequence

Constraints become composable, inspectable, versionable artifacts at the app layer, decoupled from any specific model.

Source video

A dedicated lightweight classifier layer sits between generation and delivery to gate slop.

Before

Generation output is delivered directly; quality checked (if at all) by a large LLM judge or humans.

After

Small pattern-based probes score outputs as a cheap pre/post-filter in the production pipeline.

Consequence

Adds an evaluation stage to the pipeline architecture, with latency/cost tradeoffs versus LLM judging.

Source video

Subjective quality is factored into orthogonal pillars rather than one monolithic judge.

Before

Single evaluation call/score for 'is this good?'.

After

Separable axes (repetition, fit, intent, judgment) each instrumented by a distinct mechanism.

Consequence

Failures become attributable and independently fixable; orchestration must route outputs to multiple specialized checkers.

Source video

Tradeoffs and failure modes

3

Pattern-based slop detection (probes)

Benefit

Cheap, fast, and reportedly more discriminative than LLM judges for this task (AUC 0.74).

Cost or risk

Pattern matching encodes past norms, so it may penalize legitimate novelty and can be brittle to distribution shift; an AUC of 0.74 implies substantial misclassification.

Slop probes separate designer-approved from designer-rejected sites at AUC 0.74, outperforming many LLM judge methodologies.
Open source video
Source video

Inference-time constraint enforcement

Benefit

Applies judgment where user interaction and intent actually exist.

Cost or risk

Adds per-request latency and cost for the extra scoring/verification calls, and requires a reliable brand-context source at request time.

Solving slop at inference time is crucial because that is when user interaction and intent happen.
Open source video
Source video

Encoding taste into structured APIs

Benefit

Makes brand adherence machine-verifiable and improves output fidelity.

Cost or risk

Structured brand extraction is a lossy compression of tacit design judgment; whatever the schema omits can't be enforced.

Providing structured brand context dramatically improves the fidelity and aesthetic quality of agent outputs.
Open source video
Source video

Open questions

4

How can human judgment be scaled within automated agent pipelines rather than being a manual bottleneck?

Why unresolved

The summary explicitly poses this as an open question without providing a mechanism.

Research direction

Investigate human-in-the-loop sampling strategies, active learning to retrain probes from designer labels, or hybrids where humans label only disagreement cases.

Source video

Which other subjective domains beyond design can be decoded into structured APIs and measurable pillars?

Why unresolved

Only design is evidenced (2.4M websites); generalizability is asserted but untested.

Research direction

Attempt the repetition/fit/intent/judgment decomposition on writing, code style, UX copy, or music and measure whether probes achieve comparable discriminative power.

Source video

Can probe outputs serve as verifiable reward/verification signals for training or RL of generative models, not just post-hoc filtering?

Why unresolved

The summary describes probes as measurement/detection and judgment gates but not as training signal.

Research direction

Test probes as reward models or rejection-sampling filters in a generation loop and measure downstream quality change.

Source video

What is the failure profile of pattern-based probes on genuinely novel-but-good outputs?

Why unresolved

AUC 0.74 is reported but false-positive behavior against legitimate novelty is not characterized.

Research direction

Construct an adversarial/unusual-but-good eval set and measure probe false-positive rate versus human approval.

Source video

Key claims

6
comparativeVerification needed

Slop probes separate designer-approved from designer-rejected sites at AUC 0.74.

Evidence

The slop probe separates designer-approved from designer-rejected sites at AUC 0.74.

Question

What dataset size, labeling protocol, and held-out split produced AUC 0.74, and how does it compare under identical conditions to LLM judges?

Source video
comparativeVerification needed

Probes outperform many LLM judge methodologies for slop detection.

Evidence

Slop probes separate designer-approved from designer-rejected sites at AUC 0.74, outperforming many LLM judge methodologies.

Question

Which LLM judge setups were benchmarked, and were they tuned fairly (prompt, model, calibration)?

Source video
factualVerification needed

Over 2.4 million websites from the past 10 years were analyzed to mine patterns.

Evidence

Analyzing 2.4 million websites from the past 10 years and comparing them against thousands of AI-generated sites.

Question

What was the sampling frame and how were the AI-generated comparison sites generated?

Source video
predictionVerification needed

As the cost of generation goes to zero, judgment becomes more valuable than ever.

Evidence

The cost of generation is going to zero, making judgment more valuable than ever.

Question

Can the rising value of judgment be measured empirically (e.g., pricing, demand for verification tooling) rather than asserted?

Source video
causalVerification needed

Inference time is the correct place to solve slop because that is when user interaction and intent happen.

Evidence

Solving slop at inference time is crucial because that is when user interaction and intent happen.

Question

How does inference-time enforcement compare in output quality and cost against training-time or fine-tuning-based approaches?

Source video
causalVerification needed

AI significantly accelerates pattern repetition and lack of fit across digital domains.

Evidence

AI significantly accelerates pattern repetition and lack of fit across digital domains.

Question

What quantitative evidence shows AI-generated sites are more homogeneous than human-made ones over time?

Source video

Connections

4