AI Engineer · Published 2026-10-05

Build the Right Thing: Product Engineering (Part 1) — Kent C. Dodds, EpicProduct.engineer

privateReview: unreviewedConfidence: high
product engineeringAI agentsthe mom testvalidationarchitectureagentic-systemsproduct-engineeringfailure-modesfeedback-loopscost-of-changeHuman-in-the-loop
Open on YouTube ↗

Summary

Overview

  • Speaker: Kent C. Dodds
  • Channel: AI Engineer
  • Main topic: Product engineering for software developers in the age of AI agents
  • Purpose: To help software developers and engineers transition from pure implementers to product engineers who can navigate AI agents, apply product judgment, and avoid building the wrong thing efficiently. Kent C. Dodds explores how AI agents level the implementation playing field, making software implementation cheap, fast, and widely available. As writing code becomes commoditized, the critical differentiator shifts to product engineering and product judgment—knowing what to build, validating early using frameworks like The Mom Test, and taking end-to-end responsibility for customer value rather than just closing tickets.

Topic Map

The Shift in Software Engineering Due to AI

  • Explanation: AI coding agents are eating into the work of software engineers by making implementation drastically faster and cheaper.
  • Key claims:
    • AI agents level the implementation playing field.
    • When implementation gets cheap, ideas and product judgment matter more.
    • The danger is confidently building the wrong thing faster than ever.
  • Examples:
    • Real-looking prototypes can be built in minutes, hiding scalability and edge case issues.
  • Terminology:
    • AI agents
    • Prompt engineering
    • Loop engineering
    • Flow engineering
  • Why it matters: Engineers risk becoming highly efficient at writing 10,000 lines of code a day for the wrong features unless they develop product judgment.

Product Engineering vs. Pure Implementation

  • Explanation: Product engineering requires bridging technical expertise with deep customer understanding and accountability for real-world outcomes.
  • Key claims:
    • Product engineering is not engineers becoming PMs; it is using product judgment to shape technical investment.
    • The engineering shape is part of the product decision.
    • Architecture is where change gets expensive.
  • Examples:
    • Bob Martin's phone repair truck story: experiencing the customer's actual context changes how you build software.
  • Terminology:
    • Product engineering
    • Architecture
    • Primitives
    • Distribution
  • Why it matters: Understanding customer context prevents engineers from faithfully building the wrong abstraction.

Early Idea Validation and The Mom Test

  • Explanation: Validating product ideas with real users before writing code prevents building things nobody wants.
  • Key claims:
    • Don't ask users to evaluate your idea or diagnose the problem for you.
    • Ask about specific past behavior, workarounds, cost, and frequency.
    • Existing effort and money spent are stronger evidence than stated interest.
  • Examples:
    • Wayne Allan's story of building for all of Australia for a year without validation and getting zero users.
  • Terminology:
    • The Mom Test
    • Validation theater
    • Cost of inaction
    • Workarounds
  • Why it matters: Getting early user feedback closes the loop and turns shipping into learning.

Key Points

AI agents accelerate bad practices as well as good ones

  • Explanation: Because coding agents make it extremely easy to execute tasks, engineers can quickly scale up poorly thought-out features and system bloat.
  • Evidence: Quotes from Dax Raad, Shaundal Person, and Rita Kozlov on how prototypes look real instantly.
  • Practical implication: Engineers must slow down, be intentional, and maintain strict constraints and feedback loops.

The scarce resource has shifted from implementation to judgment

  • Explanation: Implementation is becoming a commodity; deciding whether work is worth building in the first place is now the most valuable skill.
  • Evidence: Comparison of 'Can we build it?' versus 'Is it worth building?'
  • Practical implication: Engineers need deep product sense and domain understanding to make the right technical trade-offs.

User error points to design failure

  • Explanation: Borrowing from Don Norman's design principles, when users struggle or make mistakes with software, it is a system design flaw, not user error.
  • Evidence: Reference to Don Norman and the Three Mile Island operator disaster.
  • Practical implication: Engineers must understand real user context to design foolproof systems.

Frameworks, Models & Processes

The Mom Test

  • How it works: A framework for customer discovery that relies on asking about past behavior rather than hypothetical future interest.
  • Components:
    • Talk about their life instead of your idea
    • Ask about specifics in the past rather than generic opinions
    • Look for actual workarounds and money spent
  • When to use: During early idea validation and when scoping new features.

Product Engineering Balance

  • How it works: Balancing technical implementation with product context and ownership.
  • Components:
    • Data model design
    • Workflow shape
    • Observability
    • Constraints and failure modes
    • Smallest useful slice
  • When to use: When translating product requirements into technical architecture.

Examples & Case Studies

Wayne Allan's team spent a year building a high-scale microservices architecture for all of Australia.

  • Illustrates: Skipping early validation and building without testing market demand.
  • Lesson: Built for all of Australia, nobody came. You should test the market in weeks with manual integrations before committing a whole team.

Three Mile Island nuclear operators made critical errors during a crisis.

  • Illustrates: That intelligent people make mistakes when the system design is flawed.
  • Lesson: User error does not exist; bad system design does.

Actionable Takeaways

  • Immediate:
    • Stop asking users 'Would you use this?' and start asking 'When was the last time this happened?'
    • Evaluate whether features are worth building before letting AI agents code them rapidly.
  • Strategic:
    • Develop durable product engineering skills that combine technical depth with user empathy.
    • Treat architecture as high-leverage design decisions where change is expensive.
  • Questions to investigate:
    • How do technical choices impact customer outcomes?
    • What are the cost and frequency of user workarounds for the problem you are solving?

Claims Worth Verifying

  • Don Norman invented the term user experience at Apple. (historical fact)
  • Three Mile Island accident occurred in 1979 in Pennsylvania due to operator error compounded by bad system design. (historical fact)

Notable Quotes

"When AI agents level the implementation playing field, the differentiator becomes building the right thing." (at 18:21) "The default place for our new coding agent abilities to go is to work on the wrong things." (at 30:35) "A product engineer lives half in the technology and half in the customer's house." (at 40:04) "Don't ask users to evaluate your idea or diagnose the problem for you." (at 67:27) "The user feedback is what closes the loop... this solves my problem, or this is not worth pursuing." (at 81:57)

Compressed Summary

  • AI agents make software implementation cheap and fast
  • Product judgment and deciding what to build are now the primary differentiators
  • Use The Mom Test to validate ideas through past user behavior, not opinions
  • Engineers must bridge technical architecture with real customer context
  • Keywords: product engineering, ai agents, the mom test, validation, architecture
  • Core insight: As AI agents commoditize implementation, the ultimate skill for software engineers is product judgment—building the right thing rather than just building things fast.

Core insights

6
Failure Modemedium noveltymoderate evidence

Agent capability has a default gravitational pull toward the wrong work: because agents make execution trivially cheap, the unattended default is to rapidly build things that shouldn't exist. Guardrails are therefore not a safety afterthought but a first-class constraint on what agents are allowed to run.

Why it matters

Harnesses that only optimize for execution speed will amplify bad decisions at machine throughput; explicit worthiness gates and feedback loops must sit before agent execution, not after.

Generalization

Any system that lowers the cost of an action shifts the bottleneck to deciding whether the action is worth taking; autonomous executors need pre-execution judgment gates proportional to their speed.

The default place for our new coding agent abilities to go is to work on the wrong things.
Open source video
Real-looking prototypes can be built in minutes, hiding scalability and edge case issues.
Open source video
Engineers must slow down, be intentional, and maintain strict constraints and feedback loops.
Open source video
Architecturemedium noveltymoderate evidence

Architecture is the site where change is expensive, and the engineering shape is itself a product decision, not a downstream implementation detail. Technical investment choices (data model, workflow shape, observability, failure modes) determine which product changes remain cheap.

Why it matters

If architecture is scoped as 'just implementation', teams defer the decisions that lock in future cost; treating boundaries as product decisions moves the highest-leverage choices earlier.

Generalization

Boundaries and abstractions should be evaluated by their cost-of-change profile, not by internal code quality alone.

The engineering shape is part of the product decision.
Open source video
Architecture is where change gets expensive.
Open source video
Mechanismmedium noveltystrong evidence

Validation evidence should be weighted by revealed behavior rather than stated intent: specific past incidents, existing workarounds, and money already spent are stronger signals than interest or opinion. Asking users to evaluate an idea or diagnose the problem produces noise.

Why it matters

Gives a concrete, falsifiable evidence hierarchy for deciding whether to build, replacing subjective enthusiasm with observable proxies (frequency, cost, workarounds).

Generalization

Prefer revealed-behavior metrics over self-reported preferences whenever deciding whether to invest effort — applicable to agent evaluation and feature prioritization alike.

Don't ask users to evaluate your idea or diagnose the problem for you.
Open source video
Existing effort and money spent are stronger evidence than stated interest.
Open source video
Look for actual workarounds and money spent
Open source video
Mental Modelmedium noveltymoderate evidence

'User error' is a misattribution; failures by competent operators trace back to flawed system design. Attribution of failure should move from the operator to the system.

Why it matters

In agent systems this reframes error handling: repeated failures by a capable agent are a harness/runtime design defect, not a model deficiency to be blamed and ignored.

Generalization

Post-incident attribution should target system design and observability gaps rather than the actor, whether that actor is a human operator or an agent.

User error does not exist; bad system design does.
Open source video
That intelligent people make mistakes when the system design is flawed.
Open source video
Practicelow noveltymoderate evidence

Validate the smallest useful slice in weeks using manual integrations before committing a full team to a scaled architecture. Manual, low-scale operation is a deliberate probe of demand rather than a temporary hack.

Why it matters

Offers an explicit sequencing rule: prove demand at small scale and low cost before architecture investment, reversing the common build-for-scale-first pattern.

Generalization

When uncertainty is about demand rather than feasibility, defer scalable architecture and prove the slice with the cheapest possible mechanism first.

Built for all of Australia, nobody came. You should test the market in weeks with manual integrations before committing a whole team.
Open source video
Practicemedium noveltymoderate evidence

The loop between shipping and learning is closed only by real user feedback that yields a binary verdict: this solves my problem, or it isn't worth pursuing. Shipping without that loop is motion, not progress.

Why it matters

Defines a terminal condition for prototyping — a decision, not a deliverable — which is what makes iterative agent-assisted building productive rather than merely fast.

Generalization

Iteration loops need an explicit decision output; a loop that only produces artifacts cannot self-correct.

The user feedback is what closes the loop... this solves my problem, or this is not worth pursuing.
Open source video
A product engineer lives half in the technology and half in the customer's house.
Open source video

Deep dives

5

Automatable worthiness gates placed before agent execution

Research question

What is the minimal, machine-checkable pre-execution gate (smallest-useful-slice check, failure-mode check, evidence-of-demand check) that measurably reduces agent output later judged not worth building, without reintroducing the human bottleneck agents were meant to remove?

Why

Agent harnesses currently optimize throughput of implementation; if the default destination of cheap execution is the wrong work, the gate is the load-bearing component, not the model. Without a measurable gate design, teams can only add ad-hoc human review that erases the speed advantage.

The danger is confidently building the wrong thing faster than ever.
Open source video
Real-looking prototypes can be built in minutes, hiding scalability and edge case issues.
Open source video
AI agents accelerate bad practices as well as good ones
Open source video
Source video

Cost-of-change measurement per architectural boundary

Research question

Can cost-of-change be quantified per architectural boundary (data model, workflow shape, failure modes, observability) so that boundary choices are defended with measurements rather than code-quality arguments?

Why

The claim that architecture is where change gets expensive is asserted but never instrumented. If cost-of-change is measurable, architecture stops being deferred internal work and becomes a scoping input with evidence attached.

Architecture is where change gets expensive.
Open source video
The engineering shape is part of the product decision.
Open source video
Source video

Revealed-behavior demand proxies and their relative predictive weight

Research question

Which observable pre-build proxy — workaround frequency, money already spent, or cost of inaction — best predicts later adoption, and at what relative weighting?

Why

The summary supplies candidate signals but no measurement protocol, so 'validation' remains a vibe. A weighted proxy set converts the build/don't-build decision into a falsifiable prediction that can be scored against outcomes.

Don't ask users to evaluate your idea or diagnose the problem for you.
Open source video
Ask about specific past behavior, workarounds, cost, and frequency.
Open source video
Existing effort and money spent are stronger evidence than stated interest.
Open source video
Source video

Diagnostic separation of harness defects from model limits

Research question

Which observability signals — repeated failure patterns, missing affordances, ambiguous state — reliably distinguish a harness/runtime design defect from a genuinely incapable agent?

Why

If failure attribution stays on the actor, the fix is prompt tweaking instead of redesign, and the same failure recurs. A diagnostic procedure is required before 'the model can't do this' becomes an excuse.

The danger is confidently building the wrong thing faster than ever.
Open source video
AI agents accelerate bad practices as well as good ones
Open source video
Source video

Manual-slice-first sequencing versus scale-first builds

Research question

Does proving demand with manual integrations in weeks produce faster time-to-signal than committing a team to full-scale architecture first, and does this hold outside consumer products?

Why

The supporting evidence is a single narrative case. If the sequencing rule fails to generalize to infrastructure and developer tooling, then the rule needs a scope condition rather than being stated universally.

Built for all of Australia, nobody came. You should test the market in weeks with manual integrations before committing a whole team.
Open source video
Source video

Article ideas

4

Faster Failure: Why Cheap Agent Implementation Moves the Bottleneck Instead of Removing It

When implementation cost collapses, the risk profile does not improve — it relocates. The scarce resource becomes the judgment to decide what is worth building, and any pipeline that does not gate execution simply reaches the wrong outcome at machine throughput.

Angle

Argue against the 'agents remove the constraint' narrative by showing that the constraint moved upstream, and give the engineering consequence: worthiness gates, constraints, and feedback loops are load-bearing components.

Source video

Architecture Is a Product Decision, Not Implementation Detail

Boundaries are not internal cleanliness questions; they determine which product changes remain cheap. Treating architecture as a downstream implementation detail defers the highest-leverage decision until it is frozen, so boundary choices should be made and defended on a cost-of-change profile.

Angle

Reframe architecture reviews as product trade-off discussions, with cost-of-change as the shared metric between engineering and product.

Source video

Validation Theater: Why Asking People About Your Idea Produces Noise

Stated intent is the weakest available evidence. Specific past incidents, existing workarounds, and money already spent are the only signals worth acting on, which makes most 'user validation' a ritual that confirms the builder's enthusiasm.

Angle

Give an evidence hierarchy with observable proxies and show how to score a build/don't-build decision before writing code.

Source video

There Is No User Error: Failure Attribution as a System Design Function

Attributing failure to the operator — human or agent — suppresses the observability and design work that would prevent recurrence. Competent operators fail on flawed systems, so post-incident attribution must target harness, interface, and state design.

Angle

Take the operator-error analogy into agent runtimes: repeated failures by a capable agent are a runtime defect, and the fix budget belongs in redesign, not prompt tuning.

Source video

Project ideas

4

Worthiness Gate

gatehouse

Agent pipelines that pass every task through a machine-checkable pre-execution gate (smallest-useful-slice defined, cost-of-inaction stated, failure modes enumerated) produce a lower fraction of output later judged not worth building, with no significant loss in task throughput versus an ungated pipeline.

Proof of concept

Wrap an existing coding-agent pipeline with a blocking gate stage that scores each proposed task against three checks and records the verdict; run the same task queue gated and ungated and track downstream outcomes.

Measurement

Percentage of shipped agent-generated changes reverted, deprioritized, or abandoned within 30 days, plus tasks completed per week in gated versus ungated runs.

Source video

Failure Attribution Instrument

beyond-evals

Repeated-failure patterns, missing affordances, and ambiguous state signals can classify agent failures as harness defects versus model limits with inter-rater agreement above chance, and the classification predicts which fix (redesign vs. retrain/prompt) actually resolves the failure.

Proof of concept

Instrument an agent runtime to log failure traces and surrounding state, label a sample of failures with the diagnostic signals, then verify whether the predicted fix resolves the failure on replay.

Measurement

Agreement between instrument-assigned attribution and later confirmed resolution path; recurrence rate of failures after each fix type.

Source video

Manual Slice Probe

new

Proving demand with manual integrations before any scale architecture yields a faster time-to-signal than scale-first builds, and the advantage holds for developer tooling and infrastructure products, not only consumer products.

Proof of concept

Run paired rollouts in at least two product categories: one manual-slice-first, one build-to-scale-first, with the same demand-evidence instrument applied to both.

Measurement

Days (or weeks) to first binary verdict from real users, and total effort spent before the verdict was reached.

Source video

Cost-of-Change Map

movement-lab

Cost-of-change can be measured per architectural boundary by counting the modules and product surfaces touched by a defined change class, and these measurements predict which boundaries teams later report as expensive.

Proof of concept

Pick a codebase, define three change classes (new field, new workflow step, new failure-mode handling), measure blast radius per boundary, and compare the ranking against delayed-change and rework reported by the team over one quarter.

Measurement

Correlation between measured blast radius per boundary and observed rework/delay episodes attributed to that boundary.

Source video

Architectural implications

3

As implementation cost collapses, the binding constraint moves from can-we-build to should-we-build, so a worthiness gate becomes part of the execution pipeline.

Before

Agent pipeline optimized end-to-end for throughput of implementation work.

After

Agent pipeline includes a pre-execution scoping/judgment stage and strict constraints on what is allowed to be built.

Consequence

Constraints, feedback loops, and observability become load-bearing components rather than optional tooling.

Source video

Architecture is treated as a product decision because it sets where change is expensive.

Before

Architecture classified as internal, deferrable, engineering-only concern.

After

Architecture treated as a product-level trade-off made jointly with customer-value understanding.

Consequence

Data model, workflow shape, failure modes, and smallest useful slice become scoping inputs, not post-scoping implementation details.

Source video

Failure attribution shifts from the actor (user/agent) to the system design.

Before

Errors attributed to operator or agent mistakes and handled by retraining or prompt tweaks.

After

Errors traced to harness, interface, and observability design defects.

Consequence

Investment moves from blaming/tuning the actor to redesigning the surrounding system.

Source video

Tradeoffs and failure modes

3

Cheap, fast agent-assisted implementation

Benefit

Implementation becomes fast, cheap, and widely available; prototypes appear in minutes.

Cost or risk

Confidently building the wrong thing faster than ever, with real-looking prototypes hiding scalability and edge-case problems.

The danger is confidently building the wrong thing faster than ever.
Open source video
Source video

Skipping early validation to build at scale

Benefit

Full-scale architecture is ready if demand materializes.

Cost or risk

A year of effort and full-team commitment with zero users; demand was never actually tested.

Built for all of Australia, nobody came.
Open source video
Source video

Asking users about your idea

Benefit

Fast, low-effort signal that feels like validation.

Cost or risk

Produces validation theater — stated interest and opinions that are weaker evidence than actual workarounds or spending.

Don't ask users to evaluate your idea or diagnose the problem for you.
Open source video
Source video

Open questions

4

How can a worthiness gate be operationalized inside an agent execution pipeline without reintroducing the human bottleneck agents were meant to remove?

Why unresolved

The summary asserts judgment is now the scarce resource and that agents default to the wrong work, but offers no mechanism for encoding or automating that gate.

Research direction

Prototype pre-execution constraint/checklist layers (e.g., smallest-useful-slice and failure-mode checks) and measure whether they reduce wasted agent output.

Source video

Which observable proxies best predict demand — workaround frequency, money already spent, or cost of inaction — and how are they measured before building?

Why unresolved

The summary lists candidate signals but gives no measurement protocol or relative weighting.

Research direction

Design a small study instrumenting cost/frequency of existing workarounds and compare its predictions against later adoption.

Source video

How should failures be attributed between system design and actor so that a design flaw is distinguishable from a genuinely incapable agent?

Why unresolved

The claim that user error does not exist is asserted via a nuclear-operator analogy without a diagnostic procedure.

Research direction

Define observability signals (repeated failure patterns, missing affordances, ambiguous state) that distinguish harness defects from model limits.

Source video

Does measuring the smallest useful slice with manual integrations actually generalize beyond consumer products to infrastructure and developer tooling?

Why unresolved

The supporting evidence is a single case (Wayne Allan's team) presented as narrative rather than controlled result.

Research direction

Compare manual-slice-first versus build-to-scale-first rollouts across multiple product categories with time-to-signal as the metric.

Source video

Key claims

8
causalVerification needed

AI agents level the implementation playing field, making implementation cheap and widely available.

Evidence

AI agents level the implementation playing field.

Question

What measurable reduction in implementation cost/time do coding agents produce across comparable teams?

Source video
causalVerification needed

When implementation gets cheap, ideas and product judgment matter more as the differentiating skill.

Evidence

When implementation gets cheap, ideas and product judgment matter more.

Question

Does the marginal value of product judgment measurably increase as implementation cost falls?

Source video
predictionVerification needed

The default outcome of new agent capabilities is working on the wrong things.

Evidence

The default place for our new coding agent abilities to go is to work on the wrong things.

Question

What fraction of agent-generated output in a typical pipeline targets work later judged not worth building?

Source video
opinionVerification needed

Architecture is where change gets expensive, so the engineering shape is part of the product decision.

Evidence

Architecture is where change gets expensive.

Question

Can cost-of-change be measured per architectural boundary to validate which shapes are expensive?

Source video
opinionVerification needed

User error does not exist; failures reflect bad system design.

Evidence

User error does not exist; bad system design does.

Question

Are apparent operator errors reliably traceable to identifiable system-design deficiencies in post-incident analysis?

Source video
causalVerification needed

Building at full scale without early validation can yield zero users after a year of effort.

Evidence

Built for all of Australia, nobody came. You should test the market in weeks with manual integrations before committing a whole team.

Question

What is the observed failure rate of scale-first builds versus manual-slice-first builds in reaching initial users?

Source video
factualVerification needed

Don Norman invented the term user experience at Apple.

Evidence

Don Norman invented the term user experience at Apple.

Question

Is there a verifiable primary source attributing the coining of 'user experience' to Don Norman at Apple?

Source video
factualVerification needed

Three Mile Island (1979, Pennsylvania) resulted from operator error compounded by bad system design.

Evidence

Three Mile Island accident occurred in 1979 in Pennsylvania due to operator error compounded by bad system design.

Question

Do official investigations attribute the accident primarily to design-induced operator error rather than operator incompetence?

Source video

Connections

4