AI Native Dev · Published 2026-09-22

Dexter Horthy: Why We Stopped Trusting AI to Write the Plan

privateReview: unreviewedConfidence: high
agentic codingContext engineeringSoftware FactorySpec-driven developmentLLM workflowsagentic-codingcontext-engineeringspec-driven-developmentsoftware-factoryback-pressurecontext-window-limitsdeterministic-guardrailsHuman-in-the-loop
Open on YouTube ↗

Summary

Overview

  • Speaker: Dexter Horthy
  • Channel: AI Native Dev
  • Main topic: Agentic software development, planning versus executing with AI coding agents, and the necessity of human-in-the-loop context engineering.
  • Purpose: To educate AI-native developers and engineering leaders on the limitations of autonomous AI planning, the value of specs and context engineering, and how to maintain software velocity without accumulating technical debt through agentic coding. Dexter Horthy, founder of HumanLayer, discusses why engineering teams moved away from letting AI agents write and execute entire plans autonomously ('the plan mode trap'). After running experiments with Opus 4 and cloud agent SDKs, Horthy's team realized that letting LLMs generate large, unverified implementation plans leads to regression, unusable codebases, and 'the dumb zone' where context windows degrade. They rebuilt their product around strict human-led context engineering, specs as the new source of code, and compounding feedback loops.

Topic Map

The Experiment with Autonomous AI Planning

  • Explanation: Horthy shares an experiment where his team ran pull requests generated by AI plans for 4-5 months before realizing the codebase became un-navigable and slow due to regression.
  • Key claims:
    • AI-generated plan files containing full inline code implementations led to severe regression and code rot.
    • Codebases became unusable after a few months of un-reviewed agentic execution.
  • Examples:
    • Rebuilding the entire HumanLayer product from scratch after codebases became unmaintainable.
  • Terminology:
    • plan mode
    • inline diffs
    • regression
  • Why it matters: Demonstrates that un-reviewed AI planning introduces cascading technical debt that negates the productivity gains of AI.

The Spec Is the New Code

  • Explanation: Drawing on parallels with assembly language and compilation, Horthy argues that executable specifications will replace hand-written code as the primary artifact developers manage.
  • Key claims:
    • Specifications are verifiable, executable, and compile into code.
    • Checking in code directly will be replaced by checking in specifications and intent prompts.
  • Examples:
    • Comparing Java code generation and compilation to spec-driven agentic workflows.
  • Terminology:
    • spec
    • compilation
    • verification
  • Why it matters: Shifts developer focus from syntax and implementation details to intent and architecture definition.

The 'Dumb Zone' and Context Window Degradation

  • Explanation: Discussion around token limits and context window degradation when AI agents process overly large prompts or long unstructured conversations.
  • Key claims:
    • Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.
    • Context engineering requires intentional chunking and decision trees rather than dumping entire codebases into an LLM.
  • Examples:
    • Writing 100k-token prompt plans that resulted in vastly inferior code quality.
  • Terminology:
    • dumb zone
    • context window
    • token limits
  • Why it matters: Highlights technical constraints in LLM architectures that require strategic human management.

Compounding Engineering and Back-Pressure

  • Explanation: Examining how software factories improve over time by applying rigorous human feedback and constraints to agent outputs.
  • Key claims:
    • Software factories need continuous feedback loops and guardrails to prevent agentic degradation.
    • Writing detailed specs up front saves hours of review time and prevents bad patterns from proliferating.
  • Examples:
    • Using deterministic linters and static analysis (like React Doctor or Oxlint) as automated agent guardrails.
  • Terminology:
    • compounding engineering
    • back-pressure
    • software factory
  • Why it matters: Provides a mental model for sustainable scaling of AI-assisted engineering organizations.

Key Points

Plan before you build

  • Explanation: Spending time upfront on clear specifications and intent prompts saves hours of painful code review and rework later.
  • Evidence: Expected pain formula: P(you'll have to change it) * how painful the change is.
  • Practical implication: Engineers should invest 3-5 minutes writing structured specs rather than launching blind two-sentence prompts.

Avoid over-planning and balance leverage

  • Explanation: While planning is essential, over-micromanaging AI agents defeats the purpose of automation. Find the sweet spot between autonomy and control.
  • Evidence: Graph showing chance of rework vs time/effort spent upfront.
  • Practical implication: Establish clear decision trees and guardrails without writing every line of code by hand.

Taste and intent cannot be fully automated

  • Explanation: Human taste, preferences, and security policies act as differentiators that must be codified into agent constraints.
  • Evidence: Discussions on GitHub repo stars and developer preferences for code architecture.
  • Practical implication: Teams must define explicit coding standards and review policies for their AI agents.

Frameworks, Models & Processes

Expected Pain Formula

  • How it works: Calculates the anticipated friction of software changes based on the probability of rework and the friction magnitude of making those changes.
  • Components:
    • P(you'll have to change it)
    • how painful the change is
  • When to use: When deciding how much upfront planning or specification is needed before unleashing an AI coding agent.

Software Factory Leverage Loop

  • How it works: Balances the time spent planning, building, and reviewing code in an agentic development environment.
  • Components:
    • Planning / Architecture proposal
    • Agentic building
    • Review / Human feedback
    • CI/CD and monitoring
  • When to use: Structuring engineering workflows around AI coding tools.

Examples & Case Studies

Horthy's co-founder Kyle spent two weeks typing every character by hand in VS Code to redesign the data architecture for a new software version.

  • Illustrates: The necessity of manual structural definition when transitioning away from corrupted AI codebases.
  • Lesson: Sometimes foundational refactoring requires direct human control over system boundaries.

Using deterministic linters and static analysis tools as automated reviewers for agent pull requests.

  • Illustrates: Automated back-pressure enforcement in a software factory.
  • Lesson: Combine LLM flexibility with deterministic linters to catch bad patterns early.

Actionable Takeaways

  • Immediate:
    • Stop relying on un-reviewed multi-thousand-token prompt files for feature implementation.
    • Introduce automated linters and static analyzers to check agent outputs.
    • Spend 30 minutes on a structured spec before building to save hours of review.
  • Strategic:
    • Shift engineering focus from writing syntax to designing executable specifications.
    • Build compounding feedback loops to improve agent output quality over time.
    • Treat AI agents as junior developers that require structured guardrails and context.
  • Questions to investigate:
    • How do we balance agent autonomy with architectural consistency?
    • What is the optimal context window size for specialized coding tasks?
    • How will developer taste and preferences be codified in future AI workflows?

Claims Worth Verifying

  • GPT-5.5 achieved a 14.8% benchmark score on Astra. (benchmark statistic)
  • GrillMe is in the top 25 stars on GitHub repos for skills. (metric claim)

Notable Quotes

"We wanted to be living in the future. And so we said, cool, let's try this." (at 0:00) "Plan before you build. 30 minutes up front saves hours during review." (at 28:21) "Don't Over-Plan. Seek leverage." (at 30:15)

Compressed Summary

  • Autonomous AI planning without verification leads to code rot and regression.
  • Specifications and intent prompts are replacing raw hand-written code as the core developer artifact.
  • Context engineering and back-pressure are critical for managing large token windows and maintaining code quality.
  • Compounding engineering loops ensure that AI agents improve system output over time rather than accumulating tech debt.
  • Keywords: agentic coding, context engineering, software factory, spec-driven development, llm workflows
  • Core insight: Successful agentic software development requires shifting from autonomous AI planning to rigorous human-led context engineering and executable specifications.

Core insights

6
Failure Modehigh noveltystrong evidence

Letting LLM-generated plans (plan files containing full inline code implementations) execute without human review compounds into codebase rot: after 4-5 months of un-reviewed agentic PRs the codebase became un-navigable and regression-heavy, forcing a full product rebuild.

Why it matters

It is an empirically observed terminal state for autonomous planning workflows, not a theoretical risk: the productivity gain of agentic execution can be fully negated by cascading technical debt, and the remediation cost (full rewrite) dwarfs the saved review time.

Generalization

Any system that lets an unchecked generative step write the artifact that later generation depends on accumulates correlated errors that eventually require discarding the accumulated artifact rather than patching it.

AI-generated plan files containing full inline code implementations led to severe regression and code rot.
Open source video
Codebases became unusable after a few months of un-reviewed agentic execution.
Open source video
Rebuilding the entire HumanLayer product from scratch after codebases became unmaintainable.
Open source video
Empirical Resulthigh noveltymoderate evidence

There is a 'dumb zone': prompts exceeding ~100,000 tokens cause significantly degraded model performance and output quality, so context engineering must chunk and structure intent into decision trees rather than dumping the whole codebase or a giant plan into the model.

Why it matters

It gives a concrete failure threshold that changes how context is assembled and how plan size is budgeted, and it reframes 'more context' as a quality regression rather than a capability win.

Generalization

Context is a finite-quality resource with a degradation regime; agent harnesses should treat context budget as a first-class design constraint and decompose tasks to stay below it.

Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.
Open source video
Writing 100k-token prompt plans that resulted in vastly inferior code quality.
Open source video
Context engineering requires intentional chunking and decision trees rather than dumping entire codebases into an LLM.
Open source video
Architecturehigh noveltymoderate evidence

The spec becomes the new code: executable, verifiable specifications compile into code, so the primary artifact checked into version control shifts from code to specs and intent prompts.

Why it matters

It relocates the abstraction boundary of software development: what gets versioned, reviewed, and diffed changes, which implies new tooling, review flows, and verification layers above the code level.

Generalization

As generative compilation matures, the durable, human-authored artifact moves up the abstraction stack (assembly -> high-level code -> spec), and verification tooling must move up with it.

Specifications are verifiable, executable, and compile into code.
Open source video
Checking in code directly will be replaced by checking in specifications and intent prompts.
Open source video
Mechanismmedium noveltymoderate evidence

Deterministic linters and static analyzers (e.g., React Doctor, Oxlint) are repurposed as automated reviewers/back-pressure on agent PRs, combining LLM flexibility with deterministic checks to catch bad patterns early.

Why it matters

It provides a concrete, cheap enforcement layer in a software factory: deterministic guardrails scale review without human bottleneck and are directly implementable in CI.

Generalization

Use deterministic, cheap verifiers as the first line of back-pressure on generative outputs; reserve expensive/semantic evaluation for what deterministic checks cannot cover.

Using deterministic linters and static analysis (like React Doctor or Oxlint) as automated agent guardrails.
Open source video
Using deterministic linters and static analysis tools as automated reviewers for agent pull requests.
Open source video
Combine LLM flexibility with deterministic linters to catch bad patterns early.
Open source video
Mental Modelmedium noveltyweak evidence

The Expected Pain Formula — P(you'll have to change it) * how painful the change is — is used as a decision heuristic to size upfront spec/planning effort before dispatching an agent.

Why it matters

It turns the vague 'plan more vs. plan less' debate into a computable expected-cost estimate that can be tuned per task, and it makes the case for spec investment quantitative rather than aesthetic.

Generalization

Expected-cost-of-rework heuristics can gate how much upfront structure any generative pipeline demands, generalizing to any human-in-the-loop generative workflow.

Expected pain formula: P(you'll have to change it) * how painful the change is.
Open source video
Tradeoffmedium noveltymoderate evidence

There is a leverage sweet spot between autonomy and control: chance of rework falls with upfront effort but over-micromanaging the agent defeats the purpose of automation, so the goal is decision trees and guardrails rather than hand-writing every line.

Why it matters

It warns that maximizing control is itself a failure mode; teams need a curve-based stopping criterion for how much structure to impose on an agent.

Generalization

Agentic systems exhibit diminishing and then negative returns to human-imposed structure; the right operating point is task-dependent and should be measured, not assumed.

Graph showing chance of rework vs time/effort spent upfront.
Open source video
Find the sweet spot between autonomy and control.
Open source video
Establish clear decision trees and guardrails without writing every line of code by hand.
Open source video

Deep dives

5

Generalizability of un-reviewed agentic planning codebase rot

Research question

Does un-reviewed plan-file-driven agentic execution reliably cause maintainability decay and full-rewrite outcomes across codebases, or is the HumanLayer outcome specific to plans embedding full inline code implementations?

Why

The central empirical failure mode is currently a single uncontrolled case study; if it generalizes, any software factory must gate plan artifacts and agent PRs, and if not, the failure is a tooling artifact that can be engineered around.

AI-generated plan files containing full inline code implementations led to severe regression and code rot.
Open source video
Codebases became unusable after a few months of un-reviewed agentic execution.
Open source video
Rebuilding the entire HumanLayer product from scratch after codebases became unmaintainable.
Open source video
Source video

Token-length quality degradation ('dumb zone') threshold

Research question

At what prompt token length does agent output quality measurably inflect downward, and is the threshold stable across model versions and task types?

Why

A stable threshold turns context budgeting into a concrete engineering constraint; an unstable one means context engineering must be model- and task-specific rather than a fixed 100k rule.

Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.
Open source video
Writing 100k-token prompt plans that resulted in vastly inferior code quality.
Open source video
Context engineering requires intentional chunking and decision trees rather than dumping entire codebases into an LLM.
Open source video
Source video

Executable specification verification as replacement for code-level review

Research question

Can specifications be made executable and verifiable enough that spec-level tests catch the regressions that code review would have caught, making specs the primary checked-in artifact?

Why

If specs cannot carry verification, the 'spec is the new code' claim becomes documentation and the review burden remains on generated code; if they can, it relocates review, diffing and CI above the code layer.

Specifications are verifiable, executable, and compile into code.
Open source video
Checking in code directly will be replaced by checking in specifications and intent prompts.
Open source video
Source video

Estimating Expected Pain Formula inputs from historical agent PRs

Research question

Can P(rework) and change cost be predicted per task class from historical agent PR features well enough to gate upfront spec/planning effort?

Why

The formula is only a heuristic without estimation methods; instrumenting past PRs would turn it into a tunable policy that decides when to invest in specs rather than guessing.

Expected pain formula: P(you'll have to change it) * how painful the change is.
Open source video
Source video

Leverage sweet spot between agent autonomy and human control

Research question

What is the shape of the curve between upfront human structure and chance of rework, and how can teams measure their operating point rather than assume it?

Why

It warns that maximizing control is a failure mode; without a measurable stopping criterion, teams will either under-constrain agents into code rot or over-constrain them into manual work.

Graph showing chance of rework vs time/effort spent upfront.
Open source video
Find the sweet spot between autonomy and control.
Open source video
Establish clear decision trees and guardrails without writing every line of code by hand.
Open source video
Source video

Article ideas

4

The Plan Mode Trap: Why We Stopped Trusting AI to Write the Plan

Autonomous AI planning is not a productivity multiplier but a debt-accumulation mechanism unless plan artifacts are treated as human-reviewed context, not executable code; teams that skip this rebuild their codebase.

Angle

Case-based warning with a concrete remediation: human-led specs, deterministic guardrails, and context budgets.

Source video

The Spec Is the New Code—Only If It Compiles

Specs become the primary artifact only when they are executable and verifiable; otherwise spec-driven development is just documentation with extra steps.

Angle

Architectural argument: move review, diffing and CI to the spec layer, with verification as the hard requirement.

Source video

Your Context Window Is a Quality Budget, Not a Storage Unit

More context is not more capability; past a measurable token threshold, output quality degrades, so context engineering must chunk intent into decision trees and treat token budgets as design constraints.

Angle

Counter-intuitive engineering constraint backed by a concrete threshold claim and practical decomposition strategies.

Source video

Back-Pressure for Agents: Deterministic Linters as Your First Reviewers

Human review cannot scale with agent output; deterministic linters and static analysers should be the first gate on agent PRs, reserving semantic review for what deterministic checks cannot cover.

Angle

CI-as-enforcement-surface playbook for the software factory.

Source video

Project ideas

4

Agent PR Gatehouse

gatehouse

Adding a deterministic linter/static-analysis gate to agent-generated PRs reduces human review time per merged PR and blocks bad patterns without reducing merge throughput compared with human-only review.

Proof of concept

Instrument a repo's agent PR pipeline with Oxlint/React Doctor plus custom rules; run an A/B split of agent PRs through the gate vs human-only review for 4-6 weeks.

Measurement

Human review minutes per PR, escaped defect rate, false-positive dismissal rate, and merged PR throughput.

Source video

Dumb Zone Mapper

beyond-evals

For a fixed coding task suite, agent output quality has a statistically detectable inflection point at a token length that varies by model and task but is consistently below the model's nominal context limit.

Proof of concept

Benchmark harness that varies prompt context length from 1k to 200k tokens on identical tasks across 2-3 model versions, scoring functional correctness and lint violations.

Measurement

Inflection token count per model/task, quality drop magnitude past inflection, and variance across task types.

Source video

Spec Compiler & Verifier

movement-lab

Spec-level tests generated from executable specifications catch at least as high a fraction of agent-introduced regressions as human code review, allowing the spec to become the primary reviewed artifact.

Proof of concept

Prototype a spec-to-code pipeline where versioned specs compile to code and generate spec tests; compare regression escape rate and review effort against a code-review baseline on matched features.

Measurement

Escaped regression rate, spec review time, code churn after merge, and fraction of implementation decisions encoded in spec.

Source video

Rework Probability Estimator

new

Historical agent PR features can predict rework probability and change cost per task class accurately enough that an Expected-Pain-triggered spec-effort policy reduces total cycle time versus fixed planning effort.

Proof of concept

Instrument past agent PRs to label rework events and change cost, train a simple model, simulate gating upfront spec depth by predicted expected pain.

Measurement

Calibration error of predicted rework, total cycle-time reduction, and change in escaped rework rate.

Source video

Architectural implications

3

The authoritative version-controlled artifact moves from generated code to human-authored specs and intent prompts.

Before

Developers check in code directly and diff code changes.

After

Developers check in specifications and intent prompts that compile into code.

Consequence

Requires executable-spec tooling, spec-level verification, and review workflows that operate above the code layer.

Source video

Agent output review is partly delegated to deterministic linters and static analysis rather than only human review.

Before

Human reviewers inspect agent pull requests directly.

After

Deterministic linters/static analyzers act as automated reviewers and guardrails on agent PRs.

Consequence

CI becomes the enforcement surface for agent behavior; review capacity no longer bounds throughput but must be designed for deterministic checks.

Source video

Context is assembled through intentional chunking and decision trees instead of dumping the codebase or a giant plan into the model.

Before

Entire codebases or 100k-token plan files are passed to the model as context.

After

Context is deliberately chunked and structured into decision trees within a bounded token budget.

Consequence

The context-assembly layer becomes an explicit architectural component with token budgets and decomposition logic.

Source video

Tradeoffs and failure modes

3

Un-reviewed autonomous planning vs. velocity

Benefit

Agents can plan and execute entire feature plans without human intervention, nominally increasing throughput.

Cost or risk

Severe regression and code rot that made the codebase un-navigable within months, forcing a full product rebuild.

AI-generated plan files containing full inline code implementations led to severe regression and code rot.
Open source video
Source video

Large-context prompting vs. output quality

Benefit

Passing the whole codebase or a full implementation plan gives the model maximal information.

Cost or risk

Exceeding ~100k tokens enters a 'dumb zone' where model performance and output quality degrade significantly.

Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.
Open source video
Source video

Agent autonomy vs. human control

Benefit

More upfront planning and guardrails reduce chance of rework.

Cost or risk

Over-micromanaging the agent defeats the purpose of automation, so exceeding the leverage sweet spot destroys the value.

Find the sweet spot between autonomy and control.
Open source video
Source video

Open questions

4

Is the ~100,000-token 'dumb zone' threshold stable across models and tasks, or is it an artifact of one model generation?

Why unresolved

The summary reports a token figure as a claim without methodology, model version, or task breakdown.

Research direction

Run controlled benchmarks sweeping prompt length against task-accuracy/quality across model versions to find where quality inflects.

Source video

Can specifications actually be made executable and verifiable enough to replace code as the checked-in artifact, and how would that verification work?

Why unresolved

The claim is presented as an analogy to compilation and a prediction, not as a demonstrated pipeline.

Research direction

Prototype a spec-to-code pipeline with spec-level tests and measure whether spec verification catches the regressions that code review would have caught.

Source video

Does the full-rewrite outcome generalize, or is it specific to how HumanLayer's plan files embedded full inline code implementations?

Why unresolved

The case study is a single team's experiment without controls or comparison conditions.

Research direction

Replicate the un-reviewed agentic planning workflow on a controlled codebase and measure maintainability/regression over time versus reviewed baselines.

Source video

How can the inputs of the Expected Pain Formula (probability of rework, pain of change) be estimated in advance rather than guessed?

Why unresolved

The formula is offered as a heuristic with no estimation method for its variables.

Research direction

Instrument past agent PRs to derive empirical rework-probability and change-cost distributions per task class.

Source video

Key claims

7
causalVerification needed

Un-reviewed AI-generated plan files containing full inline code implementations cause severe regression and code rot.

Evidence

AI-generated plan files containing full inline code implementations led to severe regression and code rot.

Question

Does removing human review from plan-file-driven agentic execution measurably increase regression rates versus reviewed execution?

Source video
causalVerification needed

Codebases become unusable after a few months of un-reviewed agentic execution.

Evidence

Codebases became unusable after a few months of un-reviewed agentic execution.

Question

What maintainability metrics degrade over time under un-reviewed agentic execution, and on what timeline?

Source video
factualVerification needed

Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.

Evidence

Prompts exceeding 100,000 tokens result in significantly degraded model performance and output quality.

Question

At what token length does output quality measurably degrade, and does the threshold vary by model and task?

Source video
predictionVerification needed

Specifications are verifiable, executable, and compile into code.

Evidence

Specifications are verifiable, executable, and compile into code.

Question

Can specifications be made executable and verifiable in practice, and what fraction of implementation decisions can they encode?

Source video
predictionVerification needed

Checking in code directly will be replaced by checking in specifications and intent prompts.

Evidence

Checking in code directly will be replaced by checking in specifications and intent prompts.

Question

Do teams adopting spec-first agentic workflows actually cease versioning code as the primary artifact?

Source video
causalVerification needed

Writing detailed specs up front saves hours of review time and prevents bad patterns from proliferating.

Evidence

Writing detailed specs up front saves hours of review time and prevents bad patterns from proliferating.

Question

Does upfront spec effort causally reduce total review time and pattern proliferation compared to minimal-prompt baselines?

Source video
opinionVerification not requested

Teams must define explicit coding standards and review policies for their AI agents to encode taste and intent.

Evidence

Teams must define explicit coding standards and review policies for their AI agents.

Source video

Connections

4