Beyond Coding · Published 2026-09-30

Why Coding Agents Keep Making Your Codebase Worse

privateReview: unreviewedConfidence: high
Coding agentscodebase maintainabilityslopcodebenchContext engineeringtechnical debtcoding-agentscontext-engineeringbenchmark-designtechnical-debtcode-reviewHuman-in-the-loopagent-maintainability
Open on YouTube ↗

Summary

Overview

  • Speaker: Dexter Horthy
  • Channel: Beyond Coding
  • Main topic: Context Engineering and the pitfalls of AI coding agents degrading software codebases over time.
  • Purpose: To analyze why current coding agents degrade codebases, explore benchmarks like slopcodebench, and discuss agentic engineering workflows. Dexter Horthy, CEO of HumanLayer, discusses why AI coding agents often degrade codebases over time. He explains that traditional benchmarks like swe-bench and token-maxing do not reflect real-world engineering where codebases suffer from cumulative complexity without upfront problem knowledge. He covers context engineering workflows, pull request review culture, and how human-in-the-loop validation remains essential for long-term codebase maintainability.

Topic Map

The Problem with Codebase Degradation and Benchmarks

  • Explanation: Discussion on how models make codebases worse over time and why existing benchmarks fail to capture real software engineering challenges.
  • Key claims:
    • Models make codebases worse over time because they lack full upfront context.
    • Traditional benchmarks like swe-bench only test small isolated PR fixes on historical repos.
    • Slopcodebench demonstrates that models make codebases worse over time as tasks compound.
  • Examples:
    • GPT-5.5 Codex scoring 15% on slopcodebench and Astra scoring 16.5%.
  • Terminology:
    • slopcodebench
    • swe-bench
    • token maxing
  • Why it matters: Highlights that passing isolated micro-benchmarks does not mean a model can maintain a real, complex codebase over time.

Context Engineering vs. Token Maxing

  • Explanation: Examining how feeding entire problems upfront fails compared to incremental, targeted context provision.
  • Key claims:
    • Giving models the entire problem upfront leads to poor architecture and technical debt.
    • Slopcodebench tests greenfield building followed by incremental feature addition with 3 to 10 checkpoints.
    • Incremental planning and narrowing failure modes are crucial for agent reliability.
  • Examples:
    • Rewriting all of Microsoft Excel or Redis from specs vs adding features incrementally.
  • Terminology:
    • context engineering
    • greenfield
    • checkpoints
  • Why it matters: Demonstrates that effective agent usage requires structuring problems into manageable, iterative steps rather than single monolithic prompts.

Pull Requests, Code Review Culture, and Human-in-the-Loop

  • Explanation: Debating the future of PR workflows, code reviews, and human oversight when working with AI agents.
  • Key claims:
    • PR culture and human review are essential because models write unmaintainable code without deep taste.
    • Some teams eliminate PR reviews and push straight to main, but this risks severe quality degradation.
    • Design docs and prompt refinement before coding help prevent agent errors.
  • Examples:
    • Using design docs and GitHub MCP to let agents research across repos before prompting.
  • Terminology:
    • design doc
    • pull request
    • human-in-the-loop
  • Why it matters: Emphasizes that human taste, architectural understanding, and rigorous review processes cannot be fully automated away without risking software quality.

Key Points

Agents degrade codebases over time

  • Explanation: Without knowing the entire codebase context upfront, agents introduce architectural flaws and technical debt when working unattended.
  • Evidence: Slopcodebench results showing high failure rates across models as codebases evolve.
  • Practical implication: Teams must not mortgage their codebase hoping future models will automatically fix technical debt.

Incremental planning beats monolithic prompts

  • Explanation: Breaking down tasks into smaller checkpoints with targeted context yields much better code than asking an agent to one-shot a massive feature.
  • Evidence: Success with incremental PR workflows and design docs.
  • Practical implication: Invest time in writing good design docs and breaking work into small, verifiable steps.

Human review and taste remain indispensable

  • Explanation: Deterministic linters and AI reviews catch syntax errors, but human architectural taste and judgment are still required to maintain long-term codebase health.
  • Evidence: Engineers with years of experience developing taste through encountering bad architecture.
  • Practical implication: Do not skip code reviews just because code was generated by an AI agent.

Frameworks, Models & Processes

Context Engineering Workflow

  • How it works: Iteratively providing targeted context, design docs, and repository references to guide agents through complex tasks step-by-step.
  • Components:
    • Design doc creation
    • Repository research via MCP
    • Incremental PR checkpoints
    • Human review and validation
  • When to use: When building or maintaining complex software systems with AI coding agents.

Examples & Case Studies

Running GPT-5.5 Codex and Astra on slopcodebench

  • Illustrates: Models struggling to maintain codebase quality over multiple compounding tasks
  • Lesson: Current models lack long-term codebase maintenance capabilities without structured human guidance.

Actionable Takeaways

  • Immediate:
    • Avoid one-shotting massive coding tasks with AI agents.
    • Break down features into smaller, incremental PRs.
    • Use design docs and research tools before prompting agents.
  • Strategic:
    • Do not mortgage your codebase for short-term velocity.
    • Maintain rigorous code review and architectural standards.
    • Focus on human-in-the-loop workflows for complex enterprise codebases.
  • Questions to investigate:
    • How can deterministic tooling better enforce architectural maintainability in AI-generated code?
    • What is the optimal balance between autonomous agent execution and human code review?
    • How will subscription models and token pricing evolve in enterprise software development?

Claims Worth Verifying

  • GPT-5.5 Codex scored 15% on slopcodebench when tested in May (benchmark statistic)
  • Astra scored 16.5% on slopcodebench (benchmark statistic)

Notable Quotes

"Do not mortgage your codebase in the hopes that in a year the model will come out that's smart enough to fix all your problems." (at 0:09) "Actually at one point higher, the ability to maintain like complex codebases over time." (at 7:16) "The skill issue is not like couldn't use enough tokens, because that doesn't apply to most people who are doing serious work." (at 68:44)

Compressed Summary

  • Coding agents degrade codebase quality over time due to lack of upfront context.
  • Traditional benchmarks like swe-bench do not test cumulative codebase maintainability.
  • Slopcodebench proves models struggle with compounding technical debt.
  • Incremental planning, design docs, and human review are essential for agent success.
  • Keywords: coding agents, codebase maintainability, slopcodebench, context engineering, technical debt
  • Core insight: AI coding agents degrade codebases over time because they lack full upfront context, making incremental planning and human architectural oversight essential.

Core insights

6
Empirical Resulthigh noveltymoderate evidence

Benchmark design determines what you can measure: slopcodebench evaluates a greenfield build followed by incremental feature addition across 3 to 10 checkpoints, so quality degradation compounds across tasks — unlike swe-bench, which tests isolated PR fixes on historical repos and therefore cannot reveal long-horizon maintainability failure.

Why it matters

Evaluation harnesses built on static, single-shot repo tasks will report capability that does not transfer to evolving production codebases; teams selecting models on swe-bench-style scores are optimizing the wrong signal.

Generalization

For any agentic system whose output is consumed and extended repeatedly, evaluation must simulate state accumulation across steps, not just per-task success — otherwise the benchmark is blind to compounding debt.

Slopcodebench tests greenfield building followed by incremental feature addition with 3 to 10 checkpoints.
Open source video
Traditional benchmarks like swe-bench only test small isolated PR fixes on historical repos.
Open source video
Slopcodebench demonstrates that models make codebases worse over time as tasks compound.
Open source video
Mechanismmedium noveltymoderate evidence

Context delivery structure, not token volume, is the dominant variable: supplying the entire problem upfront produces poor architecture and technical debt, while incremental context provision with explicit checkpoints narrows failure modes per step.

Why it matters

Reframes the cost/quality dial away from 'more context window' toward sequencing and scoping of context, which is a harness/workflow design decision rather than a model capability decision.

Generalization

Agent runtimes should expose checkpoint boundaries as a first-class primitive, so context can be scoped to a verifiable sub-goal rather than the full problem statement.

Giving models the entire problem upfront leads to poor architecture and technical debt.
Open source video
The skill issue is not like couldn't use enough tokens, because that doesn't apply to most people who are doing serious work.
Open source video
Failure Modemedium noveltymoderate evidence

The 'mortgage the codebase' failure mode: teams accept agent-introduced architectural debt on the assumption that a future, smarter model will clean it up — but debt compounds while the cleanup assumption remains unverified.

Why it matters

It is a deferred-cost trap whose interest rate is measured in codebase maintainability, and it is not recoverable by model improvement alone if the architecture has already degraded.

Generalization

Any workflow that defers a correction to a future capability should treat the deferral as a debt with compounding cost, not as a free option.

Do not mortgage your codebase in the hopes that in a year the model will come out that's smart enough to fix all your problems.
Open source video
Teams must not mortgage their codebase hoping future models will automatically fix technical debt.
Open source video
Practicemedium noveltymoderate evidence

Tooling coverage is split by failure class: deterministic linters and AI reviews catch syntax-level errors, but architectural judgment ('taste') is accumulated by engineers through years of encountering bad architecture and is not substituted by those automated layers.

Why it matters

Defines where an automated gate is sufficient and where a human architectural reviewer remains load-bearing, which is precisely the boundary that determines whether review can be removed.

Generalization

Design agent pipelines so each automated check has a declared failure class it covers; anything outside that class needs a human gate, or the pipeline has an unowned failure mode.

Deterministic linters and AI reviews catch syntax errors, but human architectural taste and judgment are still required to maintain long-term codebase health.
Open source video
PR culture and human review are essential because models write unmaintainable code without deep taste.
Open source video
Practicemedium noveltyweak evidence

Pre-coding artifacts function as an error-prevention mechanism: writing design docs and letting agents research across repositories (via GitHub MCP) before prompting changes agent output quality, moving correction upstream of generation.

Why it matters

Shifts error handling from post-hoc review to pre-generation grounding, which is cheaper per defect and generalizable to any long-running agent task.

Generalization

Invest in machine-readable pre-task artifacts and cross-repository retrieval as the harness layer that constrains an agent before it writes code.

Using design docs and GitHub MCP to let agents research across repos before prompting.
Open source video
Design docs and prompt refinement before coding help prevent agent errors.
Open source video
Mental Modelmedium noveltyweak evidence

Task shape, not task size, predicts agent success: rewriting an entire system from a spec (e.g., Microsoft Excel or Redis) versus adding features incrementally are qualitatively different problems, because the incremental path allows knowledge to be discovered and verified in pieces.

Why it matters

Gives a criterion for decomposing work for agents: prefer tasks where required knowledge is discoverable incrementally over tasks that require a globally correct upfront design.

Generalization

Classify workload by whether correctness depends on knowledge obtainable at each step or on a full-state prior; route the former to agents, gate the latter behind human design.

Rewriting all of Microsoft Excel or Redis from specs vs adding features incrementally.
Open source video
Incremental planning and narrowing failure modes are crucial for agent reliability.
Open source video

Deep dives

5

Compounding-state benchmarks as a proxy for real codebase maintainability

Research question

Does a multi-checkpoint compounding benchmark score predict measured maintainability drift when a model is replayed against a real repository's historical change sequence?

Why

Model selection currently keys off static single-shot repo benchmarks, so if checkpoint-compounding scores do not correlate with production maintenance quality, teams are selecting on a signal that does not transfer and the whole procurement loop is miscalibrated.

Slopcodebench tests greenfield building followed by incremental feature addition with 3 to 10 checkpoints.
Open source video
Slopcodebench demonstrates that models make codebases worse over time as tasks compound.
Open source video
Traditional benchmarks like swe-bench only test small isolated PR fixes on historical repos.
Open source video
Source video

Checkpoint scoping as a first-class runtime primitive

Research question

If context is delivered per verifiable sub-goal with explicit checkpoint boundaries rather than as one full problem statement, how much does architectural quality and technical debt diverge under identical token budgets?

Why

It converts context quality from a prompt-authoring habit into an enforceable runtime construct, which is the difference between advice engineers cannot operationalize and a harness feature that constrains the agent before it writes code.

Giving models the entire problem upfront leads to poor architecture and technical debt.
Open source video
The skill issue is not like couldn't use enough tokens, because that doesn't apply to most people who are doing serious work.
Open source video
Incremental planning and narrowing failure modes are crucial for agent reliability.
Open source video
Source video

Per-failure-class ownership in automated review pipelines

Research question

Which architectural invariants (dependency direction, module boundary crossing, complexity budgets) can be encoded as deterministic CI gates, and what fraction of agent-introduced architectural debt do they catch that linters and AI review miss?

Why

Removing human review is only safe where an automated gate demonstrably owns that failure class; without a declared coverage map, the pipeline has an unowned failure mode and the review boundary cannot be retired on evidence.

Deterministic linters and AI reviews catch syntax errors, but human architectural taste and judgment are still required to maintain long-term codebase health.
Open source video
PR culture and human review are essential because models write unmaintainable code without deep taste.
Open source video
Some teams eliminate PR reviews and push straight to main, but this risks severe quality degradation.
Open source video
Source video

Pre-generation grounding artifacts as defect-prevention tooling

Research question

Do machine-readable design docs plus cross-repository retrieval measurably reduce the rate of architectural corrections requested in review, compared to prompting with the raw task description alone?

Why

It moves correction upstream of generation, where per-defect cost is lowest, and it is the only pass-1 practice that changes what the agent sees before any code exists rather than catching it after.

Using design docs and GitHub MCP to let agents research across repos before prompting.
Open source video
Design docs and prompt refinement before coding help prevent agent errors.
Open source video
Source video

Workload classification by knowledge discoverability

Research question

Can tasks be reliably classified by whether correctness depends on knowledge obtainable incrementally versus a globally correct upfront design, and does that classification predict agent success better than task size?

Why

It gives an operational routing rule for deciding what to hand to agents and what must pass through human design first, replacing intuition about task difficulty with a property of the task's knowledge structure.

Rewriting all of Microsoft Excel or Redis from specs vs adding features incrementally.
Open source video
Incremental planning and narrowing failure modes are crucial for agent reliability.
Open source video
Source video

Article ideas

4

Your Agent Benchmark Is Blind to the Only Failure That Matters

Static single-shot benchmarks measure whether a model can close an isolated ticket, but the economically relevant question is whether a codebase is still workable after fifty agent-authored changes, and those two measurements are not the same signal in either direction.

Angle

Contrast the task structure of isolated-PR benchmarks against compounding greenfield-plus-increment sequences, and argue that any team selecting models on the former is optimizing a proxy that omits the state accumulation the latter makes visible.

Source video

Context Engineering Is a Sequencing Problem, Not a Capacity Problem

The binding constraint on agent code quality is the order and scope in which context is delivered, not how much of it fits, so buying larger context windows does not address the failure mode while checkpointing the problem does.

Angle

Take the full-problem-upfront failure and argue for checkpoint boundaries as a runtime feature, showing why the token-maxing strategy is a uniform answer to a problem that is inherently about decomposition.

Source video

Mortgaging the Codebase: The Debt Bet Every Agent-Heavy Team Is Already Holding

Deferring architectural cleanup to a presumedly smarter future model is a leveraged debt position with compounding interest and unverified repayment terms, and architecture that has already degraded may not be recoverable by capability improvement at all.

Angle

Reframe the 'the next model will fix it' assumption as an unpriced bet, then specify what evidence would have to exist before the deferral is rational rather than a velocity subsidy.

Source video

Declare Your Failure Classes Before You Delete the Review Gate

Human review can only be safely removed from the parts of the pipeline where an automated check demonstrably owns that class of failure, and architectural taste currently has no automated owner, which makes removing review a transfer of unowned risk rather than an efficiency gain.

Angle

Build a coverage map of automated gates against failure classes and locate the architectural-judgment gap that remains after linters and AI review, then draw the review boundary where the map runs out.

Source video

Project ideas

4

Checkpoint Harness

movement-lab

For an identical total token budget, delivering context per checkpointed sub-goal with an explicit verification boundary before advancing produces measurably lower architectural debt at the end of a multi-step sequence than delivering the full problem statement upfront.

Proof of concept

Run the same greenfield-plus-increment task sequence through two harness configurations — monolithic prompt versus checkpoint-scoped context with a verification gate between steps — on the same model and identical token budget.

Measurement

Compare resulting codebase state at each checkpoint using dependency structure, module boundary violations, duplication, and complexity per module, plus count of later changes required to correct earlier architectural decisions.

Source video

Architecture Invariant Gate

gatehouse

A deterministic CI gate encoding dependency-direction rules, boundary-crossing rules and complexity budgets catches a measurable share of architectural degradation that neither linters nor AI review flag on agent-authored changes.

Proof of concept

Encode a small invariant set for a target repository, run it over a backlog of historical agent-authored pull requests, and compare its findings against what linters and AI review reported on the same changes.

Measurement

Overlap and unique-detection rate of the invariant gate versus linter and AI review output per pull request, plus the rate at which flagged changes were later corrected by humans.

Source video

Replay Drift

beyond-evals

A model's score on a compounding multi-checkpoint benchmark correlates with measured maintainability drift when that model is replayed against a real repository's historical change sequence.

Proof of concept

Take a production repository's ordered historical change sequence, replay a candidate model over it under a fixed harness, and compute both benchmark score and maintainability drift per candidate.

Measurement

Rank correlation between benchmark score and post-replay maintainability drift across candidate models, plus the spread in drift at equal benchmark scores.

Source video

Pre-Task Grounding Kit

new

Supplying a machine-readable design doc plus cross-repository retrieval before prompting reduces the rate of architectural corrections requested during review of agent-authored changes compared to prompting with the task description alone.

Proof of concept

On a fixed set of tasks in one repository, A/B a grounded pre-task artifact against the bare prompt, holding model and harness constant, and classify every human review correction by type.

Measurement

Architectural corrections per pull request, total review cycles to merge, and rework rate after merge within the next N dependent changes.

Source video

Architectural implications

3

Review and PR structure acts as the validation layer that currently contains agent-introduced architectural debt.

Before

PR review treated as a workflow bottleneck to be optimized away for velocity with AI-generated code.

After

PR review treated as the enforcement point for architectural standards that automated syntax-level gates cannot cover.

Consequence

Removing the review boundary offloads architectural risk to the codebase with no automated owner of that failure class.

Source video

Evaluation of coding agents is moving from static repo snapshots to stateful, multi-checkpoint task sequences.

Before

Benchmarks score isolated pull-request fixes against historical repositories.

After

Benchmarks score cumulative codebase quality across a greenfield build plus successive feature additions.

Consequence

Model selection and harness design shift toward measuring long-horizon maintainability rather than single-task pass rates.

Source video

Context provisioning becomes a harness responsibility rather than a prompt-authoring concern.

Before

Prompt the model with the complete problem statement and maximize available tokens.

After

Deliver targeted context per checkpoint, with design docs and cross-repo research supplied by tooling such as MCP.

Consequence

Agent runtimes need explicit constructs for checkpoint scoping and pre-task retrieval, not just larger context windows.

Source video

Tradeoffs and failure modes

4

One-shot problem delivery vs incremental checkpoints

Benefit

One-shot prompts minimize coordination overhead and appear faster for large features.

Cost or risk

Upfront full-problem delivery yields poor architecture and accumulating technical debt that must be repaid later.

Giving models the entire problem upfront leads to poor architecture and technical debt.
Open source video
Source video

Removing pull-request review

Benefit

Pushing straight to main removes a review bottleneck and increases apparent delivery velocity.

Cost or risk

Severe quality degradation, because agents write unmaintainable code without architectural taste and no human gate catches it.

Some teams eliminate PR reviews and push straight to main, but this risks severe quality degradation.
Open source video
Source video

Relying on future model capability to repair debt

Benefit

Accepts short-term velocity by deferring architectural cleanup to a presumed smarter future model.

Cost or risk

The deferral is an unverified bet; debt compounds in the meantime and may not be recoverable by model improvement.

Do not mortgage your codebase in the hopes that in a year the model will come out that's smart enough to fix all your problems.
Open source video
Source video

Token maxing vs context engineering

Benefit

Maximizing tokens is a simple, uniform strategy that does not require decomposing the problem.

Cost or risk

Token volume is not the binding constraint for serious engineering work, so the strategy fails to address the actual failure mode.

The skill issue is not like couldn't use enough tokens, because that doesn't apply to most people who are doing serious work.
Open source video
Source video

Open questions

3

How can deterministic tooling enforce architectural maintainability in AI-generated code, beyond the syntax-level errors linters already catch?

Why unresolved

The summary asserts deterministic linters and AI review only cover syntax-level faults, while architectural judgment is tied to human taste, leaving no described mechanism for automating the architectural gate.

Research direction

Prototype structural/architectural invariants (dependency rules, boundary checks, complexity budgets) as CI gates and measure whether they catch degradation that linters and AI review miss.

Source video

What is the optimal balance between autonomous agent execution and human code review?

Why unresolved

The summary states review is essential and that removing it risks degradation, but offers no threshold, autonomy level, or task-class specificity.

Research direction

Run controlled comparisons across autonomy levels and task classes, measuring codebase quality metrics over successive change sequences.

Source video

Does performance on a compounding benchmark like slopcodebench predict real-world codebase degradation in production repositories?

Why unresolved

The summary reports scores on a synthetic greenfield-plus-increment task sequence but offers no link to observed outcomes on real codebases.

Research direction

Replay historical production change sequences through candidate models and correlate benchmark score with measured maintainability drift.

Source video

Key claims

7
causalVerification needed

Models make codebases worse over time because they lack full upfront context.

Evidence

Models make codebases worse over time because they lack full upfront context.

Question

Is context incompleteness the dominant cause of codebase degradation, or do other factors (task scoping, review absence, model capability) contribute comparably?

Source video
factualVerification needed

GPT-5.5 Codex scored 15% on slopcodebench.

Evidence

GPT-5.5 Codex scoring 15% on slopcodebench and Astra scoring 16.5%.

Question

What is the actual reported score, benchmark configuration, and evaluation date for GPT-5.5 Codex on slopcodebench?

Source video
factualVerification needed

Astra scored 16.5% on slopcodebench.

Evidence

GPT-5.5 Codex scoring 15% on slopcodebench and Astra scoring 16.5%.

Question

What is the actual reported score and benchmark configuration for Astra on slopcodebench?

Source video
factualVerification needed

Slopcodebench tests greenfield building followed by incremental feature addition with 3 to 10 checkpoints.

Evidence

Slopcodebench tests greenfield building followed by incremental feature addition with 3 to 10 checkpoints.

Question

What is the exact task structure, checkpoint count, and scoring rubric of slopcodebench?

Source video
comparativeVerification needed

Traditional benchmarks like swe-bench only test small isolated PR fixes on historical repos.

Evidence

Traditional benchmarks like swe-bench only test small isolated PR fixes on historical repos.

Question

Do swe-bench-style benchmarks fully omit stateful, compounding-task evaluation, or do variants already exist?

Source video
causalVerification needed

Giving models the entire problem upfront leads to poor architecture and technical debt.

Evidence

Giving models the entire problem upfront leads to poor architecture and technical debt.

Question

Under a controlled comparison, does full-problem upfront context measurably degrade architectural quality versus incremental checkpointed delivery?

Source video
causalVerification needed

Removing PR reviews and pushing straight to main risks severe quality degradation.

Evidence

Some teams eliminate PR reviews and push straight to main, but this risks severe quality degradation.

Question

How much measurable quality degradation follows removal of PR review in AI-agent-heavy codebases?

Source video

Connections

4