AI Engineer · Published 2026-09-26

Fixing the PR Bottleneck — Matt Pocock, AIHero

privateReview: unreviewedConfidence: high
ai engineerpull requestautomationCode reviewtestingagentic-workflowscode-reviewpull-requestsdeep-modulestest-qualitycontext-separationci-cd
Open on YouTube ↗

Summary

Overview

  • Speaker: Matt Pocock
  • Channel: AI Engineer
  • Main topic: Fixing the PR bottleneck in software engineering using automated checks, automated review, and human review with AI agents.
  • Purpose: To educate AI engineers on how to manage the explosion of PRs caused by AI coding agents by implementing structured automated and human review processes. Matt Pocock discusses how AI agents have vastly increased the number of pull requests (PRs) being generated, exacerbating the traditional PR bottleneck. He presents a three-layered framework—Automated Checks, Automated Review, and Human Review—designed to speed up software delivery while maintaining high code quality. By leveraging cheap CPU-based automated checks as lie detectors, refining AI agent prompts via targeted coding standards (CODING_STANDARDS.md) rather than global prompts, and optimizing human review around two-way doors and clear 'why' context (via tools like /pr, /show-me, and /retro), teams can stop the slop and make code review matter.

Topic Map

The PR Bottleneck in the Age of AI

  • Explanation: Organizations have always struggled with unreviewed PRs, a problem that has scaled massively due to AI coding agents imposing new constraints and generating more code.
  • Key claims:
    • AI agents promise to scale developer output, leading to more PRs and work.
    • Without proper gating, this creates a 'software factory' slipstream of low-quality PRs.
  • Examples:
    • Massive backlogs of unreviewed PRs laying around.
  • Terminology:
    • Software factory
    • PR bottleneck
    • Accelerators
    • Brakes
  • Why it matters: Unchecked AI code generation leads to software entropy and overwhelming review burdens.

The Three-Layered Cake of PR Review

  • Explanation: A structured defense against bad code comprising Automated Checks, Automated Review, and Human Review.
  • Key claims:
    • Automated checks are cheap CPU cycles used as lie detectors for code quality.
    • Automated review uses AI agents to check code against coding standards before human review.
    • Human review is the final, most costly step and should be made as fast as possible.
  • Examples:
    • Linting and type checking at layer one; AI diff review at layer two; human sign-off at layer three.
  • Terminology:
    • Automated checks
    • Automated review
    • Human review
  • Why it matters: Layering defense mechanisms prevents bad code from ever reaching human eyes.

Automated Checks and Lie Detectors

  • Explanation: Automated checks (linting, tests, type checking) are cheap, but they can lie when tests are tautological or structure-sensitive.
  • Key claims:
    • Checks cost CPU cycles, not tokens or human effort.
    • Green CI does not mean code is ready to merge because checks can lie.
    • Tautological tests and structure-sensitive tests are common failure modes of AI-generated tests.
  • Examples:
    • Tests that re-assert the implementation (e.g., expecting 280 because the constant is 280).
  • Terminology:
    • Tautological tests
    • Structure-sensitive tests
    • Automated checks
  • Why it matters: Understanding how automated checks fail prevents false confidence in CI pipelines.

Codebase Design and Deep Modules

  • Explanation: Designing codebases with deep modules (complex behavior hidden behind simple interfaces) reduces structure-sensitive tests.
  • Key claims:
    • Deep modules improve codebase design and reduce brittle tests.
    • Using /improve-codebase-architecture restructures codebases into deep modules.
  • Examples:
    • Consolidating hand-rolled validation expressions into a single predicate module.
  • Terminology:
    • Deep modules
    • Locality
    • Leverage
    • Seam
  • Why it matters: Good codebase design makes it harder for AI agents to write bad tests and code.

Automated Review Standards

  • Explanation: How to structure automated review agents so they enforce coding standards effectively without overloading the implementation agent.
  • Key claims:
    • Do not put coding standards in your implementer agent; keep implementation and review separate.
    • Automated review should output commits, not comments.
    • Use /code-review to read CODING_STANDARDS.md and check diffs.
  • Examples:
    • Separating /implement (Make It Work) from /code-review (Make It Good).
  • Terminology:
    • Implementer agent
    • Reviewer agent
    • CODING_STANDARDS.md
    • Commits not comments
  • Why it matters: Separation of concerns prevents AI agents from getting overwhelmed and producing low-quality feedback.

Human Review and PR Optimization

  • Explanation: Optimizing human review by focusing on one-way vs. two-way doors, blast radius, pseudocode, and visual summaries.
  • Key claims:
    • Some reviews are more important than others; distinguish one-way from two-way doors.
    • Use /pr to generate human-friendly PR bodies optimized for review.
    • Use /show-me to provide visual diagrams of changes rather than walls of text.
  • Examples:
    • Mermaid sequence diagrams generated automatically to explain component interactions.
  • Terminology:
    • One-way door
    • Two-way door
    • Blast radius
    • Pseudocode
    • /show-me
    • /retro
  • Why it matters: Making understanding the 'why' as fast as possible reduces the human review bottleneck.

Key Points

Automated checks are cheap but can lie

  • Explanation: While CPU cycles are inexpensive, automated tests generated by AI can be tautological or overly sensitive to internal code structure, creating false confidence.
  • Evidence: AI agents writing tests that simply assert constant values match the implementation.
  • Practical implication: Do not rely solely on green CI; ensure tests actually verify behavior rather than structure.

Separate implementation from review

  • Explanation: Overloading a single AI agent prompt with exploration, coding, debugging, and strict style guidelines degrades its output.
  • Evidence: The /implement command handles making it work (Red/Green), while /code-review handles making it good (Refactor).
  • Practical implication: Use distinct agents with specialized context windows for implementing versus reviewing code.

Automated review should produce commits, not comments

  • Explanation: Having AI reviewers leave lengthy comments burdens human reviewers with parsing feedback and applying fixes.
  • Evidence: Reviewer agents that directly commit fixes turn PRs into polished artifacts.
  • Practical implication: Configure review agents to make direct fixes so humans review finished work.

Optimize human review for 'why', not just 'how'

  • Explanation: Human reviewers need to understand the architectural intent and blast radius quickly without reading verbose text.
  • Evidence: Using /show-me for visual sequence diagrams and /pr for structured PR descriptions.
  • Practical implication: Adopt tools that summarize changes visually and highlight whether a change is a one-way or two-way door.

Frameworks, Models & Processes

The Three-Layered PR Defense

  • How it works: Layers automated checks, automated review, and human review sequentially to catch errors and enforce quality.
  • Components:
    • Automated Checks (Linting, type checking, tests)
    • Automated Review (AI agent checking diffs against standards)
    • Human Review (Targeted human sign-off on high-risk changes)
  • When to use: Whenever deploying AI coding agents that generate high volumes of pull requests.

The Refactor / Red-Green Workflow for AI

  • How it works: Splits the AI coding workflow into distinct phases executed by separate agent commands.
  • Components:
    • /implement (Make It Work / Red-Green)
    • /code-review (Make It Good / Refactor / Coding Standards)
    • /retro (Suggests automated checks and standards updates)
  • When to use: When setting up custom CLI tools or workflows for AI engineer teams.

Examples & Case Studies

An AI agent generated a test for a character limit constant by asserting that the constant equals its own value.

  • Illustrates: Tautological tests (tests that only re-assert implementation).
  • Lesson: Automated checks can lie when tests are poorly structured.

Refactoring three call sites with duplicated visibility expressions into a single predicate module.

  • Illustrates: Deep modules and codebase design (locality and leverage).
  • Lesson: Good architecture makes code easier for both AI agents and humans to maintain.

Actionable Takeaways

  • Immediate:
    • Stop putting coding standards in global prompt files like AGENTS.md; put them in CODING_STANDARDS.md.
    • Ensure automated review agents commit fixes rather than leaving verbose comments.
    • Use /show-me to generate visual diagrams for PR reviews.
  • Strategic:
    • Treat the AI coding process (exploration, implementation, debugging) as equally important as the code itself.
    • Design codebases with deep modules to minimize structure-sensitive tests.
    • Adopt a compounding review loop using /retro to update coding standards based on human review insights.
  • Questions to investigate:
    • How many of our automated tests are tautological or structure-sensitive?
    • Can we decouple our implementation agent from our review agent?
    • How can we integrate visual sequence diagrams into our standard PR template?

Claims Worth Verifying

  • Automated checks only cost CPU cycles, not tokens or human effort. (conceptual)
  • Separating implementation and review into distinct agent contexts improves output quality. (architectural)

Notable Quotes

"Fixing The PR Bottleneck, Automated Checks, Automated Review, Human Review" (at 0:06) "More Speed equals More PRs" (at 5:47) "Stop The Slop, Higher code quality = faster human review" (at 5:55) "Checks Are Cheap, CPU cycles, not tokens or human effort" (at 6:56) "Checks Can Lie, Does a green CI mean that the code is ready for merge? No." (at 7:34) "Commits, Not Comments, Automated review should NOT burden humans with comments to review and fix." (at 25:20)

Compressed Summary

  • AI agents increase PR volume, worsening the PR bottleneck.
  • Use a three-layer defense: Automated Checks, Automated Review, and Human Review.
  • Separate implementation (/implement) from review (/code-review) to avoid overloading AI agents.
  • Automated review should make direct commits, not leave comments.
  • Optimize human review by focusing on blast radius, pseudocode, and visual diagrams (/show-me).
  • Keywords: ai engineer, pull request, automation, code review, testing
  • Core insight: To fix the AI-induced PR bottleneck, teams must layer cheap automated checks, specialized automated review agents that commit fixes, and human-friendly PR workflows.

Core insights

6
Architecturemedium noveltymoderate evidence

Splitting implementation and review into distinct agents with separate context windows improves output quality; cramming exploration, coding, debugging, and style enforcement into one prompt degrades results. The /implement agent handles 'Make It Work' (Red/Green) while /code-review handles 'Make It Good' (Refactor).

Why it matters

Challenges the single-prompt 'do everything' agent pattern and suggests specialized agents with narrow context produce higher-quality work—directly informs multi-agent pipeline design.

Generalization

Separating generation from critique across distinct context windows is a general pattern for reducing instruction interference and improving reliability in agentic systems.

Do not put coding standards in your implementer agent; keep implementation and review separate.
Open source video
Overloading a single AI agent prompt with exploration, coding, debugging, and strict style guidelines degrades its output.
Open source video
Separating /implement (Make It Work) from /code-review (Make It Good)
Open source video
Mechanismhigh noveltymoderate evidence

An automated review agent should emit commits (direct fixes) rather than comments, converting PRs into polished artifacts before human review and removing the burden of parsing and applying feedback from humans.

Why it matters

Changes the interface of review agents from advisory (comments) to acting (commits), which affects tool permissions, auditability, and where automation stops and human judgment begins.

Generalization

Designing review/critic agents as actors that resolve issues rather than reporters shifts downstream human effort from remediation to verification.

Automated review should output commits, not comments.
Open source video
Reviewer agents that directly commit fixes turn PRs into polished artifacts.
Open source video
Configure review agents to make direct fixes so humans review finished work.
Open source video
Failure Modehigh noveltymoderate evidence

Automated checks (linting, tests, type checking) are cheap CPU-cycle-based 'lie detectors' but can produce false confidence: AI-generated tests are frequently tautological (asserting a constant equals itself) or structure-sensitive (bound to internal implementation).

Why it matters

Green CI is not a reliable readiness signal; teams must design against specific AI test-generation failure modes rather than trusting test pass rates.

Generalization

Cheap verification layers can be systematically gamed by the generator producing them, so verification mechanisms must be evaluated for their own fidelity, not just their existence.

Green CI does not mean code is ready to merge because checks can lie.
Open source video
Tautological tests and structure-sensitive tests are common failure modes of AI-generated tests.
Open source video
Checks cost CPU cycles, not tokens or human effort.
Open source video
Mental Modelhigh noveltymoderate evidence

Codebase architecture is a lever on agent output quality: designing deep modules (complex behavior behind simple interfaces) reduces structure-sensitive tests, making it harder for AI agents to write brittle tests and bad code.

Why it matters

Moves codebase design from a style concern to a control mechanism over agent behavior—architectural boundaries act as guardrails on generated tests and code.

Generalization

Abstraction quality (locality, leverage, seam) directly constrains the failure modes available to code-generating agents, so refactoring becomes an agent-reliability intervention.

Deep modules improve codebase design and reduce brittle tests.
Open source video
Good codebase design makes it harder for AI agents to write bad tests and code.
Open source video
Consolidating hand-rolled validation expressions into a single predicate module.
Open source video
Practicemedium noveltymoderate evidence

Coding standards should live in a targetable artifact (CODING_STANDARDS.md) read by a dedicated review agent rather than as global instructions injected into the implementer, enabling review to be diff-scoped and standards to be versioned independently.

Why it matters

Decouples style enforcement from generation, allowing standards to evolve and be applied selectively to diffs without contaminating the implementation context.

Generalization

Externalizing policy into a reviewable, referenceable artifact (rather than embedding it in the generator prompt) supports independent iteration and clearer separation of concerns.

Use /code-review to read CODING_STANDARDS.md and check diffs.
Open source video
Do not put coding standards in your implementer agent; keep implementation and review separate.
Open source video
Practicemedium noveltymoderate evidence

Human review effort should be triaged by reversibility (one-way vs two-way doors) and blast radius, and reviewers should be given the 'why' fast via structured PR bodies (/pr) and visual sequence diagrams (/show-me, Mermaid) instead of verbose text.

Why it matters

Treats human attention as the most expensive resource in the pipeline and optimizes the review interface around architectural intent rather than diff reading.

Generalization

When generation cost collapses, review becomes the bottleneck; optimizing the human-facing representation of change (not the change itself) is the highest-leverage intervention.

Some reviews are more important than others; distinguish one-way from two-way doors.
Open source video
Use /show-me to provide visual diagrams of changes rather than walls of text.
Open source video
Making understanding the 'why' as fast as possible reduces the human review bottleneck.
Open source video

Deep dives

4

Automated detection of tautological and structure-sensitive tests in AI-generated code

Research question

How reliably can tautological or structure-sensitive tests be detected automatically, and what precision/recall can be achieved by static analysis of test code?

Why

Green CI is a false signal if tests are tautological or structure-sensitive, undermining the first layer of defense. Without reliable detection, teams cannot trust automated checks, and bad code reaches human review.

Green CI does not mean code is ready to merge because checks can lie.
Open source video
Tautological tests and structure-sensitive tests are common failure modes of AI-generated tests.
Open source video
Tests that re-assert the implementation (e.g., expecting 280 because the constant is 280).
Open source video
Source video

Defect-catching efficacy of the three-layer review defense

Research question

What proportion of defects are caught at each layer (automated checks, automated review, human review) in practice, and how does this vary by defect type?

Why

Understanding layer efficacy enables targeted investment. If automated checks catch few real defects due to lying tests, automated review must compensate; if human review catches most, optimizing its speed is critical.

Layering defense mechanisms prevents bad code from ever reaching human eyes.
Open source video
Linting and type checking at layer one; AI diff review at layer two; human sign-off at layer three.
Open source video
Human review is the final, most costly step and should be made as fast as possible.
Open source video
Source video

Impact of separating coding standards into a dedicated review agent versus global prompt instructions

Research question

Does moving coding standards out of the implementer prompt and into a dedicated review agent (e.g., CODING_STANDARDS.md) measurably improve code quality and reduce review latency compared to global instructions?

Why

Challenges the common practice of embedding style rules in implementation prompts. If separation improves outcomes, it informs agent architecture and prompt design for AI coding pipelines.

refining AI agent prompts via targeted coding standards (CODING_STANDARDS.md) rather than global prompts
Open source video
Automated review uses AI agents to check code against coding standards before human review.
Open source video
Source video

Effect of human review optimization (visual summaries, reversibility triage) on review cycle time and defect escape

Research question

Do visual summaries (e.g., Mermaid diagrams) and one-way vs two-way door triage measurably shorten human review cycle time without increasing defect escape rates?

Why

Human review is the most costly step. Optimizing the interface of change representation could be the highest-leverage intervention when generation cost collapses.

optimizing human review around two-way doors and clear 'why' context
Open source video
Human review is the final, most costly step and should be made as fast as possible.
Open source video
Source video

Article ideas

4

Green CI is a Lie: How AI-Generated Tests Game Your Pipeline

Automated checks are not a reliable gate because AI-generated tests are often tautological or structure-sensitive, producing false confidence. Teams must implement detection of these failure modes to restore trust in CI.

Angle

Expose the false confidence in green CI, provide concrete examples of tautological and structure-sensitive tests, and argue for test-quality gates before human review.

Source video

Don't Put Coding Standards in Your Implementer: The Case for Separate Review Agents

Separating implementation and review into distinct agents with dedicated standards artifacts improves output quality and reduces instruction interference compared to monolithic prompts.

Angle

Argue against the common practice of cramming everything into one prompt, provide architectural rationale and evidence from agent-based code review experiments.

Source video

Review Agents Should Commit, Not Comment: Rethinking the Interface of Automated Review

Automated review agents that commit fixes directly shift human review from remediation to verification, reducing review time and making PRs polished artifacts, but require careful permissioning and auditability.

Angle

Argue for a new interface for review agents, discuss trade-offs, and outline implementation patterns for safe autonomous commits.

Source video

Architecture as Guardrail: How Deep Modules Constrain AI-Generated Code

Designing codebases with deep modules is not just a style choice but a control mechanism that reduces structure-sensitive tests and brittle code generated by AI agents.

Angle

Connect software architecture to agent reliability, argue for refactoring as an AI-quality intervention, and provide metrics to evaluate impact.

Source video

Project ideas

4

Tautology Linter for AI-Generated Tests

beyond-evals

A static analysis tool can detect tautological and structure-sensitive tests with >90% precision and recall on a corpus of AI-generated test suites.

Proof of concept

Build a linter that parses test assertions, identifies patterns like constant-equals-constant or assertions mirroring implementation structure, and flags them. Test on a dataset of AI-generated tests from a representative codebase.

Measurement

Compare tool output against human-labeled ground truth; compute precision, recall, and F1 score.

Source video

Separate Review Agent vs Global Prompt A/B Test

gatehouse

Teams using a dedicated review agent with CODING_STANDARDS.md will show a statistically significant reduction in style-related review comments and review cycle time compared to teams using global instructions in the implementer prompt.

Proof of concept

Implement two pipelines in a controlled repository: one with global coding standards in the implementer prompt, one with a separate /code-review agent reading CODING_STANDARDS.md. Run both on a set of feature PRs generated by the same AI implementer.

Measurement

Count of style violations per PR, human review time per PR, and developer satisfaction survey.

Source video

Impact of Commit-Producing vs Comment-Producing Automated Reviewers

gatehouse

Automated reviewers that commit fixes directly reduce human review time by at least 30% without increasing defect escape rate compared to reviewers that only leave comments.

Proof of concept

Instrument a CI pipeline to randomly assign PRs to two reviewer modes: commit-mode and comment-mode. Track review time and post-merge defects for a period.

Measurement

Average human review time per PR, defect escape rate (bugs found in production per PR), and human trust survey.

Source video

Quantifying the Effect of Deep Module Refactoring on AI-Generated Test Quality

new

Refactoring a codebase to deep modules reduces the incidence of structure-sensitive tests generated by AI agents by at least 50%.

Proof of concept

Select a codebase with shallow modules. Have an AI agent generate tests for a set of features. Measure brittle test incidence. Refactor to deep modules. Have the same agent generate tests for equivalent features. Measure again.

Measurement

Proportion of generated tests classified as structure-sensitive by a static analyzer or human labelers.

Source video

Architectural implications

3

Implementation and review responsibilities are being split into separate specialized agents instead of one generalist agent.

Before

A single agent prompt handles exploration, coding, debugging, and style enforcement.

After

Distinct /implement and /code-review agents operate with separate context windows and standards artifacts (CODING_STANDARDS.md).

Consequence

Reduced instruction interference and higher-quality output, at the cost of orchestrating multiple agents and passing artifacts between them.

Source video

Review agents are granted write authority (committing fixes) rather than read-only advisory output.

Before

AI reviewers leave comments that humans must parse and apply.

After

AI reviewers commit fixes directly, and humans review finished artifacts.

Consequence

Human review shifts from remediation to verification; requires careful permissioning and audit trails for autonomous commits.

Source video

Codebase abstraction quality is treated as a defense layer against bad AI-generated code and tests.

Before

Architecture is a style/maintainability concern orthogonal to AI tooling.

After

Deep modules and clear seams are used deliberately to constrain what a code-generating agent can produce.

Consequence

Refactoring becomes an agent-reliability investment, coupling architecture decisions to QA outcomes.

Source video

Tradeoffs and failure modes

3

Cheap automated checks

Benefit

Checks cost CPU cycles rather than tokens or human effort, making them a cheap first gate.

Cost or risk

Green CI can be misleading because AI-written tests are tautological or structure-sensitive, producing false confidence.

Green CI does not mean code is ready to merge because checks can lie.
Open source video
Source video

Automated review producing commits instead of comments

Benefit

PRs become polished artifacts and humans avoid parsing and applying feedback.

Cost or risk

Granting review agents commit authority changes auditability and control; reviewers may rubber-stamp agent-authored fixes.

Reviewer agents that directly commit fixes turn PRs into polished artifacts.
Open source video
Source video

Optimizing human review for speed

Benefit

Faster understanding of the 'why' reduces the human bottleneck.

Cost or risk

Compression via summaries and diagrams can hide detail; triage by one-way vs two-way door may misjudge risk.

Some reviews are more important than others; distinguish one-way from two-way doors.
Open source video
Source video

Open questions

4

How reliably can tautological or structure-sensitive tests be detected automatically?

Why unresolved

The summary identifies these failure modes but does not describe a detection mechanism or metric.

Research direction

Build a linter/benchmark that flags tests asserting constants matching implementation or bound to internal structure, and measure false-positive/negative rates.

Source video

Does moving coding standards out of the implementer prompt measurably improve output versus global instructions?

Why unresolved

The recommendation is asserted but no comparative result is provided in the summary.

Research direction

Run controlled A/B experiments comparing global-prompt vs. dedicated reviewer-agent enforcement on code-quality and review-latency metrics.

Source video

What net effect does agent-committed review fixes have on human trust and downstream defect rates?

Why unresolved

The summary claims polished PRs but provides no measurement of quality or regressions.

Research direction

Instrument pipelines to track defect escape rates and review time for commit-producing vs. comment-producing reviewers.

Source video

Does deep-module refactoring quantitatively reduce structure-sensitive test generation by agents?

Why unresolved

The causal link is claimed but not demonstrated with data in the summary.

Research direction

Measure brittle-test incidence before and after architecture refactors on the same codebase.

Source video

Key claims

7
causalVerification needed

AI coding agents generate so many PRs that they exacerbate the existing PR bottleneck.

Evidence

AI agents promise to scale developer output, leading to more PRs and work.

Question

Does AI agent adoption measurably increase PR volume and review queue depth in real teams?

Source video
causalVerification needed

Automated checks can 'lie' because AI-generated tests are often tautological or structure-sensitive.

Evidence

Green CI does not mean code is ready to merge because checks can lie.

Question

What fraction of AI-generated tests exhibit tautological or structure-sensitive patterns, and can they be detected reliably?

Source video
opinionVerification needed

Coding standards should not be placed in the implementer agent; implementation and review must be separate.

Evidence

Do not put coding standards in your implementer agent; keep implementation and review separate.

Question

Does separating standards into a reviewer agent outperform unified prompting on quality metrics?

Source video
opinionVerification needed

Automated review should output commits rather than comments.

Evidence

Automated review should output commits, not comments.

Question

Do commit-producing reviewers reduce human review time without increasing defect escape rates?

Source video
causalVerification needed

Deep modules reduce structure-sensitive tests and make it harder for agents to write bad code.

Evidence

Deep modules improve codebase design and reduce brittle tests.

Question

Is there a measurable reduction in brittle tests after refactoring to deep modules?

Source video
causalVerification needed

Optimizing human review around 'why', reversibility, and visual summaries reduces the human review bottleneck.

Evidence

Making understanding the 'why' as fast as possible reduces the human review bottleneck.

Question

Do visual summaries and one/two-way door triage measurably shorten review cycle time?

Source video
causalVerification needed

A three-layered defense (automated checks, automated review, human review) prevents bad code from reaching human eyes.

Evidence

Layering defense mechanisms prevents bad code from ever reaching human eyes.

Question

What proportion of defects are caught at each layer in practice?

Source video

Connections

4