AI Engineer · Published 2026-09-30

The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI

privateReview: unreviewedConfidence: high
Code reviewAI agentsmergeabilityevalssoftware engineeringcode-reviewai-agentsHuman-in-the-loopprompt-injectionbottleneck-analysisreview-harness
Open on YouTube ↗

Summary

Overview

  • Speaker: Laurie Voss
  • Channel: AI Engineer
  • Main topic: The impact of AI code generation on code review bottlenecks and why human review remains essential.
  • Purpose: To provide data-backed insights on how engineering teams can adapt to the AI coding revolution by rethinking code review and moving up levels of abstraction. Laurie Voss, Head of Developer Relations at Arize AI, analyzes economic and telemetry studies on how AI coding agents affect software development workflows. While AI has drastically lowered the cost of generating code (yielding up to 8x faster coding speeds), it has shifted the bottleneck to code reviews, which still operate at human speed. The talk explores industry experiments by companies like OpenAI, Anthropic, and Cursor, examining whether code reviews can be fully automated or if 'mergeability' requires human-in-the-loop judgment.

Topic Map

The Code Generation vs. Code Review Bottleneck

  • Explanation: Discusses economic data showing that developers using AI agents write 741% more lines of code but ship only 30% more software, highlighting code review as the primary bottleneck.
  • Key claims:
    • AI agents make generating code 8x faster, but shipping software only 1.3x faster.
    • Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.
  • Examples:
    • Economists tracking over 100,000 GitHub developers using AI telemetry.
  • Terminology:
    • autonomous agents
    • code review
    • bottleneck
  • Why it matters: It demonstrates that code generation is no longer the primary constraint in software engineering; verification and trust are.

Can We Skip Human Code Review?

  • Explanation: Examines attempts by companies like OpenAI and Anthropic to automate code reviews entirely or rely on agent-to-agent reviews.
  • Key claims:
    • OpenAI built an internal product with zero manually written code using 3 engineers over 5 months.
    • Anthropic's security reviewer can be talked out of its findings by the very code it is reviewing.
  • Examples:
    • Stripe and Anthropic migrating millions of lines of code to new languages in days using AI agents.
  • Terminology:
    • human-in-the-loop
    • prompt injection
    • false positives
  • Why it matters: Shows that fully removing humans from review loops introduces severe vulnerabilities and slop accumulation.

The Problem with Mergeability and Benchmarks

  • Explanation: Explores how passing test suites is not equivalent to mergeability, highlighting research from Meter, Cognition, and others.
  • Key claims:
    • Passing tests only checks behavioral correctness, not external code quality or subtle bugs.
    • AI models score significantly lower on human review rubrics than on standard test execution.
  • Examples:
    • Meter hiring open-source maintainers to review pull requests and finding automated reviews failed 50% of the time on mergeability.
  • Terminology:
    • mergeability
    • frontier code
    • swe-bench
    • false positives
  • Why it matters: Establishes that 'mergeability' is the ultimate benchmark for next-generation AI coding models.

The Evolution of the Human Engineering Role

  • Explanation: Argues that human reviewers are moving up the stack from reviewing line-by-line code to designing review systems, tests, and abstractions.
  • Key claims:
    • Code review isn't dead, but it is transforming into an engineered system.
    • Humans must codify domain knowledge and company context into review harnesses rather than reading diffs manually.
  • Examples:
    • Dexter Forthy's public reversal after trying 6 months of no code review and having to rip out large parts of his system.
  • Terminology:
    • review harness
    • higher level of abstraction
    • production
  • Why it matters: Defines the winning strategy for engineering teams in the AI era: investing in reliable review guardrails.

Key Points

Code generation is cheap; verification is expensive

  • Explanation: While AI has made producing plausible code extremely cheap, knowing whether to trust it remains high-cost and time-consuming.
  • Evidence: Studies show +741% lines of code written vs +30% software shipped.
  • Practical implication: Engineering teams must optimize for review throughput rather than raw code output.

Tests do not capture everything

  • Explanation: Passing automated test suites does not guarantee code is mergeable or free of external side effects.
  • Evidence: Meter study showed open-source maintainer PRs passing test suites failed human mergeability checks 50% of the time.
  • Practical implication: Relying solely on test pass rates leads to silent code quality degradation.

Automated reviewers suffer from false positives and prompt injection

  • Explanation: AI review agents can be easily fooled by vulnerable code dressed up in clean commit messages.
  • Evidence: Anthropic's security reviewer fooled 88% of the time by vulnerable code.
  • Practical implication: Human meta-review or robust multi-pass systems are required to validate automated reviewers.

Frameworks, Models & Processes

Multi-Pass Review Strategy

  • How it works: Running multiple review passes over code diffs with shuffled reviewer orders and explicit instructions to be suspicious.
  • Components:
    • Multiple review passes
    • Shuffled reviewer order
    • Default suspicion prompt
  • When to use: When scaling automated code review pipelines using LLMs.

Examples & Case Studies

OpenAI built an internal product with ~1 million lines of code written entirely by agents over 5 months using 3 engineers.

  • Illustrates: AI's unprecedented capacity for autonomous code generation.
  • Lesson: Agents can scaffold entire projects, but require rigorous review harnesses.

Dexter Forthy stopped code review for 6 months, resulting in catastrophic system degradation that required ripping out large parts of the codebase.

  • Illustrates: The danger of removing human oversight from code integration.
  • Lesson: Please read the code; skipping review entirely does not work.

Actionable Takeaways

  • Immediate:
    • Stop reviewing pull requests line-by-line if it leads to burnout.
    • Invest in building reliable review harnesses and automated testing pipelines.
  • Strategic:
    • Codify domain knowledge and company context into automated evaluation frameworks.
    • Move human effort up the stack from code execution to system design and review guardrails.
  • Questions to investigate:
    • How do we build a robust benchmark for true code mergeability?
    • How can we effectively meta-review our automated code review agents?

Claims Worth Verifying

  • Developers using autonomous agents wrote 741% more code but shipped 30% more software. (statistical study)
  • Automated reviewers can be misled by vulnerable code 88% of the time. (experimental finding)

Notable Quotes

"The cost of producing plausible code has collapsed. The cost of knowing whether to trust it has not." (at 3:37) "Humans may review pull requests, but aren't required to." (at 8:32) "Passes tests is not the same as mergeable." (at 10:40) "Whoever writes the definition of good writes next year's models." (at 16:48) "The teams that win the next few years won't be the ones that generate the most code. They'll be the ones who can say with evidence why they trust what they shipped." (at 38:27)

Compressed Summary

  • AI code generation is 8x faster, but shipping is only 1.3x faster due to review bottlenecks.
  • Passing tests does not equal mergeability; external code quality and side effects matter.
  • Automated reviewers struggle with false positives and prompt injection.
  • Humans must move up the stack from writing code to building review systems and eval harnesses.
  • Keywords: code review, ai agents, mergeability, evals, software engineering
  • Core insight: Code review isn't dead, but humans must move up the stack from line-by-line inspection to designing robust automated review systems and evals.

Core insights

6
Empirical Resultmedium noveltystrong evidence

Accelerating code generation without accelerating verification just relocates the bottleneck: AI agents made generation ~8x faster but shipping only ~1.3x faster, with developers writing +741% lines of code for +30% shipped software. The constraint moves to the slowest unaccelerated step (human review).

Why it matters

It reframes AI coding ROI: investment should target review/verification throughput, not generation speed, because generation is no longer the limiting factor. Any team measuring only lines-of-code or generation velocity is optimizing an unconstrained stage.

Generalization

In any multi-stage pipeline where an upstream stage is automated and a downstream stage is not, throughput gains cap at the downstream stage's rate, and the unaccelerated stage becomes the system's design center.

AI agents make generating code 8x faster, but shipping software only 1.3x faster.
Open source video
Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.
Open source video
Studies show +741% lines of code written vs +30% software shipped.
Open source video
Empirical Resulthigh noveltystrong evidence

Passing a test suite is a behavioral-correctness signal only, not a mergeability signal: it does not capture external code quality or subtle side effects. In one study, PRs that passed test suites still failed human mergeability checks 50% of the time.

Why it matters

Any agent loop gated on 'tests pass' will silently admit code that humans would reject half the time in that study, causing quality degradation that tests cannot detect. It argues for a separate mergeability evaluation layer distinct from test execution.

Generalization

Automated pass/fail gates measure only the dimension they encode; using them as a proxy for a broader acceptance criterion (mergeability, deployability, safety) creates systematic false negatives that compound over time.

Passing automated test suites does not guarantee code is mergeable or free of external side effects.
Open source video
Meter study showed open-source maintainer PRs passing test suites failed human mergeability checks 50% of the time.
Open source video
Passes tests is not the same as mergeable.
Open source video
Failure Modehigh noveltymoderate evidence

Automated reviewers consume the very artifact they judge as untrusted input, making them attackable: Anthropic's security reviewer was fooled 88% of the time by vulnerable code presented convincingly (e.g., clean commit messages). Self-review and agent-to-agent review inherit a prompt-injection trust problem.

Why it matters

A reviewer agent that reads the diff it is reviewing has no trusted channel; adversarial or merely well-phrased code can suppress findings. This is a security boundary problem, not just an accuracy problem, and it constrains how far review can be delegated.

Generalization

Any evaluator whose input is also the object under evaluation and is producer-controlled is vulnerable to persuasive/injected framing; separating the evaluator's trust domain from the artifact's content is required.

Anthropic's security reviewer fooled 88% of the time by vulnerable code.
Open source video
Anthropic's security reviewer can be talked out of its findings by the very code it is reviewing.
Open source video
Mechanismmedium noveltymoderate evidence

A concrete mitigation for automated review is a multi-pass reviewer: run several review passes over the same diff with shuffled reviewer order and an explicit default-suspicion instruction, rather than a single trusting pass.

Why it matters

It offers a directly implementable harness design (ensembling/reordering + adversarial stance prompt) that targets the false-positive and dismissal failure modes of single-pass LLM review, and can be evaluated as a benchmarked configuration.

Generalization

For LLM-as-judge style tasks, decorrelating passes by order randomization and defaulting the evaluator to a suspicious prior reduces single-shot framing errors at the cost of additional inference.

Running multiple review passes over code diffs with shuffled reviewer orders and explicit instructions to be suspicious.
Open source video
Components: Multiple review passes, Shuffled reviewer order, Default suspicion prompt
Open source video
Architecturemedium noveltymoderate evidence

Review is being refactored from a manual human activity into an engineered artifact: humans codify domain knowledge and company context into review harnesses and evaluation frameworks, moving up the stack from line-by-line diffs to designing the review system itself.

Why it matters

It relocates where engineering effort belongs: the durable asset becomes the harness/eval context (which encodes org-specific mergeability criteria), not reviewer attention on individual PRs. This is what makes review throughput scalable without deleting review.

Generalization

When a judgment task is partially automated, the human role migrates to authoring the evaluator's specification and guardrails, and that specification becomes the maintainable, versioned system of record.

Codify domain knowledge and company context into automated evaluation frameworks.
Open source video
Code review isn't dead, but it is transforming into an engineered system.
Open source video
Humans must codify domain knowledge and company context into review harnesses rather than reading diffs manually.
Open source video
Mental Modelhigh noveltymoderate evidence

The economic asymmetry is about trust, not text: the cost of producing plausible code has collapsed while the cost of knowing whether to trust it has not. Plausibility and trustworthiness have decoupled.

Why it matters

It names the core variable teams should instrument. Optimizing plausibility (fast generation, passing tests) does not move trust, so trust-establishing mechanisms (evals, mergeability rubrics, meta-review) are the scarce resource.

Generalization

When generative systems make outputs cheap and convincing, value shifts to verification and provenance; the scarcity is epistemics, not production.

The cost of producing plausible code has collapsed. The cost of knowing whether to trust it has not.
Open source video

Deep dives

5

Mergeability as a first-class evaluation target distinct from test execution

Research question

Can a human-reviewed mergeability rubric be encoded as a reproducible evaluation dataset whose scores diverge measurably from test-pass rates, and what rubric dimensions account for that divergence?

Why

It attacks the central claim that 'passing tests is not mergeable': if divergence between test-pass and human-mergeability scores can be quantified and decomposed, teams get a concrete, versioned acceptance layer instead of an unfalsifiable slogan, and model selection can optimize the right signal.

Passing tests only checks behavioral correctness, not external code quality or subtle bugs.
Open source video
Meter hiring open-source maintainers to review pull requests and finding automated reviews failed 50% of the time on mergeability.
Open source video
AI models score significantly lower on human review rubrics than on standard test execution.
Open source video
Source video

Trust-domain separation for reviewer agents (meta-review and reviewer attackability)

Research question

If the reviewer's evaluative channel is structurally separated from the artifact's persuasive content, how much does reviewer false-negative and dismissal rate drop compared to a reviewer that reads the raw diff?

Why

A reviewer that ingests the artifact it judges has no trusted channel, so persuasion and injection convert directly into suppressed findings; this is a security boundary design question, not a model-quality question, and it bounds how far review can be delegated.

Anthropic's security reviewer can be talked out of its findings by the very code it is reviewing.
Open source video
Shows that fully removing humans from review loops introduces severe vulnerabilities and slop accumulation.
Open source video
Source video

Multi-pass, order-shuffled, suspicion-defaulted review harnesses as a benchmarked configuration

Research question

How much do N shuffled-order passes with an explicit default-suspicion instruction reduce false negatives relative to a single trusting pass, and how does the reduction scale with pass count and cost?

Why

It converts a vague 'keep humans in the loop' stance into a tunable harness parameter with a measurable cost curve, making automated review an engineering artifact that can be benchmarked rather than an all-or-nothing replacement.

Passing tests only checks behavioral correctness, not external code quality or subtle bugs.
Open source video
Meter hiring open-source maintainers to review pull requests and finding automated reviews failed 50% of the time on mergeability.
Open source video
Source video

Minimal human-in-the-loop configurations between full manual review and no review

Research question

Which intervention points (spec/rubric authoring, sampling of diffs, meta-review of reviewer findings) preserve mergeability while keeping shipped-software throughput above the human-review ceiling?

Why

Both extremes are evidenced to fail: manual review caps throughput at unchanged human speed, and removing review produces slop accumulation and severe vulnerabilities. The middle design space is where deployable practice lives and it is currently undefined.

Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.
Open source video
Shows that fully removing humans from review loops introduces severe vulnerabilities and slop accumulation.
Open source video
OpenAI built an internal product with zero manually written code using 3 engineers over 5 months.
Open source video
Source video

Segmenting the generation/shipping asymmetry by review practice

Research question

Does the lines-written versus software-shipped asymmetry vary systematically with review harness maturity, team size, and codebase type, or is it a uniform industry aggregate?

Why

If the asymmetry is flat, the bottleneck is structural and universal; if it varies with harness maturity, then review infrastructure is a measurable lever and teams can be ranked and benchmarked on it rather than on generation velocity.

AI agents make generating code 8x faster, but shipping software only 1.3x faster.
Open source video
Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.
Open source video
Source video

Article ideas

4

Your CI Green Light Is Not a Merge Signal: The Case for a Separate Mergeability Gate

Test-pass gates encode behavioral correctness and silently admit code humans would reject; teams should run mergeability as a distinct pipeline stage with its own human-derived rubric, because a single pass/fail gate cannot proxy a broader acceptance criterion without compounding false negatives.

Angle

Dismantle the assumption that 'tests pass' equals 'safe to merge' by contrasting test-execution scoring against human review rubrics and tracing the practical consequence of a missing acceptance layer.

Source video

Your Reviewer Agent Is Inside the Blast Radius

An automated reviewer that reads the diff it is judging has no trusted channel, so persuasive or injected framing in the artifact is indistinguishable from evidence about the artifact; review reliability therefore requires trust-domain separation and a meta-review tier, not a stronger model.

Angle

Reframe automated review failures as a security-boundary problem rather than an accuracy problem, and derive concrete architectural separation requirements.

Source video

Generation Got 8x Faster and Shipping Got 1.3x Faster: Stop Optimizing the Unconstrained Stage

When an upstream stage is automated and a downstream stage is not, throughput caps at the downstream stage, so reporting lines-of-code and generation velocity as productivity measures rewards investment in the stage that is no longer the constraint; the review harness is the design center.

Angle

Use the generation-versus-shipping asymmetry to argue that AI coding ROI metrics should instrument verification throughput, and show what a review-throughput budget would look like.

Source video

Code Review Isn't Dead, It Changed Substrate

Review is migrating from human attention on individual diffs to human-authored, versioned review harnesses that encode org-specific mergeability criteria, which makes the harness the durable engineering asset and redirects reviewer effort to spec and guardrail authoring.

Angle

Trace the shift in where engineering effort lands and what it means for reviewer roles, harness maintenance, and eval ownership.

Source video

Project ideas

4

MergeabilityRubric-Bench

beyond-evals

A human-authorable mergeability rubric applied to the same PRs that pass their test suites will flag a substantial fraction of test-passing PRs as unmergeable, and rubric scores will correlate with human maintainer decisions better than test-pass outcomes do.

Proof of concept

Build an evaluation dataset of PRs that pass CI, have domain maintainers score each on an explicit multi-dimension mergeability rubric (external quality, side effects, subtle bug risk), then score the same PRs with frontier models under both a test-pass lens and a rubric lens, reporting rank correlation against maintainer verdicts.

Measurement

Divergence rate between test-pass and human-unmergeable verdicts; Spearman correlation of model rubric scores versus maintainer scores; per-dimension false-negative rates.

Source video

SuspicionEnsemble Reviewer

gatehouse

Multiple review passes over identical diffs with shuffled reviewer order and a default-suspicion instruction reduce reviewer false negatives on vulnerable-code diffs compared with a single trusting pass, and false-negative rate falls monotonically with pass count up to a saturation point.

Proof of concept

Build a harness that runs K passes with randomized pass ordering and a suspicion-defaulted prompt, then evaluate on a red-team set of vulnerable diffs framed with clean commit messages and plausible structure; ablate pass count, ordering shuffles, and instruction stance.

Measurement

False-negative rate versus single-pass baseline; delta per additional pass; cost in tokens/latency per incremental true finding.

Source video

Reviewer Trust-Boundary Harness

gatehouse

Removing persuasive content from the reviewer's input channel (structuring the diff and commit narrative so producer-authored framing cannot act as evidence) reduces the rate at which a reviewer is talked out of its own findings.

Proof of concept

Implement three reviewer input modes — raw diff plus author narrative, diff with narrative stripped, and diff plus narrative isolated as untrusted quoted data — and measure finding-dismissal behavior on the same adversarial diff set.

Measurement

Finding-suppression rate per input mode; proportion of cases where the reviewer reverses an initially correct finding after reading producer framing.

Source video

Human-In-The-Loop Placement Sweep

movement-lab

There exists a human intervention placement (rubric authoring plus sampled meta-review) that retains mergeability at human-review levels while raising shipped-software throughput above the full-manual-review ceiling, and it outperforms both full manual review and zero human review.

Proof of concept

Configure a review pipeline where the human role is parameterized across placements — none, rubric author only, sampling reviewer of diffs, meta-reviewer of agent findings, full manual — and run each configuration against a fixed backlog of agent-generated changes with a mergeability success/failure outcome per change.

Measurement

Shipped-change throughput per configuration versus human mergeability acceptance rate; frontier of throughput against mergeability rejection rate; human hours consumed per 100 merged changes.

Source video

Architectural implications

3

Human review has become the throughput ceiling now that generation is ~8x faster while shipping is only ~1.3x faster.

Before

Optimize the code-writing step; reviewers read diffs manually as a downstream chore.

After

Treat the review harness as the primary production system with its own evals, throughput budget, and iteration loop.

Consequence

Engineering investment shifts from code-generation tooling to review/verification infrastructure, and the reviewer's output becomes guardrail specifications rather than inline comments.

Source video

Test-suite gates and mergeability are distinct acceptance criteria (PRs passing tests still failed human mergeability checks 50% of the time).

Before

A single CI gate ('tests pass') admits code to merge.

After

Introduce a separate mergeability evaluation layer, since tests only cover behavioral correctness, not external quality or subtle side effects.

Consequence

Requires a new benchmark and evaluation rubric for mergeability, and a pipeline stage distinct from CI test execution.

Source video

Automated reviewers are themselves untrusted components vulnerable to adversarial artifacts (security reviewer fooled 88% of the time).

Before

Trust the automated reviewer as a reliable gate or replace human review with it.

After

Layer human or multi-pass meta-review on top of automated reviewers, with the reviewer's trust domain separated from the reviewed artifact.

Consequence

Adds a meta-review tier and probably an ensemble of review passes; full removal of humans is not viable.

Source video

Tradeoffs and failure modes

4

Removing human review entirely

Benefit

Eliminates the human-speed bottleneck and maximizes autonomous shipping velocity.

Cost or risk

Catastrophic system degradation requiring ripping out large parts of the codebase; skipping review entirely does not work.

Dexter Forthy stopped code review for 6 months, resulting in catastrophic system degradation that required ripping out large parts of the codebase.
Open source video
Source video

Single-pass automated review

Benefit

Cheap, scalable, fast review without human attention.

Cost or risk

High false-positive and dismissal rates; the reviewer can be talked out of its own findings by the code it reviews, and is fooled 88% of the time by vulnerable code.

Anthropic's security reviewer can be talked out of its findings by the very code it is reviewing.
Open source video
Source video

Line-by-line human review at AI generation volume

Benefit

Preserves human judgment and institutional context on each change.

Cost or risk

Human review speed is unchanged, so it caps total throughput and produces reviewer burnout.

Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.
Open source video
Source video

Trusting test pass rates as quality proxy

Benefit

Simple, automatable, universally available signal for gating merges.

Cost or risk

Silent code quality degradation because tests do not capture mergeability or side effects.

Passing automated test suites does not guarantee code is mergeable or free of external side effects.
Open source video
Source video

Open questions

4

How do we build a robust benchmark for true code mergeability?

Why unresolved

Mergeability is defined by human review rubrics and external quality that existing test-execution benchmarks (e.g., SWE-bench-style pass/fail) do not encode, and automated reviewers disagree with human judgment roughly half the time in the cited study.

Research direction

Construct human-reviewed mergeability rubrics as an evaluation dataset and measure model scores against them, rather than against test execution.

Source video

How can we effectively meta-review our automated code review agents?

Why unresolved

Reviewer agents consume the untrusted artifact under review and can be persuaded out of findings and fooled 88% of the time, so the validation layer for reviewers is not established.

Research direction

Evaluate multi-pass, order-shuffled, suspicion-defaulted ensembles, or a separate human meta-review tier, and measure how much each reduces false-negative findings.

Source video

What is the minimal human-in-the-loop configuration that preserves mergeability without recreating the human-speed bottleneck?

Why unresolved

Evidence shows both extremes fail: full manual review caps throughput, while 6 months of no review caused catastrophic degradation; the middle design space is undefined.

Research direction

Prototype review harnesses that vary where humans intervene (spec authoring, sampling, meta-review) and measure shipped-software throughput versus mergeability failure rates.

Source video

Does the +741% lines / +30% shipped asymmetry hold across teams of different sizes and domains?

Why unresolved

The figures come from telemetry on 100,000+ GitHub developers, but the studies are summarized as aggregated snapshots without breakdown by team maturity, domain, or review practice.

Research direction

Replicate the telemetry analysis segmented by review process, harness maturity, and codebase type.

Source video

Key claims

8
comparativeVerification needed

AI agents make code generation ~8x faster, but shipping software only ~1.3x faster, with a +741% increase in lines written versus +30% shipped.

Evidence

Studies show +741% lines of code written vs +30% software shipped.

Question

What is the exact study, sample, and definition of 'lines written' and 'software shipped' behind the 741%/30% and 8x/1.3x figures?

Source video
causalVerification needed

Human review speed has not changed, making review the industry-wide bottleneck.

Evidence

Human review speed has stayed exactly the same, creating a massive industry-wide bottleneck.

Question

Is review speed measured directly, and does it remain constant across AI-assisted versus non-AI-assisted teams?

Source video
factualVerification needed

Automated reviewers can be misled by vulnerable code 88% of the time.

Evidence

Anthropic's security reviewer fooled 88% of the time by vulnerable code.

Question

What was the experimental setup, prompt, and attack method that produced the 88% fooling rate, and does it generalize beyond that reviewer?

Source video
comparativeVerification needed

Passing a test suite is not equivalent to mergeability; PRs passing tests failed human mergeability checks 50% of the time.

Evidence

Meter study showed open-source maintainer PRs passing test suites failed human mergeability checks 50% of the time.

Question

What rubric did the open-source maintainers apply, and is the 50% failure rate specific to automated reviews or to AI-generated PRs generally?

Source video
factualVerification needed

OpenAI built an internal product with ~1 million lines of code written entirely by agents over 5 months using 3 engineers.

Evidence

OpenAI built an internal product with ~1 million lines of code written entirely by agents over 5 months using 3 engineers.

Question

How much human review was involved per merged change, and what was the defect/rollback rate?

Source video
factualVerification needed

Stripe and Anthropic migrated millions of lines of code to new languages in days using AI agents.

Evidence

Stripe and Anthropic migrating millions of lines of code to new languages in days using AI agents.

Question

What verification harness was used to confirm the migrated code was correct, and were the migrations fully merged into production?

Source video
opinionVerification needed

Multi-pass review with shuffled reviewer order and a default-suspicion instruction is the recommended way to scale LLM review pipelines.

Evidence

Running multiple review passes over code diffs with shuffled reviewer orders and explicit instructions to be suspicious.

Question

How much do shuffled-order multi-pass reviews reduce false negatives compared with single-pass review, and at what cost?

Source video
causalVerification not requested

Stopping code review entirely causes catastrophic system degradation.

Evidence

Please read the code; skipping review entirely does not work.

Source video

Connections

5