AI Engineer · Published 2026-08-17

How to Kill the Code Review — Ankit Jain, Aviator

privateReview: unreviewedConfidence: high
Code reviewai verificationalignmentautomationintent-drivencode-reviewai-verificationinvariantsllm-judgespec-driven-developmentintent-captureci-guardrails
Open on YouTube ↗

Summary

Overview

  • Speaker: Ankit Jain
  • Channel: AI Engineer
  • Main topic: Re-engineering code review processes in the era of AI-generated code by moving alignment upstream and automating semantic verification.
  • Purpose: To propose a new paradigm for code review and verification where AI handles generation and initial testing, humans review intent and architecture rather than line-by-line diffs, and recurring feedback is codified into automated guards. Ankit Jain, co-founder of Aviator, explores why traditional line-by-line human code reviews are failing as code volume increases exponentially due to AI agents. Presenting a four-part thesis, he argues that code reviews are fundamentally about alignment and knowledge sharing rather than just diff verification. He introduces 'Spec-Driven' and 'Intent-Driven' development frameworks, the AI Slop Register concept to codify recurring review comments into automated checks, and a verification system combining deterministic checks with runtime LLM judges to ensure high semantic accuracy without manual bottlenecking.

Topic Map

The Problem with Code Review Today

  • Explanation: With code volume increasing rapidly due to AI, traditional line-by-line code reviews have become a major bottleneck. Metrics show massive code churn, increased incident-to-PR ratios, and longer median review times, leading teams to bypass reviews or merge code without proper scrutiny.
  • Key claims:
    • Code volume and code churn are increasing exponentially, causing severe bottlenecks in code review.
    • Traditional line-by-line reviews are no longer effective, with incident-to-PR ratios and median review times increasing.
    • Over 30 percent of code changes are being merged today without any review at all.
  • Examples:
    • Metrics cited from Accelerate/Atlassian reports showing +861% code churn, +243% incidents-to-PR ratio, and +441% median time in review.
  • Terminology:
    • Code churn
    • Incidents-to-PR ratio
    • Medium time in review
  • Why it matters: It proves that human-driven line-by-line code review does not scale in the AI era and creates a massive bottleneck for engineering velocity.

Alignment vs. Semantic Accuracy

  • Explanation: Code review is historically about two things: semantic accuracy (catching bugs, conventions, security issues) and alignment (knowledge sharing, mentorship, architectural feedback, onboarding). AI can assist with semantic accuracy, but human alignment must survive upstream.
  • Key claims:
    • Code review is not just about code review; a large part of it is alignment, knowledge sharing, mentorship, and architectural feedback.
    • Semantic accuracy needs better tooling, but human alignment must survive.
  • Examples:
    • Contrasting solo coding projects where zero alignment is needed against team-based collaborative coding where knowledge sharing is vital.
  • Terminology:
    • Semantic accuracy
    • Alignment
    • Knowledge sharing
  • Why it matters: Recognizing that reviews serve an alignment and cultural purpose prevents teams from blindly trusting AI-generated code without team consensus.

Flaws in Spec-Driven Development

  • Explanation: Spec-driven development resembles waterfall methodology from the 1970s because it lacks a feedback loop: specs are written before the work teaches the author anything, leading to spec drift when implementation surfaces decisions the spec never anticipated.
  • Key claims:
    • Spec-driven development is waterfall in new clothes because there is no feedback loop.
    • LLMs are non-deterministic, so static specs fail to capture everything during implementation.
  • Examples:
    • Comparing 1970s waterfall models (Requirements -> Design -> Implementation -> Verification) with modern spec-driven AI development.
  • Terminology:
    • Spec-driven development
    • Waterfall
    • Specs drift
  • Why it matters: Highlighting why pure spec-driven AI generation falls short when encountering non-deterministic AI decisions.

Intent-Driven Development and the AI Slop Register

  • Explanation: Intent lives in Jira tickets, PRDs, and most importantly, interactive prompts where real decisions are made. The AI Slop Register captures recurring review comments and codifies them into automated guards so developers do not have to repeat the same review feedback manually.
  • Key claims:
    • Intent does not only live in a spec; it lives in Jira tickets, design docs, and prompts.
    • The AI Slop Register automatically compiles recurring code review comments into automated checks that run on every pull request.
  • Examples:
    • Using Aviator verify platform to codify reviewer comments into invariants like 'Use the Money type for currency amounts, not float'.
  • Terminology:
    • Intent-driven development
    • AI Slop Register
    • Invariants
  • Why it matters: It turns tribal knowledge and repetitive code review feedback into reusable, automated guardrails.

The New Verification System: Two Halves, One Loop

  • Explanation: Combining intent capture and semantic verification into a single loop. Sessions become acceptance criteria, criteria plus invariants create test plans, and preview environments run automated checks so reviewers review intent and evidence rather than raw diffs.
  • Key claims:
    • The session becomes the criteria, converting user responses into acceptance criteria and test plans.
    • Reviewers no longer review diffs line-by-line; they review intent, architecture, and verification evidence.
  • Examples:
    • A workflow showing Capture (Session -> Acceptance criteria), Build (AI Slop Register -> Test plan), Run (Preview env -> Results), and Review (Reviewer checks intent, not diff).
  • Terminology:
    • Acceptance criteria
    • Test plan
    • Preview environment
    • Deterministic check
    • LLM judge
  • Why it matters: It shifts the human reviewer's role from reading raw code lines to evaluating test evidence against stated intent.

Key Points

Traditional code review is broken at scale

  • Explanation: With 861% higher code churn and increasing incident rates, manual line-by-line review is a bottleneck that fails to keep up with AI-generated code productivity.
  • Evidence: Data from Accelerate and Atlassian reports showing surging PR volumes and review times.
  • Practical implication: Engineering teams must stop relying solely on manual line-by-line code reviews for AI-generated code.

Code review is primarily about alignment, not just inspection

  • Explanation: Reviews serve as knowledge sharing, architectural guidance, mentorship, and team alignment rather than just catching syntax errors.
  • Evidence: Comparison of solo coding versus team collaboration workflows.
  • Practical implication: New AI review tools must preserve human collaboration and architectural alignment.

Spec-driven development lacks feedback loops

  • Explanation: Writing specs upfront without iterative feedback leads to spec drift because LLM implementations encounter unforeseen edge cases.
  • Evidence: Parallels drawn with 1970s waterfall methodology.
  • Practical implication: Intent must be captured dynamically from prompts and interactive sessions during development.

The AI Slop Register automates recurring feedback

  • Explanation: Codifying repeated review comments into automated rules ensures the same mistakes are never manually reviewed twice.
  • Evidence: Aviator's verify platform feature demo showing automated invariant checks.
  • Practical implication: Engineers should mine past review comments to build automated guardrails.

Balance deterministic checks with LLM judges

  • Explanation: Use deterministic checks where possible (assertions, status codes) and LLM runtime judges where necessary (screenshots, fuzzy semantics).
  • Evidence: Framework distinguishing between deterministic code verification and AI-powered runtime evaluation.
  • Practical implication: The AI model acts as a tool inside the harness, never as the sole judge of record.

Frameworks, Models & Processes

The Five-Layers of Trust Model

  • How it works: A multi-layered framework to build trust into AI-generated code without requiring manual line-by-line reviews.
  • Components:
    • Compare multiple options
    • Deterministic guardrails
    • Acceptance criteria
    • Permission systems
    • Adversarial verification
  • When to use: When designing automated AI coding verification and governance pipelines.

Intent-Driven Verification Loop

  • How it works: Captures user session intent, transforms it into acceptance criteria and test plans, runs verification in preview environments, and presents evidence to reviewers instead of diffs.
  • Components:
    • Capture session intent
    • AI Slop Register and invariants
    • Test plan generation
    • Preview environment execution
    • Intent-based review
  • When to use: When replacing traditional pull request code review workflows with AI-augmented verification.

Examples & Case Studies

Ankit Jain discussed writing a post on Latent Space about the five layers of trust and how to kill code review.

  • Illustrates: The initial genesis of the framework for eliminating traditional code reviews.
  • Lesson: Frameworks evolve from practical industry bottlenecks observed across engineering teams.

Using Aviator verify to take user prompt intent ('Use the Money type for currency amounts, not float') and codify it into a persistent check.

  • Illustrates: How the AI Slop Register turns recurring review comments into automated checks.
  • Lesson: Every recurring comment should be codified into an automated guardrail so humans never review it twice.

Actionable Takeaways

  • Immediate:
    • Mine your last 1,000 review comments to identify repeatable patterns.
    • Build an AI Slop Register to codify recurring feedback into automated PR checks.
    • Shift review focus from raw diff inspection to intent and verification evidence.
  • Strategic:
    • Adopt intent-driven development over rigid spec-driven waterfall models.
    • Balance deterministic verification checks with LLM-powered runtime judges.
    • Preserve human alignment, mentorship, and architectural oversight while automating semantic code checks.
  • Questions to investigate:
    • How do we maintain team alignment and mentorship when AI agents generate the majority of code?
    • What is the optimal balance between deterministic test assertions and LLM runtime evaluation?
    • How can engineering organizations scale their verification systems as code generation velocity increases?

Claims Worth Verifying

  • Over 30 percent of pull requests are merged today with no review at all. (statistical)
  • Code churn has increased by 861% while median review time has increased by 441%. (statistical)

Notable Quotes

"When AI reviews AI in a nobody reads, you've configured the wrong thing." (at 3:35) "Code review is not about code review." (at 5:02) "Intent doesn't only live in a spec." (at 9:12) "Every recurring comment is a guardrail that you don't have to review again." (at 12:31) "Deterministic where you can. LLM where you must." (at 19:06)

Compressed Summary

  • Traditional line-by-line code reviews are failing due to exponential AI code generation and high code churn.
  • Code review is fundamentally about team alignment, mentorship, and architecture, not just syntax inspection.
  • Spec-driven development acts like waterfall without feedback loops, leading to spec drift.
  • The AI Slop Register turns recurring review comments into automated PR guardrails.
  • The new verification loop replaces diff reading with session intent capture, test plans, and preview environment evidence.
  • Keywords: code review, ai verification, alignment, automation, intent-driven
  • Core insight: Code review in the AI era must pivot from manual line-by-line diff inspection to upstream intent alignment and automated semantic verification.

Core insights

6
Mechanismhigh noveltymoderate evidence

The 'AI Slop Register' closes the loop from human review to automation: recurring code review comments are compiled into durable, machine-checkable invariants that execute on every pull request, so the same feedback is never given twice.

Why it matters

It reframes review feedback as a compilable artifact rather than ephemeral human attention, which is the only way review effort stops growing linearly with AI-generated PR volume.

Generalization

Any recurrent human judgment applied to agent output (style rules, schema constraints, domain invariants) should be promoted from a comment into a versioned, executable guard that runs automatically thereafter.

The AI Slop Register automatically compiles recurring code review comments into automated checks that run on every pull request.
Open source video
Using Aviator verify platform to codify reviewer comments into invariants like 'Use the Money type for currency amounts, not float'.
Open source video
Mental Modelmedium noveltymoderate evidence

Code review decomposes into two distinct functions—semantic accuracy (bugs, conventions, security) and alignment (knowledge sharing, mentorship, architectural feedback, onboarding)—and only the first is a legitimate target for automation; the second must be relocated rather than replaced.

Why it matters

It forces an explicit architectural decision about which human checkpoint to remove and where to reintroduce the alignment function, instead of assuming an AI reviewer substitutes for review wholesale.

Generalization

When automating a human-in-the-loop process, first separate the verification function from the socialization/consensus function; automate the former and re-site the latter upstream.

Code review is not just about code review; a large part of it is alignment, knowledge sharing, mentorship, and architectural feedback.
Open source video
Semantic accuracy needs better tooling, but human alignment must survive.
Open source video
Tradeoffhigh noveltymoderate evidence

Spec-driven development reproduces the 1970s waterfall failure mode because it has no feedback loop, and LLM non-determinism causes spec drift: implementation surfaces decisions the spec never anticipated, invalidating the spec as a source of truth.

Why it matters

It is a direct counter-argument to the dominant 'write a spec, then let agents build it' pattern and implies intent must be captured dynamically from the development session instead of frozen upfront.

Generalization

Static specifications degrade when the generating system is non-deterministic and the work itself is the primary source of new information; intent capture must be continuous, not front-loaded.

Spec-driven development is waterfall in new clothes because there is no feedback loop.
Open source video
LLMs are non-deterministic, so static specs fail to capture everything during implementation.
Open source video
Architecturemedium noveltymoderate evidence

Intent is distributed across Jira tickets, design docs, and—most importantly—interactive prompts, and the interactive session itself becomes the acceptance criteria, which then feed the test plan.

Why it matters

It relocates the authoritative artifact from a static document to a captured session trace, which is the input the verification pipeline actually needs to generate meaningful checks.

Generalization

Agentic pipelines should treat the interaction transcript as the primary requirements artifact, deriving structured acceptance criteria from it rather than assuming a pre-written spec exists.

Intent does not only live in a spec; it lives in Jira tickets, design docs, and prompts.
Open source video
The session becomes the criteria, converting user responses into acceptance criteria and test plans.
Open source video
Architecturemedium noveltymoderate evidence

Verification is implemented as a hybrid: deterministic checks plus runtime LLM judges executed in a preview environment, producing evidence the reviewer evaluates instead of a line-by-line diff.

Why it matters

It defines a concrete division of labor between cheap, reliable deterministic gates and expensive, semantically rich LLM judgment, and changes what review tooling must render.

Generalization

Layer deterministic verification first and reserve LLM judges for semantics that cannot be expressed as a check; surface results as evidence artifacts, not raw diffs, to the human escalation path.

a verification system combining deterministic checks with runtime LLM judges to ensure high semantic accuracy without manual bottlenecking.
Open source video
Reviewers no longer review diffs line-by-line; they review intent, architecture, and verification evidence.
Open source video
Empirical Resultlow noveltymoderate evidence

Manual line-by-line review has already collapsed in practice at current code volumes, with a large fraction of changes merged with no review at all, alongside steep increases in churn, incident-to-PR ratio, and time in review.

Why it matters

It quantifies the baseline failure, indicating that review bypass is the de facto current behavior rather than a hypothetical risk of AI-generated code.

Generalization

Human review gates degrade silently into rubber-stamping or bypass once submission volume exceeds human processing capacity; the gate must be redesigned, not exhortation-enforced.

Over 30 percent of code changes are being merged today without any review at all.
Open source video
+861% code churn, +243% incidents-to-PR ratio, and +441% median time in review
Open source video

Deep dives

5

Lifecycle management of an invariant registry compiled from review comments

Research question

How are compiled invariants kept current, deduplicated, and conflict-free as the codebase and conventions change, and how does the false-positive rate scale as rules accumulate?

Why

Compiling recurring review comments into durable machine-checkable invariants only pays off if the rule set does not rot into over-broad, conflicting, or stale checks that developers learn to suppress; without a curation model the automation decays into the same rubber-stamping it replaced.

The AI Slop Register automatically compiles recurring code review comments into automated checks that run on every pull request.
Open source video
Using Aviator verify platform to codify reviewer comments into invariants like 'Use the Money type for currency amounts, not float'.
Open source video
Source video

Comparative reliability of deterministic checks versus runtime LLM judges in a hybrid verification gate

Research question

What are the precision, recall, latency, and cost per PR of deterministic checks, runtime LLM judges, and the hybrid combination against a labeled defect set?

Why

The architecture explicitly layers cheap deterministic gates with expensive semantic LLM judgment, but the summary asserts 'high semantic accuracy' without any measured evaluation; the gate cannot be trusted or sized without numbers, and LLM judges are themselves non-deterministic, introducing verdict variability into the pipeline.

a verification system combining deterministic checks with runtime LLM judges to ensure high semantic accuracy without manual bottlenecking.
Open source video
Reviewers no longer review diffs line-by-line; they review intent, architecture, and verification evidence.
Open source video
Source video

Re-siting the alignment function after line-by-line diff review is removed

Research question

Once the diff checkpoint is removed, what mechanism preserves mentorship, architectural feedback, and onboarding, and what proxies can measure alignment quality?

Why

The framework separates semantic accuracy from alignment and identifies only the first as automatable, but it asserts alignment 'must survive' without specifying the replacement mechanism; if the re-siting is left implicit, removing the diff silently deletes knowledge sharing and architectural drift goes undetected.

Code review is not just about code review; a large part of it is alignment, knowledge sharing, mentorship, and architectural feedback.
Open source video
Semantic accuracy needs better tooling, but human alignment must survive.
Open source video
Source video

Extraction fidelity of acceptance criteria derived from interactive session transcripts

Research question

Can acceptance criteria and test plans be reliably derived from interactive session transcripts, and what fraction of intent is lost, mis-extracted, or leaves edge cases uncovered?

Why

The pipeline relocates the authoritative requirements artifact from a static document to a captured session trace and then feeds it into test-plan generation; any extraction loss propagates directly into the verification layer, so criteria fidelity bounds the whole system's coverage.

Intent does not only live in a spec; it lives in Jira tickets, design docs, and prompts.
Open source video
The session becomes the criteria, converting user responses into acceptance criteria and test plans.
Open source video
Source video

Measuring specification drift in agent-driven implementations

Research question

How much specification drift can be measured in practice between the original spec and the decisions actually made during an agent-driven implementation?

Why

The waterfall critique is a comparative argument that only holds if drift is empirically observable; if workflows that iteratively revise the spec show no measurable drift, the counter-argument against spec-driven development weakens and intent capture need not move fully into the session.

Spec-driven development is waterfall in new clothes because there is no feedback loop.
Open source video
LLMs are non-deterministic, so static specs fail to capture everything during implementation.
Open source video
Source video

Article ideas

4

Your Review Comments Are Bugs in Your CI Config

Every recurring human review comment is an unencoded invariant, and the correct response to repeated feedback is not a firmer comment but a compiled, versioned, executable guard; teams that treat review feedback as ephemeral attention will watch review cost scale linearly with AI-generated PR volume while teams that treat it as a rule artifact do not.

Angle

Argue for review feedback as a compilable engineering deliverable, then confront the uncomfortable part: an invariant registry needs ownership, conflict detection, and deprecation, and an unowned registry becomes a false-positive generator that developers route around.

Source video

Kill the Diff, Not the Reviewer

Code review is two functions wearing one name, and only semantic accuracy is a legitimate automation target; the alignment function must be deliberately re-sited upstream rather than assumed to survive, because the summary's claim that alignment 'must survive' specifies no mechanism for it.

Angle

Decompose review into verification versus socialization, show what each automation strategy deletes, and specify the human checkpoint that replaces the diff rather than promising AI review wholesale.

Source video

Spec-Driven Development Is Waterfall in New Clothes

Front-loading a static spec onto a non-deterministic generator reproduces the waterfall failure mode because the work itself is the primary source of new information; intent must be captured continuously from the session, making the spec a derived output rather than an authoritative input.

Angle

Take the counter-position against the dominant write-a-spec-then-generate pattern and demand measurement: how much drift is observable, and do iteratively revised specs still drift?

Source video

The 30 Percent No-Review Merge Is the Baseline, Not the Risk

Review bypass is already the de facto behavior at current volumes, not a hypothetical hazard of AI-generated code, so exhorting teams to review more carefully is a failing strategy; the gate must be redesigned rather than enforced harder, because human review gates degrade silently into rubber-stamping once submission volume exceeds processing capacity.

Angle

Use the churn, incident-to-PR, and time-in-review figures to frame automation as remediation of an existing collapse rather than a speculative bet, while flagging that the underlying data window and causal attribution remain unverified.

Source video

Project ideas

4

Invariant Registry with Conflict Detection

gatehouse

Enough natural-language review comments can be compiled into correct, non-flaky automated invariants that the compiled rule set catches recurring violations without human curation, and the registry's false-positive rate stays bounded as rules accumulate across many pull requests because automated conflict and overlap detection prunes contradictory rules.

Proof of concept

Build a registry that ingests reviewer comments, compiles candidates into executable checks (e.g., 'Use the Money type for currency amounts, not float'), runs them on a shadow corpus of historical PRs, and flags contradictory or overlapping rules for deprecation.

Measurement

Percentage of comments converted into invariants that fire correctly without hand-editing; false-positive rate as a function of registry size; number of conflicting rule pairs detected; developer override rate per PR over successive weeks.

Source video

Deterministic-vs-Judge Verification Benchmark

beyond-evals

A hybrid gate of deterministic checks plus runtime LLM judges reaches precision and recall on labeled PR defects that are not materially worse than human line-by-line review, at a lower latency and cost per PR than the manual baseline, and the deterministic layer alone accounts for the majority of caught defects.

Proof of concept

Assemble a labeled corpus of historical PRs with known defects, run three configurations (deterministic only, LLM judge only, hybrid) in a preview environment, and score each on precision, recall, latency, and cost.

Measurement

Precision/recall versus human review on the same changes; verdict variance across repeated LLM-judge runs on identical inputs; marginal defects caught by the judge layer that deterministic checks miss; seconds and dollars per PR.

Source video

Session-to-Criteria Extractor

movement-lab

Acceptance criteria and test plans extracted from interactive session transcripts cover edge cases as well as human-written criteria on the same task, and the residual intent loss is concentrated in a small, identifiable class of under-specified interactions rather than spread uniformly.

Proof of concept

Capture development sessions, extract structured acceptance criteria and test plans, and compare them head-to-head against criteria written by a human for the same task using defect-escape and coverage metrics.

Measurement

Defect escape rate for session-derived versus human-written criteria; coverage delta on edge cases; fraction of criteria traceable to an explicit session utterance; human-flagged mis-extractions per task.

Source video

Alignment Proxy Instrumentation

new

Alignment value lost when line-by-line diff review is removed is recoverable through upstream intent review sessions, and this recovery is detectable through proxies such as onboarding time to first independent contribution and architectural drift measured against declared intent, rather than being an unmeasurable cultural good.

Proof of concept

Instrument two comparable teams, one keeping diff review and one running upstream intent review plus automated verification, and track onboarding time, architectural drift against acceptance criteria, and knowledge-distribution metrics over a quarter.

Measurement

Time-to-first-independent-contribution for new joiners; rate of architectural changes that contradict prior acceptance criteria; concentration of authorship/decision knowledge across team members; self-reported mentorship touchpoints.

Source video

Architectural implications

3

The human reviewer's artifact of interest shifts from the diff to a verification evidence bundle (test results, invariant checks, LLM judge verdicts).

Before

Reviewer reads the raw code diff and reasons about semantics manually.

After

Reviewer evaluates intent, architecture, and generated verification evidence against stated acceptance criteria.

Consequence

CI/review tooling must produce and render machine-generated evidence and criteria links, and the system needs a stable trace from session intent to the checks that verify it.

Source video

Code review feedback becomes a durable, versioned rule set rather than transient human commentary.

Before

Reviewer restates the same convention on each offending PR; knowledge lives in reviewer memory and comment history.

After

Recurring comments are compiled into invariants executed automatically on every PR.

Consequence

Introduces a new maintenance surface—the invariant registry—which requires ownership, conflict detection, and deprecation as the codebase evolves.

Source video

Intent capture moves from a front-loaded document to a continuous session-derived artifact.

Before

Spec written before implementation; implementation is expected to conform to it.

After

Interactive session transcript yields acceptance criteria and test plans that drive the pipeline.

Consequence

The pipeline needs session capture and criteria extraction as first-class infrastructure, and specs become derived outputs rather than authoritative inputs.

Source video

Tradeoffs and failure modes

4

Codifying review comments into invariants

Benefit

Eliminates repeated manual feedback and turns tribal knowledge into reusable automated guardrails.

Cost or risk

Invariants abstracted from individual past comments can be over-broad, conflicting, or stale as the codebase evolves, requiring ongoing curation of the rule set.

It turns tribal knowledge and repetitive code review feedback into reusable, automated guardrails.
Open source video
Source video

Automating semantic accuracy with LLM judges

Benefit

Provides semantic verification coverage that deterministic checks cannot express, without human bottlenecking.

Cost or risk

LLM judges are themselves non-deterministic, so their verdicts introduce variability and cost into the verification gate; the summary offers no measured accuracy for the judge layer.

a verification system combining deterministic checks with runtime LLM judges to ensure high semantic accuracy without manual bottlenecking.
Open source video
Source video

Removing line-by-line human review

Benefit

Removes the throughput bottleneck created by exponential code volume growth.

Cost or risk

Alignment, mentorship, and architectural feedback can be lost entirely if not deliberately re-sited upstream; the summary asserts alignment 'must survive' but does not specify the replacement mechanism.

Semantic accuracy needs better tooling, but human alignment must survive.
Open source video
Source video

Spec-driven generation

Benefit

Provides an upfront, explicit statement of intended behavior for AI generation.

Cost or risk

Without a feedback loop, the spec drifts as non-deterministic implementations encounter unanticipated decisions, reproducing waterfall failure modes.

Spec-driven development is waterfall in new clothes because there is no feedback loop.
Open source video
Source video

Open questions

4

How are invariants in the AI Slop Register kept current, deduplicated, and conflict-free as the codebase and conventions change?

Why unresolved

The summary describes compiling comments into invariants but says nothing about lifecycle management, false-positive handling, or rule deprecation.

Research direction

Prototype an invariant registry with conflict detection over rule sets and measure false-positive rate as rules accumulate across many PRs.

Source video

What is the measured reliability of runtime LLM judges relative to deterministic checks, in terms of false positives, false negatives, latency, and cost per PR?

Why unresolved

The claim of 'high semantic accuracy' is asserted without any reported evaluation numbers in the summary.

Research direction

Build a benchmark of labeled PR defects and compare deterministic checks, LLM judges, and the hybrid combination on precision/recall and cost.

Source video

How is alignment (mentorship, architectural feedback, onboarding) preserved once line-by-line diff review is removed, and how would its quality be measured?

Why unresolved

The summary states alignment must survive but does not describe the replacement mechanism or any metric for it.

Research direction

Define proxies for alignment value (onboarding time, architectural drift, knowledge-distribution metrics) and test whether upstream intent review sessions preserve them.

Source video

Can acceptance criteria and test plans be reliably derived from interactive session transcripts, and what fraction of intent is lost or mis-extracted?

Why unresolved

The summary asserts the session becomes the criteria but provides no evidence on extraction fidelity or coverage of edge cases.

Research direction

Compare session-derived acceptance criteria against human-written criteria on the same task using defect-escape and coverage metrics.

Source video

Key claims

7
factualVerification needed

More than 30 percent of code changes are merged without any review at all.

Evidence

Over 30 percent of code changes are being merged today without any review at all.

Question

What data source and population underlie the 30 percent no-review merge figure, and is it reproducible?

Source video
causalVerification needed

Code churn, incidents-to-PR ratio, and median time in review have risen sharply.

Evidence

+861% code churn, +243% incidents-to-PR ratio, and +441% median time in review

Question

Over what time window and sample were these Accelerate/Atlassian-derived deltas measured, and is the causal attribution to AI code volume justified?

Source video
comparativeVerification needed

Spec-driven development is waterfall rebranded because it lacks a feedback loop.

Evidence

Spec-driven development is waterfall in new clothes because there is no feedback loop.

Question

Do spec-driven agent workflows that include iterative spec revision still exhibit the drift failure mode attributed to waterfall?

Source video
causalVerification needed

LLM non-determinism causes static specs to miss decisions that arise during implementation.

Evidence

LLMs are non-deterministic, so static specs fail to capture everything during implementation.

Question

How much specification drift can be measured in practice between the original spec and decisions made during an agent-driven implementation?

Source video
causalVerification needed

Recurring review comments can be automatically compiled into checks that run on every pull request.

Evidence

The AI Slop Register automatically compiles recurring code review comments into automated checks that run on every pull request.

Question

What proportion of natural-language review comments can be converted into correct, non-flaky automated invariants without human curation?

Source video
opinionVerification not requested

Intent resides in Jira tickets, design docs, and prompts, not only in a spec.

Evidence

Intent does not only live in a spec; it lives in Jira tickets, design docs, and prompts.

Source video
predictionVerification needed

A verification pipeline combining deterministic checks with runtime LLM judges achieves high semantic accuracy without a manual review bottleneck.

Evidence

a verification system combining deterministic checks with runtime LLM judges to ensure high semantic accuracy without manual bottlenecking.

Question

What semantic accuracy does the hybrid deterministic-plus-LLM-judge pipeline reach compared with human line-by-line review on the same changes?

Source video

Connections

3