AI Engineer · Published 2026-10-13

What Technical Due Diligence Finds When the Deck Says “70% Automated” — Soukaïna Abelhad

privateReview: unreviewedConfidence: high
due diligenceai automationproductivity metricsunit economicsevaluation maturitytechnical-due-diligencestraight-through-processingHuman-in-the-loopevaluation-maturityunit-economicsreview-bottleneckproductivity-metricsai-attribution
Open on YouTube ↗

Summary

Overview

  • Speaker: Soukaïna Abelhad
  • Channel: AI Engineer
  • Main topic: Evaluating AI automation claims and productivity metrics during technical due diligence
  • Purpose: Provide investors and technical leaders with a robust framework (the diligence card) to critically evaluate AI startup productivity claims, automation rates, and P&L impact. Soukaïna Abelhad shares insights from conducting technical due diligence on AI startups, explaining how to evaluate metrics like '55.8% faster' and '70% automated' through three critical checks: arithmetic, definition, and the tell. She explains how high AI adoption often increases human review workload and why financial translation of metrics is essential for investors and operators.

Topic Map

Introduction to AI Technical Due Diligence

  • Explanation: Introduction of the speaker and the premise of technical due diligence when encountering inflated AI automation metrics.
  • Key claims:
    • AI decks frequently cite impressive productivity and automation numbers.
    • Due diligence requires verifying whether the numbers support the conclusions drawn.
  • Examples:
    • A controlled experiment showing 55.8% faster development on a single JavaScript HTTP server.
  • Terminology:
    • technical due diligence
    • controlled experiment
    • AI adoption
  • Why it matters: Helps distinguish between genuine productivity gains and marketing hype.

Check 1: The Arithmetic

  • Explanation: Examining whether increased AI output translates to reduced head count or simply increased downstream review workload.
  • Key claims:
    • More work entering the system often increases the downstream review queue burden.
    • Observational telemetry of 10,000+ developers showed 98% more merged PRs but 91% longer PR review time and 154% larger PR size.
  • Examples:
    • Faros AI study showing increased PR volume leading to a bottleneck in human review labor.
  • Terminology:
    • observational telemetry
    • PR review time
    • human labor tracking
  • Why it matters: Proves that AI acceleration often shifts work rather than eliminating it.

Check 2: The Definition

  • Explanation: Dissecting what 'automation' actually means by examining the spectrum from AI drafted to zero human intervention.
  • Key claims:
    • Automation claims sit on a spectrum from AI drafted with human in the loop to straight-through processing inside a named boundary.
    • AI-attributed and autonomous are distinct claims.
  • Examples:
    • Uber's claim that >70% of pull requests are AI-attributed.
  • Terminology:
    • straight-through processing
    • AI-attributed
    • human-in-the-loop
  • Why it matters: Prevents confusing assisted work with autonomous execution.

Check 3: The Tell

  • Explanation: Comparing automated evaluation results against human expert merge decisions to check evaluation maturity.
  • Key claims:
    • Automated graders are often substantially more optimistic than human maintainers.
    • Evaluation maturity is the first credibility signal to look for.
  • Examples:
    • METR study showing a 24.2 percentage-point gap between automated grader scores and maintainer merge decisions.
  • Terminology:
    • automated grader
    • maintainer merge decision
    • ground truth
  • Why it matters: Uncovers discrepancies between model benchmarks and expert judgment.

The P&L Bridge and Financial Translation

  • Explanation: Translating technical metrics into unit economics by calculating the cost per accepted autonomous task.
  • Key claims:
    • A warning or due diligence flag is not enough unless it translates into financial unit economics.
    • Failed, escalated, or reworked tasks belong in the cost bucket.
  • Examples:
    • Engineer.ai (Builder.ai) public warning by WSJ in 2019, revenue estimate cuts in 2025, and Chapter 7 filing.
  • Terminology:
    • P&L bridge
    • cost per accepted autonomous task
    • straight-through success
  • Why it matters: Ensures AI investments survive both technical validation and financial scrutiny.

Key Points

Volume vs. Labor Arithmetic

  • Explanation: High AI adoption increases PR volume, review time, and PR size, which expands review labor requirements rather than reducing headcount.
  • Evidence: Telemetry from 10,000+ developers showing +98% merged PRs, +91% review time, and +154% PR size.
  • Practical implication: Ask for 18 months of workload volume compared against fully loaded human labor.

Defining Automation Boundaries

  • Explanation: Companies conflate AI assistance with autonomous execution. True straight-through processing requires a defined boundary with zero human intervention.
  • Evidence: Uber's >70% AI-attributed PR metric vs. distinct workflow stages (drafted, edited, approved, no human intervention).
  • Practical implication: Clarify where on the spectrum the automation claim sits.

Evaluation Maturity Gap

  • Explanation: Automated evaluation benchmarks often vastly overstate success compared to human domain experts.
  • Evidence: METR findings showing a 24.2 percentage-point gap between automated graders and human maintainers.
  • Practical implication: Always verify who labeled ground truth, whether experts agree, and what bad output escaped.

Frameworks, Models & Processes

The Dilgence Card (Three Checks)

  • How it works: Three rigorous checks applied to AI productivity and automation claims before accepting them in financial or technical evaluations.
  • Components:
    • Arithmetic: Volume vs. labor (18 months, count every human touch)
    • Definition: Which rung? (Assisted, approved, or straight-through)
    • The Tell: Who defines quality? (Ground truth, agreement, judge, escapes)
  • When to use: During technical due diligence, vendor evaluations, or investment reviews of AI companies.

The P&L Bridge

  • How it works: Formula to translate workflow value into captured economic value by factoring in adoption, success, quality, and rework costs.
  • Components:
    • Eligible workflow value
    • Adoption rate
    • Straight-through success
    • Accepted quality
    • Minus AI + review + rework cost
  • When to use: When projecting financial returns or unit economics on AI-driven operational efficiency.

Examples & Case Studies

Control experiment with 95 professional developers building a JavaScript HTTP server against 12 visible tests.

  • Illustrates: The origin of the '55.8% faster' claim often cited in marketing decks.
  • Lesson: Controlled lab tasks do not reflect messy, long-horizon production environments.

METR study comparing automated graders against maintainer merge decisions across 296 AI pull requests.

  • Illustrates: The 24.2 percentage-point gap between automated benchmark scores and human expert acceptance.
  • Lesson: Automated evaluation maturity must be tested against expert human review.

Builder.ai (Engineer.ai) faced WSJ questioning in 2019, revenue estimate cuts from $220M to $55M in May 2025, and filed for Chapter 7 bankruptcy in June 2025.

  • Illustrates: How failing to turn public warnings into strict diligence gates leads to insolvency driven by financial and operational realities beyond AI claims.
  • Lesson: A warning is not a diligence gate; even if a claim survives technical checks, it must survive unit economics.

Actionable Takeaways

  • Immediate:
    • Count every human touch in the workflow rather than just looking at raw code generation volume.
    • Determine whether automation claims refer to AI assistance or zero-intervention straight-through processing.
  • Strategic:
    • Use the P&L bridge framework to calculate cost per accepted autonomous task.
    • Evaluate AI maturity by checking automated test results against expert human review data.
  • Questions to investigate:
    • Who labeled the ground truth for evaluation datasets?
    • What percentage of AI-generated output escapes into production without human review?
    • How do downstream review bottlenecks scale with increased AI-generated workload?

Claims Worth Verifying

  • High AI adoption teams merge 98% more pull requests with a 91% increase in review time. (statistical observation)
  • Automated graders score higher than human maintainer merge decisions by a 24.2 percentage-point gap. (empirical research finding)

Notable Quotes

"A true number can answer the wrong question" "The warning existed. The process did not turn it into a gate." "The job of diligence is not to ask whether the number is impressive. It's to ask what must be true for the number to matter."

Compressed Summary

  • Technical due diligence must test arithmetic, definitions, and evaluation maturity of AI claims.
  • Increased AI code generation volume frequently creates downstream human review bottlenecks.
  • Financial translation via P&L bridges connects workflow adoption to unit economics like cost per accepted autonomous task.
  • Keywords: due diligence, ai automation, productivity metrics, unit economics, evaluation maturity
  • Core insight: Evaluating AI productivity claims requires looking past raw generation volume to measure downstream review labor, strict automation definitions, and true unit economic impact.

Core insights

6
Empirical Resultmedium noveltymoderate evidence

Cheap AI generation shifts the bottleneck downstream: making code production cheaper increases the volume and size of work entering human review, so the constraint moves to reviewer labor rather than disappearing. The telemetry cited shows output volume rising far faster than review capacity, meaning net labor savings must be measured at the review stage, not the generation stage.

Why it matters

Any system that generates artifacts faster than humans can verify them will saturate on verification. Architecture and capacity planning should size the review/verification stage first and treat generation throughput as an input to that queue, not the headline metric.

Generalization

In any generate-then-verify pipeline (code, documents, tickets, data fixes), optimizing the generator relocates the bottleneck to the verifier; system throughput is governed by verification capacity and artifact size.

Observational telemetry of 10,000+ developers showed 98% more merged PRs but 91% longer PR review time and 154% larger PR size.
Open source video
More work entering the system often increases the downstream review queue burden.
Open source video
High AI adoption increases PR volume, review time, and PR size, which expands review labor requirements rather than reducing headcount.
Open source video
Mental Modelhigh noveltymoderate evidence

'Automation' is not binary but a ladder: AI-drafted-with-human-in-the-loop at one end and straight-through processing inside a named boundary at the other. 'AI-attributed' output and 'autonomous' execution are categorically different claims, so a single percentage conflates rungs of the ladder.

Why it matters

Agentic system design needs an explicit, named boundary that defines where human intervention is required and where execution is truly straight-through. Without that boundary, autonomy claims are unfalsifiable and the system's operating envelope is undefined.

Generalization

Autonomy should be expressed as a success rate inside a declared operating boundary (inputs, tools, failure handling), not as a percentage of work touched by AI.

Automation claims sit on a spectrum from AI drafted with human in the loop to straight-through processing inside a named boundary.
Open source video
AI-attributed and autonomous are distinct claims.
Open source video
Uber's claim that >70% of pull requests are AI-attributed.
Open source video
Empirical Resulthigh noveltymoderate evidence

Automated graders systematically overstate success relative to human expert acceptance decisions; the reported gap between grader scores and maintainer merge decisions was 24.2 percentage points. This implies benchmark/eval score is a biased proxy for the outcome the business actually consumes.

Why it matters

If your eval harness is the same model family or rubric that generated the output, its ceiling is inflated. Calibration against an independent expert acceptance decision, plus tracking of what bad output escaped, is required before trusting any eval-derived reliability number.

Generalization

Model-as-judge evaluation must be calibrated against independent expert ground truth; the judge's optimism is a measurable bias term, not noise.

Automated graders are often substantially more optimistic than human maintainers.
Open source video
METR study showing a 24.2 percentage-point gap between automated grader scores and maintainer merge decisions.
Open source video
Always verify who labeled ground truth, whether experts agree, and what bad output escaped.
Open source video
Mental Modelmedium noveltymoderate evidence

The correct denominator for AI unit economics is the accepted autonomous task, not the attempted or generated one. The P&L bridge multiplies eligible workflow value by adoption rate, straight-through success, and accepted quality, then subtracts AI cost plus review cost plus rework cost — placing failures, escalations, and rework explicitly in the cost bucket.

Why it matters

It converts a technical metric into a per-unit cost figure that can be compared against fully loaded human labor, and it prevents the common error of amortizing AI inference cost across generated tasks while ignoring the human review and rework that generation induces.

Generalization

Any automation ROI model should use cost-per-accepted-output as the unit, with rework and escalation as first-class cost terms rather than excluded externalities.

A warning or due diligence flag is not enough unless it translates into financial unit economics.
Open source video
Failed, escalated, or reworked tasks belong in the cost bucket.
Open source video
Minus AI + review + rework cost
Open source video
Failure Modemedium noveltymoderate evidence

Headline productivity numbers often originate in tightly scoped lab conditions — a single JavaScript HTTP server built by 95 professional developers against 12 visible tests — and do not transfer to long-horizon, messy production work. The scope of the original experiment is the hidden variable behind the headline figure.

Why it matters

When evaluating any agent benchmark or vendor claim, the scope of the task (visible tests, single artifact, short horizon) is the strongest predictor of whether the gain will reproduce in production. Reporting should include task scope and visibility of the acceptance criteria.

Generalization

Benchmark gains measured on short-horizon tasks with fully visible acceptance criteria systematically overstate gains on long-horizon tasks with hidden requirements.

A controlled experiment showing 55.8% faster development on a single JavaScript HTTP server.
Open source video
Control experiment with 95 professional developers building a JavaScript HTTP server against 12 visible tests.
Open source video
Controlled lab tasks do not reflect messy, long-horizon production environments.
Open source video
Practicemedium noveltyweak evidence

Evaluation maturity should be checked first, ahead of architecture or financial claims, because it determines whether any other number in the deck can be trusted. Diligence sequence: arithmetic (volume vs labor over ~18 months, counting every human touch), definition (which rung of autonomy), then the tell (who defines quality).

Why it matters

It gives an explicit ordering for technical vetting of an AI system, and inverts into a build principle: invest in eval maturity and human-touch instrumentation before claiming or scaling autonomy.

Generalization

Credibility of an AI system is gated on eval quality; instrumenting human touches over a long window is a precondition for any credible automation claim.

Evaluation maturity is the first credibility signal to look for.
Open source video
Ask for 18 months of workload volume compared against fully loaded human labor.
Open source video

Deep dives

4

The review-capacity crossover point for AI-generated throughput

Research question

At what ratio of generated-artifact volume to reviewer capacity does marginal AI generation become net-negative, and what artifact granularity cap (PR size, task scope) restores positive marginal value per accepted output?

Why

Aggregate telemetry (+98% merged PRs, +91% review time, +154% PR size) tells you the bottleneck moved but not where the system saturates. Without a crossover curve, teams cannot set generation rate limits, PR size caps, or agent task granularity on any basis other than intuition, and schedulers will keep producing backlog.

Observational telemetry of 10,000+ developers showed 98% more merged PRs but 91% longer PR review time and 154% larger PR size.
Open source video
More work entering the system often increases the downstream review queue burden.
Open source video
Source video

Grader optimism as a measurable function of judge/generator overlap

Research question

Does the automated-grader optimism bias (observed at 24.2 percentage points) scale with the degree of overlap between the judging model/rubric and the generating model, and does it hold across code, document, and data-fix domains?

Why

If optimism is structural rather than noise, every reliability number derived from an eval harness carries a bias term that must be reported and subtracted. Establishing whether the bias is a function of judge independence turns 'trust our eval score' into a calibratable offset, and tells you whether a cheap same-family judge is ever acceptable.

Automated graders are often substantially more optimistic than human maintainers.
Open source video
METR study showing a 24.2 percentage-point gap between automated grader scores and maintainer merge decisions.
Open source video
Always verify who labeled ground truth, whether experts agree, and what bad output escaped.
Open source video
Source video

Declaring the autonomy boundary as an architecture artifact

Research question

What is the minimum specification of a named operating boundary (allowed inputs, tools, failure paths, acceptance criteria) that makes a straight-through success rate falsifiable and comparable across two different agent systems?

Why

Without a declared boundary, 'percent automated' is unfalsifiable and blends assisted work with autonomous execution. A concrete boundary spec converts the claim into a per-boundary straight-through rate plus an explicit escalation path, which is also exactly what routing and escalation code needs to encode.

Automation claims sit on a spectrum from AI drafted with human in the loop to straight-through processing inside a named boundary.
Open source video
AI-attributed and autonomous are distinct claims.
Open source video
Uber's claim that >70% of pull requests are AI-attributed.
Open source video
Source video

Attributing review and rework cost per accepted autonomous task in a shared human queue

Research question

In a queue where reviewers handle both AI-generated and human-generated artifacts, what provenance instrumentation and attribution rule yields a defensible cost-per-accepted-autonomous-task figure?

Why

The P&L bridge names AI cost, review cost, and rework cost as separate terms, but in practice they are commingled in one human queue. Without an attribution method, the cost-per-accepted-output denominator silently absorbs or hides review and rework, and the comparison against fully loaded human labor is not honest.

A warning or due diligence flag is not enough unless it translates into financial unit economics.
Open source video
Failed, escalated, or reworked tasks belong in the cost bucket.
Open source video
Minus AI + review + rework cost
Open source video
Source video

Article ideas

4

Your 70% Automated Is a Ladder, Not a Number

A single automation percentage is an unfalsifiable metric because it conflates AI-attributed assistance with straight-through autonomous execution; the only defensible claim is a success rate inside a declared operating boundary, reported separately from attribution rate.

Angle

Take the highest-profile public automation percentage available and show that it is fully consistent with near-zero autonomy, then show what the boundary-based restatement would have to contain.

Source video

The Bottleneck Is the Verifier: Why Cheaper Generation Makes Review the Binding Constraint

Accelerating artifact generation does not reduce required labor; it converts labor into a larger, slower verification queue, so capacity planning must size the reviewer stage first and treat generation throughput as an input to that queue rather than the headline result.

Angle

Use the +98% PRs / +91% review time / +154% PR size telemetry to argue that every generate-then-verify pipeline (code, docs, tickets, data fixes) has the same shape, and derive concrete design consequences: artifact size caps, task granularity, and reviewer-throughput metrics.

Source video

When the Judge Is the Author: The 24-Point Optimism Tax on AI Evaluations

An eval harness built from the same model family or rubric that produced the output reports a structurally inflated reliability number; judge scores are only usable once calibrated against independent expert acceptance decisions and paired with an analysis of what bad output escaped.

Angle

Frame the 24.2-point grader-versus-maintainer gap as a measurable bias term rather than an anecdote, then specify the calibration layer: expert-labeled ground truth, inter-expert agreement, and escape tracking.

Source video

Warnings Don't Stop Bankruptcies: Translating Diligence Flags into Unit Economics

A technical due-diligence flag has no decision value until it is expressed as a per-unit cost comparison against fully loaded human labor, with failed, escalated, and reworked tasks loaded into the cost bucket.

Angle

Argue that the failure mode is not missing signals but unmonetized ones, and present the P&L bridge structure as the mechanism that turns a soft warning into a hard gate.

Source video

Project ideas

3

Verifier Saturation Harness

movement-lab

For a fixed reviewer pool, there exists an artifact-size cap below which marginal AI generation yields positive accepted-output-per-reviewer-hour, and above which it is net-negative; a moderate cap on artifact size raises accepted throughput even while total generation drops.

Proof of concept

A synthetic generate-then-verify pipeline where an LLM produces code/doc artifacts at configurable volume and size, a simulated reviewer with finite per-hour capacity and size-dependent review latency accepts or rejects them, and generation volume and max artifact size are swept across a grid.

Measurement

Accepted outputs per reviewer-hour and cost per accepted output as a function of (generation rate, max artifact size); locate the crossover point and the size cap that maximizes accepted throughput.

Source video

Judge-Optimism Meter

beyond-evals

Automated grader optimism bias is monotonically increasing in judge/generator overlap and is stable enough across domains to be reported as a calibrated offset rather than treated as noise.

Proof of concept

Score a shared artifact set with three graders of decreasing independence (same model family, different family, human-anchored rubric) and compare each against a multi-expert acceptance label with measured inter-annotator agreement; track escaped defects the graders passed.

Measurement

Percentage-point gap between grader score and expert acceptance, per grader condition and per domain; correlation between gap size and judge/generator overlap; escaped-defect rate.

Source video

Autonomy Boundary Ledger

gatehouse

Systems that declare an explicit operating boundary and report straight-through success rate inside it produce autonomy claims that are reproducible and comparable, while systems reporting a single blended automation percentage cannot be replicated by an independent evaluator.

Proof of concept

A spec-and-instrumentation harness: a declarative boundary file (allowed inputs, tools, failure paths, acceptance criteria) plus a runtime ledger that tags every task as straight-through, escalated, reworked, or rejected, emitting a per-boundary success rate alongside the legacy blended percentage.

Measurement

Inter-evaluator reproducibility of the autonomy number (do two independent evaluators get the same straight-through rate from the ledger?) versus variance of the blended percentage; share of tasks whose disposition is unclassifiable without a boundary.

Source video

Architectural implications

4

AI acceleration moves the constraint from generation to verification.

Before

Instrument and optimize model throughput: tokens, generations, PRs created, tasks attempted.

After

Instrument the review/verification stage as the primary capacity metric: review latency, artifact size, human touches per accepted output, queue depth.

Consequence

Schedulers, batching, and agent scope decisions should target reviewer throughput — e.g., capping PR size or task granularity — or the system will generate backlog instead of savings.

Source video

Autonomy claims need a declared operating boundary.

Before

Report a single blended 'percent automated' figure across all workflow stages.

After

Declare a named boundary (allowed inputs, tools, failure paths) and report straight-through success rate inside that boundary separately from assisted/AI-attributed rates.

Consequence

Routing and escalation logic becomes an explicit architectural artifact, and autonomy becomes measurable and comparable across systems.

Source video

Automated graders are biased optimistic proxies for expert acceptance.

Before

Use an LLM-as-judge or rubric score as the reliability metric for the agent.

After

Maintain a calibration set of expert ground-truth decisions with inter-expert agreement, and track escaped defects alongside judge scores.

Consequence

Eval harness must include a human-anchored calibration layer and an escape-analysis loop; judge scores reported with a measured optimism offset.

Source video

Rework and escalation are hidden costs of automation.

Before

Compute cost per generated task or per AI-touched task.

After

Compute cost per accepted autonomous task, loading AI inference + human review + rework + escalation into the numerator.

Consequence

Changes which workflows are worth automating: high straight-through-rate, low-rework tasks get prioritized; attractive-looking high-volume tasks with expensive review may be net negative.

Source video

Tradeoffs and failure modes

5

Generation throughput vs review labor

Benefit

98% more merged pull requests — much higher artifact output per unit of engineering time.

Cost or risk

91% longer review time and 154% larger PRs; review labor requirement expands, offsetting or reversing headcount savings.

Observational telemetry of 10,000+ developers showed 98% more merged PRs but 91% longer PR review time and 154% larger PR size.
Open source video
Source video

AI-attributed metric as autonomy proxy

Benefit

A single high percentage (>70%) is simple and flattering, and is genuinely produced by widespread AI usage.

Cost or risk

Conflates assisted work with autonomous execution; the claim cannot distinguish a human-approved draft from straight-through processing.

AI-attributed and autonomous are distinct claims.
Open source video
Source video

Automated evaluation at scale

Benefit

Cheap, fast, repeatable grading of agent output without human bottleneck.

Cost or risk

Substantially optimistic vs maintainer merge decisions (24.2pp gap), producing false confidence and unmeasured escaped defects.

METR study showing a 24.2 percentage-point gap between automated grader scores and maintainer merge decisions.
Open source video
Source video

Lab-measured productivity gains

Benefit

A clean, controlled 55.8% speedup on a well-scoped task — reproducible in that setting.

Cost or risk

Does not transfer to messy, long-horizon production environments; the number travels into decks stripped of its scope.

Controlled lab tasks do not reflect messy, long-horizon production environments.
Open source video
Source video

Flags without financial translation

Benefit

A due-diligence warning is cheap to issue and technically defensible.

Cost or risk

Warnings that never become hard gates or unit-economics constraints fail to prevent insolvency (Builder.ai trajectory).

How failing to turn public warnings into strict diligence gates leads to insolvency driven by fi
Open source video
Source video

Open questions

5

At what review-capacity ratio does AI-generated throughput become net-negative, and what artifact granularity (PR size cap, task scope) restores positive marginal value?

Why unresolved

The summary reports aggregate telemetry (+98% PRs, +91% review time, +154% PR size) but no crossover point or granularity experiment.

Research direction

Simulate or A/B test generation volume and artifact size against reviewer throughput; measure cost per accepted output as a function of max artifact size.

Source video

How large is the automated-grader optimism bias across domains, and does it scale with the judge's overlap with the generator?

Why unresolved

The summary reports a single 24.2pp gap from one study on AI pull requests; no cross-domain or judge-independence comparison exists.

Research direction

Run paired evaluations where automated graders and independent expert maintainers score the same outputs across code, docs, and data tasks; report bias as a function of judge/generator overlap.

Source video

What labeling protocol makes expert ground truth reliable enough to serve as calibration — how many experts per item, and what inter-annotator agreement threshold?

Why unresolved

The summary raises 'whether experts agree' as a check but gives no protocol or threshold.

Research direction

Build calibration sets with multi-expert labels, measure agreement, and test whether grader-judge discrepancy predicts disagreement among experts.

Source video

Does the rise in PR size and review time causally follow from AI adoption, or is it confounded by team policy, headcount change, or reviewer mix over the observation window?

Why unresolved

The evidence is explicitly observational telemetry from 10,000+ developers, without a control arm.

Research direction

Instrument a controlled cohort: staged AI rollout with a matched non-adopting team, tracking PR size, review time, and human touches per accepted change.

Source video

How should review and rework costs be attributed per accepted autonomous task in a shared human queue, where reviewers handle both AI and human output?

Why unresolved

The P&L bridge names the cost terms but the summary gives no attribution method.

Research direction

Instrument human touches with task-level provenance and build a cost-attribution model comparing fully loaded labor cost before and after automation.

Source video

Key claims

8
factualVerification needed

Across telemetry from 10,000+ developers, AI adoption correlated with 98% more merged PRs, 91% longer PR review time, and 154% larger PR size.

Evidence

Observational telemetry of 10,000+ developers showed 98% more merged PRs but 91% longer PR review time and 154% larger PR size.

Question

Is this causal, and what is the definition of 'review time' and 'PR size' across the sampled repositories?

Source video
factualVerification needed

Automated grader scores exceeded maintainer merge decisions by 24.2 percentage points in a study of 296 AI pull requests.

Evidence

METR study showing a 24.2 percentage-point gap between automated grader scores and maintainer merge decisions.

Question

What grader and which maintainers were used, and does the gap hold outside the sampled task domain?

Source video
factualVerification needed

Uber reported that more than 70% of its pull requests are AI-attributed.

Evidence

Uber's claim that >70% of pull requests are AI-attributed.

Question

What does 'AI-attributed' mean operationally at Uber — generated, edited, or approved by AI?

Source video
factualVerification needed

A widely cited 55.8% faster result came from a controlled experiment where 95 professional developers built a JavaScript HTTP server against 12 visible tests.

Evidence

Control experiment with 95 professional developers building a JavaScript HTTP server against 12 visible tests.

Question

What was the task horizon and how much of the speedup survives when acceptance criteria are hidden?

Source video
causalVerification needed

Higher AI output volume increases downstream human review labor requirements rather than reducing headcount.

Evidence

More work entering the system often increases the downstream review queue burden.

Question

Under what generation-quality regimes does the review burden curve flatten or invert?

Source video
factualVerification needed

Builder.ai/Engineer.ai went from 2019 WSJ questioning to a revenue estimate cut from $220M to $55M in May 2025 and Chapter 7 filing in June 2025.

Evidence

Builder.ai (Engineer.ai) faced WSJ questioning in 2019, revenue estimate cuts from $220M to $55M in May 2025, and filed for Chapter 7 bankruptcy in June 2025.

Question

Which specific technical and financial gaps (automation claims vs delivery) drove the collapse?

Source video
opinionVerification not requested

A due-diligence flag has no value unless it is translated into financial unit economics.

Evidence

A warning or due diligence flag is not enough unless it translates into financial unit economics.

Source video
opinionVerification not requested

Evaluation maturity is the strongest early signal of whether other AI claims are credible.

Evidence

Evaluation maturity is the first credibility signal to look for.

Source video

Connections

5