AI Engineer · Published 2026-09-30

The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX

privateReview: unreviewedConfidence: high
developer experienceproductivity metricsAI agentssdlcdora metricsagent-orchestrationsdlc-automationdeveloper-productivitydora-metricschange-failure-rateai-readinessmeasurement-framework
Open on YouTube ↗

Summary

Overview

  • Speaker: Justin Reock
  • Channel: AI Engineer
  • Main topic: Impact of AI on developer experience, productivity, delivery velocity, quality, and measurement across 400+ organizations
  • Purpose: To provide empirical data and research-backed insights into how AI adoption actually impacts developer productivity, velocity, code quality, and organizational workflows. Justin Reock, Deputy CTO at DX, shares research analyzing data from over 400 organizations and 200,000 engineers on AI's impact on developer experience and productivity. Contrary to expectations of 5x or 10x productivity gains, data shows modest median PR throughput increases (~7.76% median, 13.1% average). The presentation explores five key trends: velocity impact, quality and delivery impact (showing increased change failure rates and tension between maintainability and change confidence), demographic differences (junior engineers using AI the most, staff+ saving the most time), measurement frameworks (utilization, impact, cost), and SDLC/PDLC agent integrations illustrated through case studies from Morgan Stanley, Zapier, Faire, and Spotify.

Topic Map

Introduction and DX Platform Research

  • Explanation: Justin Reock introduces DX as a research-backed platform built on DORA, SPACE, and DevEx frameworks, focusing on gathering data regarding AI's impact on developer productivity and experience.
  • Key claims:
    • DX is a research-backed platform built from the same research team behind DORA metrics, SPACE framework, and DevEx framework.
    • The presentation focuses on organizational trends and workflow changes based on data from 400+ companies.
  • Examples:
    • Quarterly state of AI and AI impact reports looking at raw data from DX platform.
  • Terminology:
    • DX
    • DORA
    • SPACE
    • DevEx
  • Why it matters: Establishes credibility and defines the data-driven foundation of the presentation.

Trend #1: Velocity Impact

  • Explanation: Examines raw velocity metrics such as deployment frequency and PR throughput across organizations, showing steady increases in deployment frequency.
  • Key claims:
    • Deployment frequency is steadily increasing across organizations.
    • Regional trends vary (North America trending up, Europe pulling back slightly in recent quarters due to regulations, spending, and work methods).
  • Examples:
    • Deployment frequency charts across Q3 2023 to Q2 2024.
    • Perceived rate of delivery showing a minor ~4.5% increase over a full year.
  • Terminology:
    • Deployment frequency
    • PR throughput
    • Velocity metrics
  • Why it matters: Demonstrates that while deployment frequency increases, engineers' perceived rate of delivery has not scaled proportionally to AI investments.

Trend #2: Quality and Delivery Impact

  • Explanation: Explores how AI adoption impacts change failure rates, code maintainability, change confidence, and PR size.
  • Key claims:
    • Change failure rate impact is highly volatile; top performers see increases up to 2% (when industry benchmark is 4%, representing 50% more defects).
    • Code maintainability perceived score went up ~4%, but change confidence went down ~6%.
    • Median PR size has noticeably increased (from ~44 to 72 lines of code) over the study period.
  • Examples:
    • Per-company change failure rate impact chart.
    • Code maintainability vs change confidence tension chart.
  • Terminology:
    • Change failure rate
    • Code maintainability
    • Change confidence
    • PR size
  • Why it matters: Highlights a critical psychological and quality tension: developers find code easier to modify with AI, but trust the output less.

Trend #3: Demographic Differences

  • Explanation: Analyzes how AI usage and time savings vary by developer seniority and organizational size.
  • Key claims:
    • Junior engineers use AI the most due to less unlearning required.
    • Staff+ and senior engineers save roughly the same amount of time as juniors while burning fewer tokens.
    • Smaller companies lead in AI time savings due to less organizational complexity and simpler release pipelines.
  • Examples:
    • AI adoption by usage level and role seniority.
    • Weekly AI time savings by seniority level.
  • Terminology:
    • Role seniority
    • Junior engineers
    • Staff+ engineers
    • Engineering org size
  • Why it matters: Shows that adoption patterns and ROI differ significantly across organizational tiers and experience levels.

Trend #4: Measurement Framework

  • Explanation: Presents an AI measurement framework covering utilization, impact, and cost, along with AI readiness factors.
  • Key claims:
    • Measuring developer productivity and DevEx was challenging before AI and is even more compounded now.
    • Foundational DevEx and DevProd metrics still matter the most; AI metrics measure what is happening, while core metrics measure whether it is working.
    • AI readiness depends on clear documentation, data structures, manageable code, fast local CI, and non-flaky test suites.
  • Examples:
    • AI Measurement Framework table (Utilization, Impact, Cost).
    • Gauging AI Readiness checklist (Bronze, Silver, Gold tiers).
  • Terminology:
    • Utilization
    • Impact
    • Cost
    • AI readiness
    • API telemetry
  • Why it matters: Provides organizations with a structured methodology to evaluate AI investments rather than relying on hype.

Trend #5: Integration across SDLC/PDLC and Case Studies

  • Explanation: Discusses integrating AI across the entire software development lifecycle (SDLC) and product development lifecycle (PDLC), highlighting real-world success stories.
  • Key claims:
    • Code generation is not the bottleneck; code gen represents only 14-16% of the overall value stream.
    • Productivity gains are modest: median PR throughput increased by 7.76% and average by 13.1% in longitudinal studies.
    • Non-AI factors (meetings, context switching, release pipeline friction) still outweigh AI benefits.
  • Examples:
    • Morgan Stanley's DevGen AI agent for legacy code refactoring (saving 280,000+ developer hours annually).
    • Zapier's administrative automation agent ecosystem (reducing stand-ups, onboarding in ~2 weeks).
    • Faire's 'Fairey' automated code review agent (handling 3,000 code reviews/week).
    • Spotify's incident management agent platform handling 90% of incidents.
  • Terminology:
    • SDLC
    • PDLC
    • Bottleneck
    • Theory of Constraints
    • DevGen
    • Fairey
  • Why it matters: Proves that targeting non-coding bottlenecks in the SDLC with specialized AI agents yields massive real-world ROI.

Key Points

Productivity gains are more modest than expected

  • Explanation: Longitudinal studies from Nov 2024 to Feb 2026 across 400+ companies show a median PR throughput increase of 7.76% and an average of 13.1%, with no companies hitting 2x, 5x, or 10x.
  • Evidence: Weighted PR throughput increase table (Median 7.76%, Average 13.1%).
  • Practical implication: Executive leadership expecting 10x returns from token spend need to look beyond code generation and address broader workflow bottlenecks.

Code maintainability and change confidence are in tension

  • Explanation: Code maintainability scores increased by 3.8%, but change confidence dropped by 6.1% as developers trust AI-generated code less.
  • Evidence: Bar chart comparing Code Maintainability (+3.8%) and Change Confidence (-6.1%).
  • Practical implication: Organizations must invest in automated testing and review guardrails to maintain engineering confidence when adopting AI.

Foundational DevEx metrics remain paramount

  • Explanation: AI metrics measure what is happening, but core foundational metrics measure whether it is actually working.
  • Evidence: Pyramid diagram showing Core Metrics (measures whether it is working) supporting AI Metrics (measures what is happening).
  • Practical implication: Do not throw away existing DORA and DevEx metrics when adopting AI tools; correlate AI telemetry with foundational metrics.

Find and fix the actual bottleneck

  • Explanation: Applying Eli Goldratt's Theory of Constraints, saving time on tasks that are not the system bottleneck is worthless.
  • Evidence: Manufacturing process diagram showing bottleneck identification and case studies from Morgan Stanley, Zapier, Faire, and Spotify.
  • Practical implication: Target AI agents at administrative tasks, legacy refactoring, code reviews, and incident management rather than just raw code authoring.

Frameworks, Models & Processes

AI Measurement Framework

  • How it works:
  • Components:
    • Utilization (AI tool usage, DAU/WAU, percentage of PRs AI-assisted, percentage of commit code, tasks assigned to agents)
    • Impact (AI-driven time savings, developer satisfaction, DX Core 4 metrics, perceived rate of delivery, change confidence, change failure rate, human-equivalent hours)
    • Cost (AI spend total and per developer, net time gain per developer, agent hourly rate)
  • When to use: When evaluating the return on investment and organizational effectiveness of enterprise AI tooling deployments.

Gauging AI Readiness (DevEx Foundation)

  • How it works: Assesses platform and codebase readiness across Bronze, Silver, and Gold tiers to ensure infrastructure can support AI agents effectively.
  • Components:
    • Clear, accurate, well-structured documentation
    • Data structures with straightforward relations
    • Readable, manageable, modular code
    • Fast, reliable local and CI feedback loops
    • Stable (read: non-flaky) robust test suites
  • When to use: Before rolling out AI agents and coding assistants across an engineering organization.

Examples & Case Studies

Morgan Stanley built 'DevGen.AI', an internal AI agent that reads legacy code (mainframe natural, cobol, perl), gathers context, and creates PRDs for engineers.

  • Illustrates: Automating legacy code refactoring to eliminate reverse engineering.
  • Lesson: Saves the company over 280,000 developer hours annually by targeting a major non-coding bottleneck.

Zapier built an internal automation ecosystem where agents handle daily stand-ups, onboarding steps, internal reporting, and Slack communications.

  • Illustrates: Administrative automation using AI agents across the PDLC.
  • Lesson: Reduced stand-ups from 5x to 2x per week, accelerated onboarding to ~2 weeks, and increased hiring capacity.

Faire developed 'Fairey', an AI agent that automatically reviews PRs for superficial issues and posts comments.

  • Illustrates: Automated code review and quality guardrails.
  • Lesson: Completes roughly 3,000 code reviews per week, freeing senior engineers from routine checks to focus on complex architecture.

Spotify created an incident management agent platform that monitors logs, correlates alerts in real-time, and suggests remediation from runbooks.

  • Illustrates: Incident management and SRE workflow acceleration.
  • Lesson: Handles 90% of incidents at Spotify, dramatically reducing MTTR and freeing SREs for proactive reliability work.

Actionable Takeaways

  • Immediate:
    • Review current AI tool utilization alongside core DORA and DevEx metrics.
    • Assess your organization's AI readiness across documentation, code modularity, and test suite stability.
    • Identify the actual bottlenecks in your development workflow before deploying AI solutions.
  • Strategic:
    • Expect modest productivity gains (7-13% PR throughput increase) rather than magical 10x leaps from code gen alone.
    • Address organizational friction and non-AI factors (meetings, context switching) that eclipse AI time savings.
    • Expand AI adoption beyond coding assistants into SDLC/PDLC agents for code review, legacy refactoring, and incident management.
  • Questions to investigate:
    • Why is change confidence dropping despite improvements in perceived code maintainability?
    • How does our organization's PR size and deployment frequency trend compared to industry benchmarks?
    • Are we targeting AI investments at the actual bottlenecks in our delivery pipeline?

Claims Worth Verifying

  • Data sample includes over 400+ companies and 200,000 engineers (statistical)
  • Median PR throughput increase in longitudinal study is 7.76% and average is 13.1% (statistical)
  • Morgan Stanley's DevGen saves over 280,000 developer hours annually (case study metric)
  • Spotify's incident management agent handles 90% of incidents (case study metric)

Notable Quotes

"We spent $10M on tokens, where is the 10x productivity?" (at 18:24) "An hour saved on something that isn't the bottleneck is worthless" (at 26:49)

Compressed Summary

  • AI adoption across 400+ orgs shows modest median PR throughput gains of 7.76%.
  • Code maintainability scores rose 3.8%, but change confidence dropped 6.1%.
  • Junior engineers use AI most, while Staff+ and senior engineers save comparable time with fewer tokens.
  • Foundational DevEx readiness (documentation, modular code, CI/test stability) is critical for AI success.
  • Targeting SDLC bottlenecks (legacy refactoring, code reviews, incident management) yields massive organizational ROI.
  • Keywords: developer experience, productivity metrics, ai agents, sdlc, dora metrics
  • Core insight: Realizing transformative ROI from AI in software development requires targeting SDLC bottlenecks and ensuring platform AI readiness rather than relying on code generation alone.

Core insights

6
Mental Modelhigh noveltymoderate evidence

Code generation is a small fraction of the value stream (~14-16%), so optimizing it barely moves end-to-end throughput. The largest real-world ROI comes from pointing specialized agents at non-coding bottlenecks (legacy refactoring, code review, incident management, admin workflows), not at the IDE.

Why it matters

It reframes where to invest agent engineering effort: budget and architecture should follow the bottleneck (Theory of Constraints), not the most visible coding task. Teams building coding copilots may be optimizing a step that is not limiting delivery.

Generalization

For any agentic pipeline, measure the value-stream share of the step before automating it; automation of a non-bottleneck step yields sublinear system-level gains.

Code generation is not the bottleneck; code gen represents only 14-16% of the overall value stream.
Open source video
Non-AI factors (meetings, context switching, release pipeline friction) still outweigh AI benefits.
Open source video
Proves that targeting non-coding bottlenecks in the SDLC with specialized AI agents yields massive real-world ROI.
Open source video
Empirical Resultmedium noveltystrong evidence

Longitudinal measurement across 400+ orgs shows AI's measured delivery gain is modest and sublinear versus expectations: median PR throughput +7.76%, average +13.1%, with no org reaching even 2x. The gap between token spend growth and throughput growth is the central empirical result.

Why it matters

It gives a realistic prior for ROI modeling and eval design: a coding-agent deployment should be expected to move throughput by low double-digit percentages, not multiples. Any benchmark claiming 5-10x should be scrutinized for what it measured.

Generalization

Aggregate org-level productivity metrics regress heavily toward small effect sizes even when individual-task evaluations show large speedups.

median PR throughput increased by 7.76% and average by 13.1% in longitudinal studies
Open source video
no companies hitting 2x, 5x, or 10x
Open source video
Perceived rate of delivery showing a minor ~4.5% increase over a full year.
Open source video
Tradeoffhigh noveltymoderate evidence

AI adoption decouples two quality signals that normally move together: perceived code maintainability rose (~+4%) while developer change confidence fell (~-6%). Developers find AI-generated code easier to modify but trust it less, and median PR size grew from ~44 to ~72 LOC.

Why it matters

It identifies a concrete failure mode for agentic coding systems: larger AI-assisted changes plus reduced reviewer trust is a recipe for under-reviewed merges. Also suggests 'maintainability' scores alone are an insufficient quality metric for AI-generated code.

Generalization

Readability/maintainability proxies can improve while trust and verifiability degrade; evaluation harnesses need separate instruments for generation ease and reviewer confidence.

Code maintainability perceived score went up ~4%, but change confidence went down ~6%.
Open source video
Median PR size has noticeably increased (from ~44 to 72 lines of code) over the study period.
Open source video
developers find code easier to modify with AI, but trust the output less
Open source video
Empirical Resultmedium noveltymoderate evidence

Seniority changes token efficiency rather than time savings: staff+/senior engineers save roughly the same time as juniors while burning fewer tokens, and juniors adopt AI the most because they have less unlearning to do.

Why it matters

Token cost per unit of delivered value is strongly skill-dependent, so cost-per-engineer dashboards conflate adoption intensity with efficiency. It also argues against naive token-based ROI attribution.

Generalization

In agentic systems broadly, the same output quality can be reached with far fewer tokens by a more capable operator/policy; token spend is a poor proxy for value delivered.

Staff+ and senior engineers save roughly the same amount of time as juniors while burning fewer tokens.
Open source video
Junior engineers use AI the most due to less unlearning required.
Open source video
Practicemedium noveltymoderate evidence

AI readiness is a prerequisite set of environment properties, not a model property: clear documentation, clean data structures, manageable code, fast local CI, and non-flaky test suites determine whether agents can work at all. The framework separates AI metrics (utilization/impact/cost — what is happening) from foundational DevEx/DevProd metrics (whether it is working).

Why it matters

It makes test infrastructure and CI latency first-class dependencies of agent effectiveness. Flaky tests and slow local CI directly degrade agent reliability and should be treated as blockers to agent adoption, not hygiene tasks.

Generalization

Agent performance is bounded by the verifiability and turnaround time of its feedback loop; invest in fast, deterministic harnesses before scaling autonomous agents.

AI readiness depends on clear documentation, data structures, manageable code, fast local CI, and non-flaky test suites.
Open source video
AI metrics measure what is happening, while core metrics measure whether it is working.
Open source video
Foundational DevEx and DevProd metrics still matter the most
Open source video
Failure Modemedium noveltyweak evidence

Change failure rate impact is highly volatile per organization and can offset velocity gains: top performers saw increases up to 2 points against a 4% industry benchmark, i.e. roughly 50% more defects.

Why it matters

Suggests AI-assisted delivery shifts the velocity/quality frontier in noisy, org-dependent ways; deployment-frequency gains may be partially or wholly paid for in rework. Monitoring only throughput hides this.

Generalization

Automation that accelerates change production without proportionate change-verification capacity tends to surface as increased change failure rate.

Change failure rate impact is highly volatile; top performers see increases up to 2% (when industry benchmark is 4%, representing 50% more defects).
Open source video

Deep dives

5

Value-stream attribution as the ranking function for agent investment

Research question

How can per-repo and per-org value-stream share be measured stage-by-stage (cycle-time contribution, wait states, handoffs) so that specialized SDLC agents can be ranked and deployed by system-level ROI rather than by visibility?

Why

If code generation is 14-16% of the value stream, then a portfolio decision made on IDE-adjacent intuition systematically over-funds the wrong stage. A reproducible attribution method is the prerequisite for any Theory-of-Constraints-style agent roadmap, and it converts 'where should we build agents?' from an opinion into a measurement.

Code generation is not the bottleneck; code gen represents only 14-16% of the overall value stream.
Open source video
Non-AI factors (meetings, context switching, release pipeline friction) still outweigh AI benefits.
Open source video
Proves that targeting non-coding bottlenecks in the SDLC with specialized AI agents yields massive real-world ROI.
Open source video
Source video

The decay of change confidence under AI-assisted delivery

Research question

Is the drop in developer change confidence a fixed property of AI-generated changes, or a function of review capacity being exceeded by larger AI-assisted PRs — and can a review-policy intervention restore confidence without giving back throughput?

Why

Maintainability rising while confidence falls means the standard 'code quality' proxy is actively misleading for AI-assisted code. Combined with median PR size growth, this points at review capacity, not generation capacity, as the binding constraint — which changes what a review agent should be built to do.

Code maintainability perceived score went up ~4%, but change confidence went down ~6%.
Open source video
Median PR size has noticeably increased (from ~44 to 72 lines of code) over the study period.
Open source video
developers find code easier to modify with AI, but trust the output less
Open source video
Source video

Causal separation of AI gains from confounders in delivery metrics

Research question

What fraction of the observed median PR throughput increase (+7.76%) survives matched-cohort or interrupted time-series comparison against non-adopting teams inside the same organization, once deployment-frequency trends and regional shifts are controlled?

Why

Without causal separation, every downstream ROI model and vendor benchmark inherits an unidentified confounder. This determines whether the low-double-digit effect is the true AI prior or an upper bound that further controls will shrink.

median PR throughput increased by 7.76% and average by 13.1% in longitudinal studies
Open source video
no companies hitting 2x, 5x, or 10x
Open source video
Perceived rate of delivery showing a minor ~4.5% increase over a full year.
Open source video
Source video

Feedback-loop determinism as the binding constraint on agent autonomy

Research question

Do organizations meeting explicit thresholds for test determinism and local CI latency show measurably higher AI impact scores than otherwise comparable organizations, and does agent task success rate degrade predictably as local CI latency grows?

Why

Treating AI readiness as an infrastructure property makes flaky tests and slow local CI release-blocking dependencies of agent effectiveness rather than hygiene items. If the correlation holds, readiness tiers become a legitimate gate on agent rollout instead of a slide in a deck.

AI readiness depends on clear documentation, data structures, manageable code, fast local CI, and non-flaky test suites.
Open source video
AI metrics measure what is happening, while core metrics measure whether it is working.
Open source video
Foundational DevEx and DevProd metrics still matter the most
Open source video
Source video

Token efficiency as a skill-conditional signal rather than a cost signal

Research question

Which context-selection, task-decomposition, or tool-use behaviors let senior and staff+ engineers achieve comparable time savings at lower token cost, and are those behaviors teachable to juniors or purely experience artifacts?

Why

If token spend is a poor proxy for value delivered, then cost-per-engineer dashboards are actively misleading and token-based ROI attribution will misrank teams and tools. Identifying the mechanism turns a cost number into a leverage point.

Staff+ and senior engineers save roughly the same amount of time as juniors while burning fewer tokens.
Open source video
Junior engineers use AI the most due to less unlearning required.
Open source video
Source video

Article ideas

4

Your Coding Copilot Is Optimizing 14% of the Value Stream

Coding assistance sits on a small share of the delivery value stream, so agent investment should be ranked by cycle-time contribution per stage; the visible artifact (code) is not the limiting constraint, and portfolios built around the IDE will keep producing sublinear system-level returns.

Angle

Apply Theory of Constraints to SDLC agent portfolios: instrument the value stream, rank stages, then fund agents in ranked order — with a counterexample set from review, incident, and legacy refactoring work where non-coding agents produced large documented ROI.

Source video

Easier to Modify, Harder to Trust: The Two Quality Signals AI Decoupled

Perceived maintainability and reviewer change confidence have decoupled under AI-assisted delivery, and because PR size grew alongside them, any evaluation harness that reports maintainability alone will pass code that reviewers will not stand behind; review capacity, not generation capacity, is the constraint to engineer against.

Angle

Argue that quality proxies must be split into separate instruments for generation ease and reviewer verifiability, and show what a size-aware AI review policy would have to enforce.

Source video

The 7.76% Reality Check: Why Task-Level Speedups Do Not Survive Aggregation

Longitudinal org-level data showing median PR throughput up 7.76% with no organization reaching 2x is the correct prior for coding-agent ROI; individual-task benchmarks are not projections of organizational throughput, and any 5x or 10x claim is making a claim about measurement, not about delivery.

Angle

Show the gap between task-level speedup evidence and org-level effect size, then specify the matched-cohort and interrupted time-series designs required before attributing any of the gain to AI.

Source video

AI Readiness Is an Infrastructure Problem, Not a Model Problem

Agent effectiveness is bounded by documentation clarity, codebase manageability, local CI latency, and test determinism; flaky tests are therefore a blocker to agent adoption with an owner and a deadline, and readiness tiers should gate rollout rather than sit beside it.

Angle

Reframe flaky-test and CI-latency work as agent-platform dependencies, and separate 'what is happening' AI metrics (utilization, impact, cost) from 'whether it is working' DevEx/DevProd metrics when defining the gate.

Source video

Project ideas

4

Value-Stream Bottleneck Ranker

new

A specialized agent deployed at the highest cycle-time-contribution non-code stage will reduce end-to-end lead time more than an equal-effort investment in code generation, because code generation is a small share of the value stream.

Proof of concept

Instrument repo, pipeline, review, and incident telemetry to attribute cycle time per stage across a small set of repositories; rank stages by contribution and wait-state share; deploy one narrow agent (e.g., review or incident triage) at the top-ranked stage while holding the coding copilot constant.

Measurement

Per-stage cycle-time share before/after, end-to-end lead time, median PR throughput, and deployment frequency, compared against a control repository set with no new agent.

Source video

Change-Confidence Instrument

beyond-evals

An evaluation harness that measures reviewer change confidence separately from maintainability will flag AI-assisted PRs that pass maintainability scoring but carry low reviewer trust; a PR-size-aware review policy can restore confidence without reducing throughput.

Proof of concept

Ship a paired instrument that captures maintainability and change confidence per PR alongside PR size, then A/B test an AI-specific review checklist applied above a size threshold.

Measurement

Change confidence delta per cohort, PR size distribution, review depth (comments per PR, review latency), escaped defect and change failure rate, with throughput held as a non-regression constraint.

Source video

Agent Readiness Gate

gatehouse

Organizations and repositories meeting explicit test-determinism and local CI latency thresholds show measurably higher AI impact scores and higher agent task success rates than those that do not, so readiness should gate agent rollout.

Proof of concept

Define a readiness scorecard (documentation, data structures, code manageability, local CI latency, flake rate) and wire it as a pre-rollout gate for agent enablement in one org unit; run a second unit ungated as a comparison.

Measurement

Correlation between readiness tier and measured AI impact metrics, agent task success and retry rates, local CI latency and flake rate, with adoption/utilization telemetry joined to DORA/DevEx outcome metrics.

Source video

Matched-Cohort AI Impact Estimator

movement-lab

Within-org matched-cohort and interrupted time-series comparison will shrink the observed AI throughput effect relative to naive org-level trends, because deployment-frequency trends, regional shifts, and general tooling improvements absorb part of the measured gain.

Proof of concept

Build an analysis harness that pairs AI-adopting teams with non-adopting teams inside the same organization on matched baselines, plus an interrupted time-series around adoption dates, and report confidence intervals on the effect.

Measurement

Effect-size difference between naive trend estimates and matched-cohort estimates for PR throughput and perceived rate of delivery, with sensitivity to cohort matching criteria.

Source video

Architectural implications

3

Value is concentrated outside code generation, so agent portfolios should be organized around SDLC stages rather than one general coding assistant.

Before

A single code-completion/copilot surface in the IDE as the primary AI investment.

After

Multiple narrow agents targeted at distinct lifecycle stages (refactoring, review, incident response, admin/onboarding), each with its own evaluation and telemetry.

Consequence

Instrumentation must be per-stage; a single PR-throughput KPI cannot attribute value to any given agent.

Source video

AI readiness is gated on infrastructure properties (fast local CI, non-flaky tests, clean data structures).

Before

Agents deployed onto existing repos and pipelines; quality issues blamed on the model.

After

Test/CI determinism and codebase hygiene treated as prerequisites in the agent adoption path, alongside a tiered readiness checklist (Bronze/Silver/Gold).

Consequence

Agent rollouts become gated engineering projects with infra owners, delaying deployment but improving reliability.

Source video

Measurement is split into AI-specific metrics (utilization, impact, cost) and foundational DevEx/DevProd metrics.

Before

Tracking adoption and token usage as the primary success signal for AI tooling.

After

Adoption and token telemetry paired with DORA/DevEx outcome metrics, with API telemetry feeding cost and utilization views.

Consequence

Prevents adoption theater; requires joining model/API telemetry with delivery pipeline data, which is an integration and data-ownership problem.

Source video

Tradeoffs and failure modes

4

AI-assisted code maintainability vs reviewer confidence

Benefit

Perceived maintainability of AI-influenced code improved (~+4%), suggesting easier modification.

Cost or risk

Change confidence dropped (~-6%) and PR size grew from ~44 to ~72 LOC, increasing review load and the chance that low-trust code is merged with less scrutiny.

Code maintainability perceived score went up ~4%, but change confidence went down ~6%.
Open source video
Source video

Velocity gains vs change failure rate

Benefit

Deployment frequency is steadily increasing across organizations.

Cost or risk

Change failure rate impact is volatile and can rise by up to 2 points against a 4% benchmark (~50% more defects), potentially negating throughput gains.

Change failure rate impact is highly volatile; top performers see increases up to 2% (when industry benchmark is 4%, representing 50% more defects).
Open source video
Source video

Adoption intensity vs token efficiency

Benefit

Juniors adopt AI most and seniors/staff+ save comparable time.

Cost or risk

Token consumption is not proportional to savings; cost-per-engineer metrics mislead because seniors achieve the same time savings with fewer tokens.

Staff+ and senior engineers save roughly the same amount of time as juniors while burning fewer tokens.
Open source video
Source video

Optimizing code generation vs system bottleneck

Benefit

Code generation agents are the most visible and easiest to instrument.

Cost or risk

Code gen is only 14-16% of the value stream; effort there is diluted by meetings, context switching, and release pipeline friction.

Code generation is not the bottleneck; code gen represents only 14-16% of the overall value stream.
Open source video
Source video

Open questions

4

Which specific SDLC bottlenecks should specialized agents target first, and how is their share of the value stream measured?

Why unresolved

The summary states code gen is 14-16% of the value stream but does not enumerate the remaining share by stage or how the value-stream measurement was constructed.

Research direction

Instrument a value-stream map per repo/org and rank stages by cycle-time contribution; prototype targeted agents in ranked order.

Source video

Does the drop in change confidence persist as teams adapt their review practices to larger AI-assisted PRs, or is it a fixed property of AI-generated code?

Why unresolved

The summary reports a cross-sectional perception delta (~-6%) without a mechanism or remediation experiment.

Research direction

Longitudinal study of change confidence alongside PR size and review policy changes; A/B test mandatory AI-specific review checklists.

Source video

Are the reported throughput gains (~7.76% median) separable from confounders like deployment-frequency trends, regional regulation shifts, and general tooling improvements?

Why unresolved

The summary presents regional divergence (North America up, Europe pulling back) without isolating AI as the causal driver.

Research direction

Matched-cohort or interrupted time-series analysis comparing AI-adopting and non-adopting teams within the same org.

Source video

What token-efficiency mechanisms let senior engineers get equal time savings at lower token cost?

Why unresolved

The summary asserts the efficiency gap by seniority but does not describe the behavioral or tooling difference producing it.

Research direction

Instrument prompt/context patterns by seniority, test whether context-selection strategies or task decomposition explain the gap.

Source video

Key claims

7
opinionVerification needed

Longitudinal data from 400+ companies (Nov 2024–Feb 2026) shows median PR throughput increase of 7.76% and average of 13.1%, with no company reaching 2x, 5x, or 10x.

Evidence

median PR throughput increased by 7.76% and average by 13.1% in longitudinal studies

Question

How was PR throughput weighted, and were AI-adopting teams compared against a matched non-adopting control?

Source video
factualVerification needed

Code generation accounts for only 14-16% of the overall value stream and is not the delivery bottleneck.

Evidence

Code generation is not the bottleneck; code gen represents only 14-16% of the overall value stream.

Question

What measurement method produced the 14-16% value-stream share, and how is 'value stream' defined?

Source video
opinionVerification needed

Perceived code maintainability rose ~4% while change confidence fell ~6%, and median PR size rose from ~44 to ~72 LOC.

Evidence

Median PR size has noticeably increased (from ~44 to 72 lines of code) over the study period.

Question

Were maintainability and change confidence measured from the same respondents and time window, and is the delta statistically significant?

Source video
causalVerification needed

Top performers saw change failure rate increase by up to 2 points against a 4% industry benchmark, equivalent to ~50% more defects.

Evidence

Change failure rate impact is highly volatile; top performers see increases up to 2% (when industry benchmark is 4%, representing 50% more defects).

Question

Is the change failure rate increase attributable to AI-generated changes specifically, or to correlated deployment-frequency increases?

Source video
comparativeVerification needed

Staff+ and senior engineers achieve roughly the same AI time savings as juniors while consuming fewer tokens, and juniors adopt AI most due to less unlearning.

Evidence

Staff+ and senior engineers save roughly the same amount of time as juniors while burning fewer tokens.

Question

How were token counts normalized across roles, and is self-reported time savings the only outcome measure?

Source video
opinionVerification needed

AI readiness depends on clear documentation, clean data structures, manageable code, fast local CI, and non-flaky test suites.

Evidence

AI readiness depends on clear documentation, data structures, manageable code, fast local CI, and non-flaky test suites.

Question

Do the Bronze/Silver/Gold readiness tiers correlate empirically with measured AI impact across orgs?

Source video
comparativeVerification needed

Targeting non-coding SDLC bottlenecks with specialized agents produced large documented ROI (e.g., Morgan Stanley 280,000+ developer hours annually, Faire 3,000 code reviews/week, Spotify handling 90% of incidents).

Evidence

Spotify's incident management agent platform handling 90% of incidents.

Question

Are these figures independently verified, and what baseline and denominator define each percentage or hours-saved number?

Source video

Connections

4