The Pragmatic Engineer · Published 2025-07-23

Measuring the impact of AI on software engineering – with Laura Tacho

privateReview: unreviewedConfidence: high
Developer productivityAI coding assistantsdora metricsmeasurement frameworkdeveloper experienceai-coding-assistantsmeasurement-frameworkdora-metricsdeveloper-experiencechange-failure-rateadoption-enablementagent-cost-accounting
Open on YouTube ↗

Summary

Overview

  • Speaker: Gergely Orosz and Laura Tacho
  • Channel: The Pragmatic Engineer
  • Main topic: Measuring developer productivity and the actual impact of AI tools on software engineering using data rather than hype
  • Purpose: To debunk AI coding hype with real data, present structured measurement frameworks for AI developer productivity, and guide engineering leaders on evaluating AI ROI. Laura Tacho, CTO at DX, joins Gergely Orosz to discuss the real data behind rolling out AI coding assistants at companies like Booking.com and Workhuman. They debunk common media oversimplifications and hype around AI code generation, examining developer satisfaction paradoxes, the DX AI Measurement Framework, and practical ways engineering leaders can measure and evaluate AI adoption.

Topic Map

Debunking AI Hype and Media Oversimplification

  • Explanation: Discussion on sensationalist media headlines regarding AI replacing developers and the actual reality on the ground.
  • Key claims:
    • Most hype in the media about AI gets things wrong due to oversimplification.
    • The burden is on engineers to set the record straight using actual data.
    • Sensational headlines sell ads and attract clicks, requiring high media and data literacy from leaders.
  • Examples:
    • Forbes: Are Coders' Jobs At Risk? AI's Impact On The Future Of Programming
    • CIO: AI coding assistants wave goodbye to junior developers
    • Gizmodo: OpenAI Just Released a Coding Tool to 'Help' Programmers (Replace Their Jobs, Probably)
  • Terminology:
    • media literacy
    • data literacy
    • ads-supported media
    • hype cycle
  • Why it matters: Engineering leaders need realistic expectations to avoid poor strategic investments and unwarranted panic among developers.

The AI Productivity Paradox and Developer Satisfaction

  • Explanation: Examining why time savings from AI do not necessarily equate to higher developer job satisfaction.
  • Key claims:
    • AI accelerates parts of the job developers enjoy (writing code), leaving behind administrative work, meetings, and toil.
    • Developers who enjoy coding feel less satisfied when AI reduces their coding time in favor of toil.
    • Acceptance rates of AI suggestions do not measure overall business impact or developer happiness.
  • Examples:
    • Developers reporting that debugging tricky stack traces saves more time than AI code generation.
  • Terminology:
    • developer satisfaction
    • cognitive load
    • toil
    • acceptance rate
  • Why it matters: Understanding developer sentiment prevents mistaking tool adoption for true productivity or team morale improvement.

DX AI Measurement Framework

  • Explanation: A structured framework covering Utilization, Impact, and Cost to measure AI's effect on software engineering.
  • Key claims:
    • Measuring only lines of code or acceptance rate is insufficient for business impact.
    • Framework categorizes metrics into Utilization, Impact, and Cost.
    • Combining self-reported developer experience metrics with system/workflow metrics provides a complete picture.
  • Examples:
    • DX AI Measurement Framework table showing Utilization (DAU/WAU, % PRs AI-assisted), Impact (AI-driven time savings, DORA 4 metrics, developer satisfaction), and Cost (AI spend per developer, net time-gain ROI).
  • Terminology:
    • DX AI Measurement Framework
    • DAU/WAU
    • DORA metrics
    • developer experience index
  • Why it matters: Gives engineering leaders objective tools to justify AI spend and evaluate actual productivity gains.

Case Studies: Booking.com and Workhuman

  • Explanation: Real-world examples of how large organizations successfully rolled out AI coding assistants and measured adoption.
  • Key claims:
    • Adoption is the key to getting better results from AI tools.
    • Concerted organizational efforts like office hours and workshops drive high adoption rates.
    • Workhuman achieved 65% developer adoption through intentional enablement.
  • Examples:
    • Booking.com and Workhuman adoption initiatives.
    • Workhuman reaching 65% weekly/daily usage of AI tools compared to 50% industry median.
  • Terminology:
    • adoption rate
    • enablement
    • cohort analysis
    • consumption-based pricing
  • Why it matters: Demonstrates that tools alone do not drive productivity without deliberate organizational change management.

Code Review and Stack Trace Analysis as Top Time-Savers

  • Explanation: Survey data showing what developers actually report as their biggest time-saving use cases for AI assistants.
  • Key claims:
    • AI code generation is not the primary time-saver reported by developers.
    • Debugging tricky stack traces and refactoring existing code rank higher for time savings than mid-loop code generation.
  • Examples:
    • Survey chart showing stack trace analysis and refactoring existing code at the top of time-saving use cases.
  • Terminology:
    • stack trace analysis
    • refactoring
    • mid-loop code generation
    • code explanation
  • Why it matters: Helps teams focus AI tooling and training on the highest-value developer workflows.

Key Points

Code volume is not business value

  • Explanation: Measuring productivity by lines of code generated or accepted suggestions is misleading, as more code can mean higher maintenance and complexity.
  • Evidence: DORA studies showing that higher AI adoption can correlate with slight decreases in delivery stability due to increased batch size and complexity.
  • Practical implication: Focus metrics on delivery throughput, change failure rate, and developer experience rather than raw code output.

Adoption requires intentional enablement

  • Explanation: Simply purchasing AI tool licenses does not guarantee usage; companies must actively train and support developers.
  • Evidence: Workhuman reached 65% adoption through office hours, workshops, and proactive change management.
  • Practical implication: Allocate resources for training, internal community building, and enablement when rolling out new dev tools.

AI makes large changes easier, increasing risk

  • Explanation: AI tools allow developers to write large amounts of code quickly, which can increase change failure rates if batch sizes grow too large.
  • Evidence: DORA data indicating a slight estimated reduction in delivery stability when AI adoption increases significantly without process safeguards.
  • Practical implication: Enforce rigorous code reviews, automated testing, and small batch sizes even when using AI.

Frameworks, Models & Processes

DX AI Measurement Framework

  • How it works: Measures AI productivity across three core pillars: Utilization, Impact, and Cost, blending quantitative tool metrics with developer experience data.
  • Components:
    • Utilization (DAU/WAU, % PRs AI-assisted, tasks assigned to agents)
    • Impact (AI-driven time savings, developer satisfaction, DORA 4 metrics, change confidence, human equivalent hours)
    • Cost (AI spend per developer, net time gain per developer, agent hourly rate)
  • When to use: When evaluating the return on investment and team impact of introducing AI coding assistants or autonomous agents.

Examples & Case Studies

Workhuman implemented structured enablement programs for AI coding assistants.

  • Illustrates: How intentional organizational effort drives developer tool adoption.
  • Lesson: Active enablement and training push adoption rates well above the industry median.

DORA study on AI adoption impact on delivery stability.

  • Illustrates: The risk of increased batch sizes and lower stability when AI speeds up code creation.
  • Lesson: Speed gains must be balanced with robust CI/CD and review practices to maintain stability.

Actionable Takeaways

  • Immediate:
    • Measure AI tool adoption (DAU/WAU) alongside developer sentiment.
    • Focus AI assistance on high-value tasks like stack trace debugging and refactoring.
    • Avoid using lines of code as a productivity metric.
  • Strategic:
    • Treat AI rollouts as organizational experiments requiring structured measurement.
    • Balance speed metrics with quality and stability metrics (DORA).
    • Prepare for consumption-based pricing models and justify tool ROI to leadership.
  • Questions to investigate:
    • What are our developers actually using AI assistants for most frequently?
    • How does our AI spend compare to the net time saved per developer?
    • Are we seeing any degradation in delivery stability or code quality following AI tool adoption?

Claims Worth Verifying

  • Industry median adoption for AI coding assistants is around 50%. (statistic)
  • Average AWS engineer spends only 20% of their time coding. (statistic)
  • DORA study predicts a slight decrease in delivery stability when AI adoption increases by 25%. (research finding)

Notable Quotes

"The best way to deal with hype is with data." (at 118:25) "Data beats hype every time." (at 1123:37) "AI isn't this like magic bullet that's going to solve everything." (at 733:01)

Compressed Summary

  • AI hype in media oversimplifies productivity gains and ignores developer sentiment.
  • DX AI Measurement Framework evaluates AI across Utilization, Impact, and Cost.
  • Stack trace analysis and refactoring rank higher for time savings than code generation.
  • High adoption requires active enablement, office hours, and structured training.
  • Speed increases from AI can negatively impact stability if batch sizes grow too large.
  • Keywords: developer productivity, ai coding assistants, dora metrics, measurement framework, developer experience
  • Core insight: AI coding tools save developer time primarily through debugging and refactoring rather than raw code generation, and realizing true ROI requires measuring developer experience alongside delivery performance and cost.

Core insights

6
Empirical Resulthigh noveltymoderate evidence

The highest-value AI use cases reported by developers are debugging tricky stack traces and refactoring existing code — not in-loop code generation. This inverts the common assumption that code generation is the primary value driver of coding assistants and implies tooling, UX, and training investment should target code comprehension and modification workflows instead.

Why it matters

If agent/tool design optimizes for generation throughput while the real time savings come from comprehension and refactoring, the product roadmap, prompt/tool surface, and evaluation suite are all mis-targeted; it also suggests retrieval/context over the codebase matters more than raw generation quality.

Generalization

Automation value concentrates where context assembly and reasoning over existing state is hardest, not where output emission is easiest — applies to agents generally, not just coding assistants.

Debugging tricky stack traces and refactoring existing code rank higher for time savings than mid-loop code generation.
Open source video
AI code generation is not the primary time-saver reported by developers.
Open source video
Mental Modelhigh noveltymoderate evidence

AI time savings can reduce developer satisfaction because AI absorbs the enjoyable part of the job (writing code) while leaving the residual work — administrative tasks, meetings, toil. Time saved is therefore not a proxy for developer experience or morale.

Why it matters

Teams that measure only time saved or acceptance rate can ship a tool that increases throughput while degrading retention, engagement, and the developer experience index; it forces measurement of sentiment alongside throughput.

Generalization

Automating a task's pleasant portion reallocates human effort toward its least desirable residue; agent design should account for the shape of remaining work, not just total volume removed.

AI accelerates parts of the job developers enjoy (writing code), leaving behind administrative work, meetings, and toil.
Open source video
Developers who enjoy coding feel less satisfied when AI reduces their coding time in favor of toil.
Open source video
Failure Modemedium noveltymoderate evidence

Higher AI adoption can correlate with slightly reduced delivery stability because AI makes large changes easy to produce, inflating batch size and complexity. Speed of change authoring is a stability risk unless batch size, review rigor, and test automation are actively constrained.

Why it matters

Any system that lowers the marginal cost of producing change needs a compensating control on change size; otherwise the throughput gain is partially paid back in change failure rate, which is directly observable in DORA-style telemetry.

Generalization

Lowering the cost of a risky action shifts the bottleneck to the review/verification layer; harnesses should enforce bounded change units and gating proportional to generation velocity.

DORA studies showing that higher AI adoption can correlate with slight decreases in delivery stability due to increased batch size and complexity.
Open source video
AI tools allow developers to write large amounts of code quickly, which can increase change failure rates if batch sizes grow too large.
Open source video
Architecturemedium noveltymoderate evidence

Productivity measurement is architecturally layered into three separated pillars — Utilization, Impact, Cost — because adoption telemetry (utilization) cannot stand in for business outcome (impact). Utilization signals such as DAU/WAU, % PRs AI-assisted, and tasks assigned to agents are leading indicators only.

Why it matters

It prescribes a separation of concerns in the measurement plane: tool-level instrumentation, delivery/outcome metrics, and cost accounting must be independent subsystems combined at query time, not conflated into one adoption number.

Generalization

For any AI system in production, adoption/config telemetry, outcome metrics, and cost accounting belong in distinct layers with explicit joins, since optimizing the adoption layer alone creates false confidence.

Measuring only lines of code or acceptance rate is insufficient for business impact.
Open source video
Framework categorizes metrics into Utilization, Impact, and Cost.
Open source video
Combining self-reported developer experience metrics with system/workflow metrics provides a complete picture.
Open source video
Predictionmedium noveltyweak evidence

The framework includes agent-specific cost and unit-economics metrics — AI spend per developer, net time gain per developer, agent hourly rate, human equivalent hours — and a utilization metric for 'tasks assigned to agents', indicating a shift from inline assistant measurement toward delegated-agent accounting.

Why it matters

Per-agent hourly rate and human-equivalent-hours convert agent work into a comparable unit of account, which is a prerequisite for capacity planning, budget attribution, and deciding when a task should be delegated to an agent versus a human.

Generalization

As agents take on delegated tasks rather than inline suggestions, the natural unit of measurement shifts from suggestion acceptance to task-level cost-per-outcome.

Utilization (DAU/WAU, % PRs AI-assisted, tasks assigned to agents)
Open source video
Cost (AI spend per developer, net time gain per developer, agent hourly rate)
Open source video
Impact (AI-driven time savings, developer satisfaction, DORA 4 metrics, change confidence, human equivalent hours)
Open source video
Practicelow noveltymoderate evidence

Adoption is the binding constraint on realized AI value: licenses do not produce usage, and deliberate enablement (office hours, workshops, change management) is what moves adoption — Workhuman reached 65% versus a ~50% industry median.

Why it matters

It reframes rollout as an organizational change problem rather than a procurement problem, implying budget and staffing for enablement are part of the technical deployment, not overhead; low adoption silently invalidates any downstream ROI measurement.

Generalization

Instrumentation of a new capability is meaningless without a measured adoption floor; adoption-support mechanisms are a required component of deployment, not an optional extra.

Simply purchasing AI tool licenses does not guarantee usage; companies must actively train and support developers.
Open source video
Workhuman reached 65% adoption through office hours, workshops, and proactive change management.
Open source video

Deep dives

5

Where AI time savings actually concentrate: comprehension and refactoring versus generation

Research question

Do observed task-level workflow traces confirm that debugging and refactoring, rather than in-loop code generation, dominate AI-attributable time savings, and what context-assembly capability explains the inversion?

Why

If the value center is comprehension and modification of existing code, then roadmap, tool surface, and evaluation suites optimized for generation throughput are mis-targeted; it also shifts the hard engineering problem to codebase retrieval and context assembly rather than output emission.

Debugging tricky stack traces and refactoring existing code rank higher for time savings than mid-loop code generation.
Open source video
AI code generation is not the primary time-saver reported by developers.
Open source video
Source video

The toil residual: whether automating enjoyable work structurally shifts human effort to the least desirable residue

Research question

Is the post-automation shift of human effort toward administrative work, meetings, and toil an inherent property of automating enjoyable tasks, or an artifact of how these teams deployed the tools?

Why

It separates 'time saved' from 'developer experience improved' as measurement categories, and forces adoption decisions to include the shape of the remaining work rather than only the volume removed; a tool can raise throughput while degrading satisfaction and retention.

AI accelerates parts of the job developers enjoy (writing code), leaving behind administrative work, meetings, and toil.
Open source video
Developers who enjoy coding feel less satisfied when AI reduces their coding time in favor of toil.
Open source video
Source video

Generation velocity as a stability risk: compensating controls at the change-integration boundary

Research question

Which specific process safeguards (batch-size thresholds, review depth, test coverage gates) quantitatively offset the delivery-stability loss associated with rising AI adoption?

Why

Any system that lowers the marginal cost of producing a risky change shifts the bottleneck to verification; without a bound on change units, the throughput gain is partially paid back in change failure rate, which is directly observable in delivery telemetry.

DORA studies showing that higher AI adoption can correlate with slight decreases in delivery stability due to increased batch size and complexity.
Open source video
AI tools allow developers to write large amounts of code quickly, which can increase change failure rates if batch sizes grow too large.
Open source video
Source video

Layered measurement plane: separating utilization telemetry from outcome and cost accounting

Research question

Does separating Utilization, Impact, and Cost into independently instrumented pillars, joined only at reporting time, produce measurably different investment decisions than a single acceptance-rate or adoption number?

Why

The separation distinguishes 'nobody is using it' from 'people use it but nothing improves', which are fixed by entirely different interventions; conflating them creates false confidence and invalidates any downstream ROI claim, especially when adoption itself is low.

Measuring only lines of code or acceptance rate is insufficient for business impact.
Open source video
Framework categorizes metrics into Utilization, Impact, and Cost.
Open source video
Combining self-reported developer experience metrics with system/workflow metrics provides a complete picture.
Open source video
Source video

Unit economics of delegated agent work: agent hourly rate and human equivalent hours

Research question

Can per-task attribution and cost accounting for delegated agent runs produce reliable build-versus-delegate decisions, or does agent work resist task-level outcome attribution?

Why

Agent hourly rate and human equivalent hours convert agent work into a comparable unit of account, which is a prerequisite for capacity planning, budget attribution, and deciding per task class whether to delegate to an agent or a human; without it, delegated work is unmeasured spend.

Cost (AI spend per developer, net time gain per developer, agent hourly rate)
Open source video
Utilization (DAU/WAU, % PRs AI-assisted, tasks assigned to agents)
Open source video
Source video

Article ideas

4

Your AI Coding Assistant Is Optimizing the Wrong Half of the Job

If the largest reported time savings come from debugging stack traces and refactoring existing code rather than in-loop generation, then tooling, training, and evaluation built around generation throughput are systematically mis-targeted, and the real engineering problem is context assembly over an existing codebase rather than output emission.

Angle

Contrast the generation-centric narrative with the reported time-savings ranking, then derive concrete consequences for tool surface, retrieval/context design, and evaluation suites.

Source video

Time Saved Is Not Satisfaction: The Toil Residual

Measuring only time saved or acceptance rate can validate a tool that increases throughput while degrading the developer experience, because AI absorbs the enjoyable portion of the job and leaves administrative work, meetings, and toil behind as the residue of human effort.

Angle

Take the satisfaction paradox seriously as a design constraint: reason about the shape of the remaining work, not just the volume removed, and pair throughput metrics with sentiment measurement.

Source video

Cheap Code, Expensive Change: Why AI Adoption Shows Up in Change Failure Rate

Lowering the marginal cost of authoring code inflates batch size and complexity, so generation velocity is a stability risk unless batch size, review rigor, and test automation are enforced as pipeline-level constraints independent of how fast code is written.

Angle

Frame AI adoption as a bottleneck shift from authoring to verification, and argue that the compensating control belongs in the integration boundary rather than in developer discipline.

Source video

Adoption Is Part of the Deployment, Not Overhead

Because licenses do not produce usage, any ROI measurement taken at low adoption is measuring nothing; enablement mechanisms such as office hours, workshops, and change management must be budgeted and staffed as a required component of the technical deployment, not an optional extra.

Angle

Reframe rollout as an organizational change problem with a measured adoption floor, and show how low adoption silently invalidates the entire downstream measurement stack.

Source video

Project ideas

4

Change-Risk Governor

gatehouse

Enforcing bounded change units plus review gating proportional to AI generation velocity keeps change failure rate flat as AI adoption rises, without erasing the authoring time savings.

Proof of concept

A CI-side interceptor that computes diff size and complexity for incoming changes, caps or splits oversized AI-authored batches, and escalates review depth or required test coverage as change velocity increases.

Measurement

Change failure rate and batch-size distribution against AI adoption level, compared to a control group with no governor, with lead time tracked to confirm savings are not lost.

Source video

Triangulated Time-Savings Audit

beyond-evals

Self-reported AI time savings systematically diverge from trace-derived task time in a direction predictable from task type, with the largest bias on generation tasks and the smallest on debugging and refactoring tasks.

Proof of concept

Instrument task-level workflow traces for a cohort of developers for a fixed period and compare against their self-reported per-task time-savings estimates over the same window.

Measurement

Mean and distribution of divergence between self-report and trace-derived task time, segmented by task type; correlation of self-report with system telemetry.

Source video

Three-Pillar Measurement Plane

new

Keeping Utilization, Impact, and Cost as independently instrumented pillars joined only at query time changes investment decisions compared to acceptance-rate-driven dashboards, because it separates low adoption from usage without improvement.

Proof of concept

A reporting layer with three separate ingestion paths (tool telemetry, delivery/outcome and survey data, cost accounting) that can only be combined through explicit joins, with no conflated productivity score.

Measurement

Decision audit comparing interventions chosen from the three-pillar view versus an acceptance-rate dashboard, plus classification accuracy of low-adoption versus low-impact cases.

Source video

Agent Task Cost Attribution Ledger

movement-lab

Per-task cost attribution using agent hourly rate and human equivalent hours yields stable per-task-class delegation thresholds that beat a uniform delegate-everything policy on net time gain per developer.

Proof of concept

A ledger that records each delegated agent run with cost, task class, and outcome, computes agent hourly rate and human equivalent hours per task class, and emits a build-versus-delegate recommendation per class.

Measurement

Net time gain per developer and cost per accepted outcome under threshold-based delegation versus uniform delegation; stability of thresholds across weeks.

Source video

Architectural implications

3

Adoption metrics and outcome metrics are structurally different concerns that must not collapse into a single productivity number.

Before

A single metric such as suggestion acceptance rate or lines of code generated was treated as the productivity signal.

After

Three independent measurement pillars — Utilization, Impact, Cost — joined at reporting time, mixing tool telemetry with self-reported developer experience and system/workflow data.

Consequence

ROI decisions can distinguish 'nobody is using it' from 'people use it but nothing improves', which are fixed by entirely different interventions.

Source video

Cheap code authoring requires a compensating control at the change-integration boundary.

Before

Change size was implicitly bounded by the human cost of writing code; review capacity naturally matched authoring rate.

After

Review rigor, automated testing, and small batch sizes must be enforced as pipeline-level constraints independent of how fast code is generated.

Consequence

Generation velocity can be increased without proportionally increasing change failure rate, at the cost of added CI/review gating infrastructure.

Source video

Agent work is beginning to be accounted for in comparable units to human work.

Before

Assistant value was measured via acceptance rate of inline suggestions, with no cost model per unit of work.

After

Metrics such as agent hourly rate, net time gain per developer, and human equivalent hours place delegated agent tasks in an economic frame comparable to human labor.

Consequence

Enables capacity planning and build-vs-delegate decisions per task class, and makes cost a first-class dimension of agent evaluation.

Source video

Tradeoffs and failure modes

3

AI-assisted change velocity

Benefit

Developers can produce large amounts of code quickly, saving time on authoring.

Cost or risk

Larger batch sizes and increased complexity correlate with slight decreases in delivery stability and higher change failure rates.

AI tools allow developers to write large amounts of code quickly, which can increase change failure rates if batch sizes grow too large.
Open source video
Source video

Time savings versus developer satisfaction

Benefit

AI removes time from the coding portion of the job.

Cost or risk

The remaining work skews toward administrative tasks, meetings, and toil, which can lower satisfaction among developers who enjoy coding.

Developers who enjoy coding feel less satisfied when AI reduces their coding time in favor of toil.
Open source video
Source video

Tool procurement versus realized adoption

Benefit

Purchasing licenses makes AI assistance available at scale immediately.

Cost or risk

License purchase does not guarantee usage; without enablement, adoption stays low and measured ROI collapses.

Simply purchasing AI tool licenses does not guarantee usage; companies must actively train and support developers.
Open source video
Source video

Open questions

4

Which specific process safeguards (batch-size thresholds, review depth, test coverage gates) quantitatively offset the delivery-stability loss associated with rising AI adoption?

Why unresolved

The summary reports a correlation between AI adoption and slightly reduced stability but names no measured threshold or counterfactual control configuration.

Research direction

Run controlled rollouts varying batch-size caps and automated-test gates while tracking change failure rate against AI adoption levels.

Source video

Are self-reported AI time savings reliable enough to drive ROI decisions, or do they need triangulation with system/workflow telemetry?

Why unresolved

The framework mixes self-reported developer experience data with system metrics but the summary offers no validation of the self-report component or its bias profile.

Research direction

Compare self-reported AI-driven time savings against trace/task-level workflow measurements for the same developers over the same period.

Source video

Does the shift toward 'tasks assigned to agents' require a fundamentally different measurement methodology than inline assistant utilization?

Why unresolved

The framework lists agent task assignment and agent hourly rate as metrics but the summary gives no method for attributing outcomes to delegated agent work.

Research direction

Prototype per-task attribution and cost accounting for delegated agent runs and compare against inline-suggestion acceptance metrics.

Source video

Is the post-automation shift of human work toward toil an inherent property of automating enjoyable tasks, or an artifact of how these teams deployed the tools?

Why unresolved

The summary states the satisfaction paradox as an observed pattern without testing alternative deployments that redistribute rather than remove the enjoyable work.

Research direction

Compare satisfaction across teams that route AI to toil-heavy tasks versus code generation, holding time savings constant.

Source video

Key claims

7
causalVerification needed

Higher AI adoption can correlate with slight decreases in delivery stability, attributed to increased batch size and complexity.

Evidence

DORA studies showing that higher AI adoption can correlate with slight decreases in delivery stability due to increased batch size and complexity.

Question

Does the DORA dataset establish a directional causal link, and how large is the estimated effect after controlling for team and service characteristics?

Source video
comparativeVerification needed

Debugging stack traces and refactoring existing code are higher time-saving use cases than in-loop code generation.

Evidence

Debugging tricky stack traces and refactoring existing code rank higher for time savings than mid-loop code generation.

Question

What sample size and survey methodology produced this ranking, and does it hold when measured against observed workflow data rather than self-report?

Source video
opinionVerification needed

Measuring only lines of code or acceptance rate is insufficient for business impact.

Evidence

Measuring only lines of code or acceptance rate is insufficient for business impact.

Question

Is there empirical evidence that acceptance-rate-optimized teams make worse tooling or investment decisions than outcome-metric-optimized teams?

Source video
comparativeVerification needed

Workhuman reached 65% weekly/daily AI tool usage versus a 50% industry median, via office hours, workshops, and change management.

Evidence

Workhuman reaching 65% weekly/daily usage of AI tools compared to 50% industry median.

Question

What population and time window define the 50% industry median, and how much of the 65% is attributable to enablement versus pricing, mandate, or tool quality?

Source video
causalVerification needed

Purchasing AI tool licenses does not guarantee usage.

Evidence

Simply purchasing AI tool licenses does not guarantee usage; companies must actively train and support developers.

Question

Is there a quantified relationship between enablement investment and adoption rate across a broad sample of organizations?

Source video
causalVerification needed

AI makes parts of the job developers enjoy faster while leaving administrative work, meetings, and toil behind.

Evidence

AI accelerates parts of the job developers enjoy (writing code), leaving behind administrative work, meetings, and toil.

Question

Does AI adoption measurably shift the composition of developer time toward administrative categories in time-use data?

Source video
factualVerification needed

Combining self-reported developer experience metrics with system/workflow metrics provides a more complete picture than either alone.

Evidence

Combining self-reported developer experience metrics with system/workflow metrics provides a complete picture.

Question

What is the measured incremental predictive value of developer-experience survey data over system/workflow telemetry for forecasting delivery outcomes?

Source video

Connections

5