The Pragmatic Engineer · Published 2026-10-07

Building resilient systems with Sam Newman

privateReview: unreviewedConfidence: high
microservicesresiliencedistributed systemsidempotencyObservabilitydistributed-systemscognitive-debtchaos-engineeringai-coding-assistants
Open on YouTube ↗

Summary

Overview

  • Speaker: Sam Newman
  • Channel: The Pragmatic Engineer
  • Main topic: Building Resilient Distributed Systems and Microservices
  • Purpose: To educate software engineers and architects on the practical principles of distributed systems, resilience engineering, and how to design systems that survive real-world failures. In this interview, Gergely Orosz and Sam Newman discuss the realities of building resilient distributed systems and microservices. They cover the three core rules of distributed systems (information cannot be transmitted instantaneously, sometimes the intended recipient is unavailable, and resource pools are finite), the four dimensions of resilience (robustness, rebound, graceful extensibility, and sustained adaptability), and the implications of using AI coding assistants in software engineering.

Topic Map

The Three Rules of Distributed Systems

  • Explanation: Sam Newman outlines three fundamental physical constraints that govern all distributed systems and microservice architectures.
  • Key claims:
    • You cannot send information instantaneously from point A to point B.
    • Sometimes the thing you want to talk to isn't there.
    • Resource pools are not infinite.
  • Examples:
    • Fiber optic cable cuts in New Jersey causing flight cancellations.
    • Database connection pools getting exhausted during traffic spikes.
  • Terminology:
    • distributed systems
    • resource pools
    • latency
  • Why it matters: Understanding these physical realities prevents architects from assuming ideal network and hardware conditions.

Microservices as an Architecture of Last Resort

  • Explanation: Newman explains why microservices should not be adopted by default for magic scaling benefits, but rather as a tool to achieve organizational autonomy.
  • Key claims:
    • Microservices are an architecture of last resort.
    • They are a tool to achieve organizational autonomy and high-degree team independence.
    • They push systems toward a more aggressive distribution of state.
  • Examples:
    • Uber and Netflix adopting microservices to scale team structures rather than just technical performance.
  • Terminology:
    • microservices
    • monolith
    • organizational autonomy
  • Why it matters: Teams often adopt microservices for the wrong reasons, leading to unnecessary complexity without organizational benefits.

Four Dimensions of Resilience

  • Explanation: Drawing on David Woods' resilience engineering framework, Newman discusses robustness, rebound, graceful extensibility, and sustained adaptability.
  • Key claims:
    • Robustness is the ability to absorb known perturbations without failing.
    • Rebound is how quickly a system recovers when something goes wrong.
    • Graceful extensibility is handling the unexpected when known defenses fail.
    • Sustained adaptability is the ongoing evolution of the system to changing environments.
  • Examples:
    • Netflix Chaos Monkey turning off parts of the system to test rebound and robustness.
    • Kubernetes pod failures and automatic replacements.
  • Terminology:
    • robustness
    • rebound
    • graceful extensibility
    • sustained adaptability
    • resilience engineering
  • Why it matters: Provides a comprehensive mental model for evaluating system resilience beyond simple uptime metrics.

Idempotency Keys vs. Fingerprints

  • Explanation: A deep dive into how to handle duplicate requests safely in distributed APIs using idempotency keys and cryptographic fingerprints.
  • Key claims:
    • Idempotency keys require clients to send unique identifiers for operations, which can be painful to retrofit.
    • Fingerprints hash request payloads on the server side to detect duplicate operations automatically.
    • Both approaches have trade-offs regarding false negatives and client integration friction.
  • Examples:
    • Payment processing APIs receiving duplicate charge requests due to network retries.
  • Terminology:
    • idempotency key
    • fingerprint
    • retry storms
  • Why it matters: Prevents catastrophic double-processing bugs in financial and transactional distributed systems.

AI Coding Assistants, Cognitive Debt, and Cognitive Surrender

  • Explanation: Examining the risks of relying on AI tools for code generation, including the loss of deep understanding and architectural coherence.
  • Key claims:
    • The tech world is quite naive about AI and fundamentally misunderstands its impact on engineering depth.
    • Cognitive surrender occurs when developers stop understanding how their systems work because AI wrote the code.
    • Production is truth, and code review alone is insufficient when dealing with AI-generated code.
  • Examples:
    • Developers generating complex algorithms with LLMs without understanding edge cases or failure modes.
  • Terminology:
    • cognitive debt
    • cognitive surrender
    • LLMs
    • tacit knowledge
  • Why it matters: Highlights the danger of trading short-term coding speed for long-term system maintainability and understanding.

Key Points

Distributed systems physics cannot be ignored

  • Explanation: Network latency, component unreliability, and resource exhaustion are unavoidable laws of distributed architecture.
  • Evidence: Real-world cloud outages and network severances.
  • Practical implication: Architects must design for timeouts, circuit breakers, and backpressure from day one.

Production is truth

  • Explanation: Specifications, documentation, and mental models are flawed; only how code behaves in production reveals the true nature of a system.
  • Evidence: Discrepancies between expected system behavior and actual production failures.
  • Practical implication: Invest heavily in observability, chaos engineering, and real-world testing rather than relying solely on static code reviews.

AI tools increase cognitive debt

  • Explanation: Using AI to generate code without deep comprehension leads to systems that nobody on the team fully understands.
  • Evidence: Developers accepting AI suggestions without reviewing underlying architectural implications.
  • Practical implication: Teams must maintain shared mental models of architecture and use simulation-based testing like Antithesis to verify correctness.

Frameworks, Models & Processes

David Woods' Resilience Engineering Framework

  • How it works: Evaluates system resilience across four distinct operational dimensions to understand how systems respond to known and unknown pressures.
  • Components:
    • Robustness
    • Rebound
    • Graceful Extensibility
    • Sustained Adaptability
  • When to use: When designing, auditing, or troubleshooting distributed systems and enterprise architectures.

CAP Theorem & PACELC

  • How it works: Describes the fundamental trade-offs in distributed data stores between consistency, availability, and partition tolerance, extended by PACELC to account for latency.
  • Components:
    • Consistency
    • Availability
    • Partition Tolerance
    • Latency
  • When to use: When choosing and configuring distributed databases and replication strategies.

Examples & Case Studies

AWS US-East-1 region outages taking down numerous major services.

  • Illustrates: Rule number 2 and 3 of distributed systems: components fail and resource pools saturate.
  • Lesson: Single-region dependencies and unconstrained retries can cascade failures globally.

Payment gateway processing duplicate charges due to network timeout retries.

  • Illustrates: The necessity of idempotency in financial transactions.
  • Lesson: Use idempotency keys or request fingerprinting to ensure operations are safely retryable.

Actionable Takeaways

  • Immediate:
    • Review retry logic in your services to prevent retry storms.
    • Implement proper circuit breakers and timeouts across service boundaries.
    • Evaluate where your system relies on single points of failure.
  • Strategic:
    • Treat microservices as an organizational scaling tool, not a default technical choice.
    • Balance AI-assisted coding with rigorous simulation and testing to avoid cognitive surrender.
    • Invest in observability and telemetry (like OpenTelemetry) to understand production truth.
  • Questions to investigate:
    • Where are the hidden resource constraints in our distributed call graph?
    • How do we handle duplicate requests and ensure idempotency across our APIs?
    • Do our engineers understand the codebase, or have we fallen into cognitive surrender with AI tools?

Claims Worth Verifying

  • You can send information instantaneously from point A to point B. (physical law violation)
  • Resource pools in distributed systems are infinite. (architectural misconception)

Notable Quotes

"Microservices is an architecture of last resort." (at 0:00) "The first is that you can't send information instantaneously from point A to point B." (at 0:13) "Sometimes the thing you want to talk to isn't there." (at 0:17) "Resource pools are not infinite." (at 0:20) "The tech world in general is quite naive about AI and fundamentally misunderstands..." (at 0:33) "Production is truth." (at 1:49)

Compressed Summary

  • Distributed systems are governed by physical limits: speed of light, component failures, and finite resources.
  • Microservices are an architectural tool of last resort, best used for organizational autonomy.
  • Resilience consists of four dimensions: robustness, rebound, graceful extensibility, and sustained adaptability.
  • AI coding assistants risk introducing cognitive debt and cognitive surrender if teams stop understanding their code.
  • Keywords: microservices, resilience, distributed systems, idempotency, observability
  • Core insight: Distributed systems inevitably fail due to physical limits, requiring deliberate resilience engineering and deep human understanding rather than naive reliance on magical tooling.

Core insights

5
Failure Modehigh noveltymoderate evidence

AI-generated code creates 'cognitive surrender' — developers lose understanding of their own systems — which breaks the assumption that code review is a sufficient correctness gate; verification must shift toward production behavior and simulation-based testing.

Why it matters

For agentic systems and AI-assisted engineering, the human review layer degrades precisely as generation speed increases, so teams need alternative verification mechanisms (simulation, production telemetry) rather than relying on reviewers who no longer hold a mental model.

Generalization

Any pipeline where generation outpaces human comprehension needs an executable/simulation oracle, not just human inspection.

Cognitive surrender occurs when developers stop understanding how their systems work because AI wrote the code.
Open source video
Production is truth, and code review alone is insufficient when dealing with AI-generated code.
Open source video
Teams must maintain shared mental models of architecture and use simulation-based testing like Antithesis to verify correctness.
Open source video
Mechanismmedium noveltymoderate evidence

Duplicate-request safety can be handled by client-supplied idempotency keys or by server-side request fingerprinting (hashing payloads to auto-detect duplicates); the two approaches trade off client integration friction against false-negative risk.

Why it matters

In agentic/tool-executing systems, retries are routine, so double-execution of side-effecting tools (payments, writes) is a first-class hazard; choosing where the dedup responsibility lives changes retrofit cost and correctness guarantees.

Generalization

Idempotency enforcement can be pushed to the server as content-fingerprinting when clients cannot be modified, at the cost of detection ambiguity.

Idempotency keys require clients to send unique identifiers for operations, which can be painful to retrofit.
Open source video
Fingerprints hash request payloads on the server side to detect duplicate operations automatically.
Open source video
Both approaches have trade-offs regarding false negatives and client integration friction.
Open source video
Mental Modelmedium noveltystrong evidence

Resilience decomposes into four distinct dimensions (robustness, rebound, graceful extensibility, sustained adaptability), separating the handling of known perturbations from the handling of novel pressures when known defenses fail.

Why it matters

Uptime-style metrics collapse these dimensions; treating 'graceful extensibility' as a separate axis forces designers to plan for failures outside the anticipated set rather than only hardening against known ones.

Generalization

Reliability evaluation of any system (including agents) should distinguish absorbing known faults from adapting to unforeseen ones.

Robustness is the ability to absorb known perturbations without failing.
Open source video
Graceful extensibility is handling the unexpected when known defenses fail.
Open source video
Sustained adaptability is the ongoing evolution of the system to changing environments.
Open source video
Architecturemedium noveltymoderate evidence

Microservices should be adopted as an architecture of last resort, chosen for organizational autonomy and team independence, not for automatic scaling benefits; they inherently push the system toward aggressive distribution of state.

Why it matters

Directly challenges the default 'decompose into services' instinct that also drives over-eager multi-agent decomposition; the primary benefit is organizational, and the cost is distributed-state complexity.

Generalization

Decomposition (services, agents) should be justified by organizational/ownership needs, not assumed performance gains.

Microservices are an architecture of last resort.
Open source video
They are a tool to achieve organizational autonomy and high-degree team independence.
Open source video
They push systems toward a more aggressive distribution of state.
Open source video
Practicelow noveltymoderate evidence

Production behavior is the only reliable ground truth about a system; specifications, documentation, and mental models diverge from reality, so investment should go into observability, chaos engineering, and real-world testing rather than static review.

Why it matters

Establishes an empirical feedback loop as the primary correctness signal, which is essential where formal specs of complex or AI-generated behavior are unreliable.

Generalization

When system behavior is hard to specify, trust runtime evidence over documentation.

Production is truth, and code review alone is insufficient when dealing with AI-generated code.
Open source video
Invest heavily in observability, chaos engineering, and real-world testing rather than relying solely on static code reviews.
Open source video

Deep dives

4

Detecting cognitive surrender before it becomes a production failure

Research question

Which cheap, observable signals (per-subsystem explainability coverage, bus-factor of understanding, defects first found in production per deploy) predict the loss of a shared mental model, and how far do they lead actual production failures?

Why

If the human review layer degrades exactly as generation speed rises, reliability engineering needs a leading indicator rather than waiting for the incident that reveals nobody understood the code.

Cognitive surrender occurs when developers stop understanding how their systems work because AI wrote the code.
Open source video
Production is truth, and code review alone is insufficient when dealing with AI-generated code.
Open source video
Using AI to generate code without deep comprehension leads to systems that nobody on the team fully understands.
Open source video
Source video

Where deduplication belongs: client idempotency keys vs server-side payload fingerprinting

Research question

Under what payload distributions and mutation patterns do server-side fingerprints produce false negatives, and at what measured false-negative rate does retrofitting client-supplied idempotency keys become worth the integration cost?

Why

Retries are routine in agentic and tool-executing systems, so double-execution of side-effecting operations is a first-class hazard; the placement of dedup responsibility determines both retrofit cost and the correctness guarantee you can actually claim.

Idempotency keys require clients to send unique identifiers for operations, which can be painful to retrofit.
Open source video
Fingerprints hash request payloads on the server side to detect duplicate operations automatically.
Open source video
Both approaches have trade-offs regarding false negatives and client integration friction.
Open source video
Source video

Exercising graceful extensibility: can chaos engineering reach unknown-unknowns?

Research question

Do fault-injection suites that generate failures outside the system's declared failure taxonomy produce findings that catalog-driven chaos runs systematically miss, and do those findings survive re-injection?

Why

Uptime-style testing collapses absorbing known faults with adapting to unforeseen ones; if chaos engineering only samples from the anticipated set, teams will believe they have covered a dimension they have never exercised.

Graceful extensibility is handling the unexpected when known defenses fail.
Open source video
Netflix Chaos Monkey turning off parts of the system to test rebound and robustness.
Open source video
Source video

Which defect classes in AI-generated code escape review but surface in production?

Research question

What categories of defects are systematically invisible to human review of AI-generated changes yet detectable by simulation-based or production testing, and what is the cost of the coverage gap?

Why

It converts a general warning about AI code into an actionable test-investment decision: you can only justify simulation infrastructure if you can name the defect classes review is structurally unable to catch.

Production is truth, and code review alone is insufficient when dealing with AI-generated code.
Open source video
Teams must maintain shared mental models of architecture and use simulation-based testing like Antithesis to verify correctness.
Open source video
Invest heavily in observability, chaos engineering, and real-world testing rather than relying solely on static code reviews.
Open source video
Source video

Article ideas

4

Cognitive Surrender Is a Reliability Defect, Not a Culture Problem

If AI-generated code outpaces the team's mental model, code review stops being a correctness gate at all; reliability engineering should therefore treat 'no one can explain this subsystem' as a detectable defect class and shift the correctness budget to simulation and production telemetry rather than to more reviewers.

Angle

Reframe an observed developer-experience phenomenon as an SLO and incident-accounting problem, with concrete places where the quality gate has to move.

Source video

Microservices and Multi-Agent Systems Share the Same Mistake: Decomposing for Scale You Never Had

Decomposition is an organizational instrument, not a performance instrument; services and agents adopted for assumed scaling benefits pay the full cost of aggressively distributed state without collecting the autonomy payoff, so the justification test should be ownership boundaries rather than throughput.

Angle

Transfer the microservices-of-last-resort argument to agent decomposition and give an explicit justification test teams can apply before splitting anything.

Source video

Idempotency Is a Placement Problem, Not a Feature

Retry-safe side effects fail in practice because dedup placement is chosen by convenience rather than by measured tolerance for false negatives; the client-key versus server-fingerprint decision should be made per endpoint against an explicit ambiguity budget.

Angle

Decision framework plus failure taxonomy for duplicate-request handling, including retrofit paths for APIs with clients you cannot change.

Source video

Uptime Is a Vanity Metric: Scoring Systems on Four Resilience Axes

Collapsing resilience into uptime or SLO attainment hides the axis that actually determines survival — handling the unforeseen when known defenses fail — so teams should report robustness, rebound, graceful extensibility, and sustained adaptability separately, because each one funds different engineering work.

Angle

Replace a single reliability number with a four-axis profile and show which remediation decisions change once the axes are separated.

Source video

Project ideas

4

Surrender Probe

beyond-evals

Repositories whose AI-generated subsystems have no human who can explain them will show a higher share of defects first detected in production (rather than pre-merge) per deploy than matched repositories with named subsystem explainers.

Proof of concept

Tool that maps a codebase's subsystems to named explainers, injects a short periodic 'explain this module' probe into PR review, and joins the resulting explainability-coverage score with defect records labeled by discovery stage.

Measurement

Correlation between subsystem explainability coverage and the production-discovered share of defects; lag between a coverage drop and a subsequent change in incident rate.

Source video

Fingerprint Dedup Sidecar

gatehouse

A server-side payload-fingerprint dedup layer retrofitted onto an existing API with uncooperative clients will catch duplicate side-effecting operations above a target recall while keeping false negatives below a threshold set by payload variability.

Proof of concept

Sidecar that canonicalizes and hashes request payloads on side-effecting endpoints, keeps a short-TTL fingerprint ledger, and replays stored responses for detected duplicates; benchmarked head-to-head against a client-supplied idempotency key baseline over a replayed request corpus.

Measurement

Duplicate-detection recall and false-negative rate across realistic payload distributions, plus retrofit cost measured in required client-side changes.

Source video

Unknown-Unknown Fault Injector

new

Fault suites that mutate outside the system's declared failure taxonomy surface a different class of findings than catalog-driven chaos runs, and fixes derived from them remain stable under re-injection.

Proof of concept

Simulation harness that derives a failure taxonomy from the system's own defenses (timeouts, retries, circuit breakers) and injects perturbations covered by no declared defense, with standard chaos-monkey style injection as the control arm on the same service.

Measurement

Count and severity of findings unique to out-of-taxonomy injection versus control, and the fraction of resulting remediations that survive a second injection round.

Source video

Four-Axis Resilience Scorecard

movement-lab

Teams that receive feedback across all four resilience dimensions choose different remediations and improve recovery time more than teams given single-axis uptime/SLO feedback on comparable systems.

Proof of concept

Instrument that derives per-dimension scores from existing signals (incident data, chaos experiment results, configuration drift) and runs a controlled comparison between teams shown a four-axis profile and teams shown a single reliability number.

Measurement

Change in mean recovery time and in the share of remediations targeting graceful extensibility and sustained adaptability rather than only robustness.

Source video

Architectural implications

2

Deduplication responsibility can move from the client to the server via payload fingerprinting.

Before

Clients must be trusted/ modified to generate and pass unique idempotency keys per operation.

After

The server hashes request payloads and detects duplicates autonomously without client changes.

Consequence

Enables retrofitting idempotency onto existing APIs with uncooperative clients, but introduces false-negative detection ambiguity.

Source video

Verification responsibility shifts away from human code review when code is AI-generated.

Before

Code review by engineers who hold a mental model of the system acts as the correctness gate.

After

Simulation-based testing and production telemetry act as the correctness gate because reviewers may no longer understand the code.

Consequence

Teams must invest in simulation infrastructure and shared architectural mental models, changing where quality assurance lives.

Source video

Tradeoffs and failure modes

3

Idempotency keys vs server-side fingerprints

Benefit

Idempotency keys give explicit, unambiguous deduplication control; fingerprints require no client-side integration.

Cost or risk

Idempotency keys are painful to retrofit and demand client cooperation; fingerprints risk false negatives in duplicate detection.

Both approaches have trade-offs regarding false negatives and client integration friction.
Open source video
Source video

Microservices adoption

Benefit

Achieves organizational autonomy and high-degree team independence.

Cost or risk

Pushes the system toward aggressive distribution of state, adding complexity when adopted for assumed scaling benefits.

They push systems toward a more aggressive distribution of state.
Open source video
Source video

AI coding assistants

Benefit

Short-term coding speed and generation of complex code.

Cost or risk

Cognitive debt and cognitive surrender — nobody on the team fully understands the resulting system, and code review alone cannot catch it.

Using AI to generate code without deep comprehension leads to systems that nobody on the team fully understands.
Open source video
Source video

Open questions

3

How can a team detect or measure cognitive surrender before it manifests as a production failure?

Why unresolved

The summary describes the phenomenon but provides no detection mechanism or metric.

Research direction

Define measurable proxies for shared mental-model coverage (e.g., who can explain which subsystem) and correlate with incident rates.

Source video

How effective is server-side payload fingerprinting compared to idempotency keys in practice, especially regarding false negatives?

Why unresolved

The summary states a trade-off exists but gives no empirical comparison or thresholds.

Research direction

Benchmark duplicate-detection accuracy and false-negative rates across realistic request payload distributions.

Source video

How do you systematically test for graceful extensibility (handling the unexpected) when known defenses have failed?

Why unresolved

The framework names the dimension but the summary offers no method for exercising unknown-unknown failures.

Research direction

Explore simulation-based testing (e.g., Antithesis-style) and adversarial/chaos scenarios targeting unanticipated failure modes.

Source video

Key claims

7
opinionVerification not requested

Microservices should be adopted as an architecture of last resort rather than a default for scaling.

Evidence

Microservices are an architecture of last resort.

Source video
causalVerification needed

Idempotency keys are painful to retrofit because they require clients to send unique identifiers.

Evidence

Idempotency keys require clients to send unique identifiers for operations, which can be painful to retrofit.

Question

What integration cost does adding idempotency keys impose on existing clients, and can it be avoided with server-side fingerprinting?

Source video
factualVerification needed

Server-side fingerprinting can automatically detect duplicate operations by hashing request payloads.

Evidence

Fingerprints hash request payloads on the server side to detect duplicate operations automatically.

Question

Under what payload conditions do fingerprint-based dedup methods produce false negatives?

Source video
causalVerification needed

Code review alone is insufficient for AI-generated code because production is the only ground truth.

Evidence

Production is truth, and code review alone is insufficient when dealing with AI-generated code.

Question

What defect classes in AI-generated code escape code review but are caught by simulation-based or production testing?

Source video
causalVerification needed

Cognitive surrender occurs when developers stop understanding their systems because AI wrote the code.

Evidence

Cognitive surrender occurs when developers stop understanding how their systems work because AI wrote the code.

Question

Does reliance on AI code generation measurably reduce developers' ability to reason about their systems' failure modes?

Source video
opinionVerification needed

Simulation-based testing (e.g., Antithesis) is a recommended method to verify correctness of AI-influenced systems.

Evidence

Teams must maintain shared mental models of architecture and use simulation-based testing like Antithesis to verify correctness.

Question

How much defect-detection coverage does simulation-based testing add over conventional test suites for complex systems?

Source video
opinionVerification not requested

The tech world is naive about AI and misunderstands its impact on engineering depth.

Evidence

The tech world is quite naive about AI and fundamentally misunderstands its impact on engineering depth.

Source video

Connections

4