Sequoia Capital · Published 2026-10-06

Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

privateReview: unreviewedConfidence: high
ai infrastructuretpuoptical circuit switchinggoodputpower constraintsai-infrastructureoptical-circuit-switchingpower-constraintshardware-software-co-designfailure-recoverypower-density
Open on YouTube ↗

Summary

Overview

  • Speaker: Amin Vahdat
  • Channel: Sequoia Capital
  • Main topic: AI infrastructure, data center design, hardware-software co-optimization, and the economics and physics of building frontier AI systems.
  • Purpose: To provide technical insights and strategic perspectives on the engineering, economic, and physical challenges of scaling AI infrastructure to meet the demands of frontier models. In this in-depth conversation, Amin Vahdat, head of Google's AI infrastructure, discusses the unprecedented CapEx boom in AI build-outs, the differences between traditional and AI data centers, the physics of power and cooling constraints, and the deep hardware-software co-design required to scale TPU and GPU clusters. Vahdat explores the trade-offs between specialization and generality in hardware, the future of networking with optical circuit switching, and the strategic roadmaps for 2036 and beyond.

Topic Map

AI Data Center Evolution vs Non-AI Data Centers

  • Explanation: Comparing traditional data center architecture with AI-specific infrastructure, highlighting specialization and power density.
  • Key claims:
    • AI data centers share many similarities with non-AI data centers in terms of physical enclosure and electrical yards.
    • The fundamental difference is specialization: AI infrastructure requires purpose-built co-design of hardware, cooling, and power distribution.
    • Storage racks require far less power compared to TPU or GPU racks hitting hundreds of kilowatts.
  • Examples:
    • A storage rack using 10-40 kilowatts versus an AI accelerator rack easily hitting hundreds of kilowatts today and potentially megawatts in the future.
  • Terminology:
    • CapEx
    • AI data center
    • power density
    • accelerator rack
    • co-optimization
  • Why it matters: Understanding infrastructure differences clarifies why standard data center designs fail for modern frontier AI workloads.

Flops vs Goodput and Failure Management at Scale

  • Explanation: Evaluating AI performance metrics beyond raw theoretical FLOPS, focusing on real failure rates and goodput at 100,000+ accelerator scale.
  • Key claims:
    • FLOPS is a vanity metric; goodput and workload performance are what actually matter.
    • At 100,000 accelerator scale, hardware components fail constantly.
    • Detecting, recovering from, and checkpointing failures continuously is like finding a needle in a haystack over hours, days, or weeks.
  • Examples:
    • A single chip failure in a synchronous training job can bring the entire 100,000-chip computation to a halt.
  • Terminology:
    • goodput
    • FLOPS
    • accelerator scale
    • checkpointing
    • failure recovery
  • Why it matters: Scale introduces systemic fragility where hardware reliability and fast checkpoint restoration dictate actual model training speed.

TPU Program Evolution & Co-Design

  • Explanation: Tracing Google's custom silicon journey from 2013 inference chips to modern training clusters and co-design principles.
  • Key claims:
    • Google started custom silicon in 2013 as a contrarian bet against Moore's Law limitations.
    • The first chip focused on inference; the second on training; transformers revolutionized the TPU program.
    • Co-design between hardware and software allows optimization at the level of matrix multiply units and remote memory access.
  • Examples:
    • Developing specialized matrix multiplication units and sparse core accelerators tuned directly to transformer model architectures.
  • Terminology:
    • TPU
    • contrarian bet
    • transformer architecture
    • matrix multiply unit
    • sparse core
  • Why it matters: Demonstrates the power of vertical integration and hardware-software co-design in driving exponential performance gains.

Networking, Optics, and Optical Circuit Switching

  • Explanation: The critical role of high-speed networking, wave division multiplexing, and optical circuit switching in scaling data center infrastructure.
  • Key claims:
    • Traditional electrical packet switching hits scaling limits due to power, wiring density, and impedance mismatch across layers.
    • Optical circuit switching transmits and moves data entirely in the optical domain without electrical conversion.
    • Using MEMS micro-electrical mirrors enables dynamic reconfiguration of network topologies without moving fibers.
  • Examples:
    • Connecting compute and storage clusters using optical circuit switches to create dynamic, reconfigurable network spines.
  • Terminology:
    • wave division multiplexing
    • optical circuit switching
    • MEMS mirrors
    • packet switching
    • network topology
  • Why it matters: Electrical interconnect bottlenecks make optical networking essential for connecting massive distributed AI clusters with low latency and high bandwidth.

Power Constraints, Siting, and Clean Energy

  • Explanation: Addressing the gigawatt-scale power demands of AI data centers and the interplay with global energy grids and clean energy sources.
  • Key claims:
    • Power is the single most fundamental constraint in scaling AI data center infrastructure.
    • A gigawatt-scale data center requires massive regional power capacity, clean energy sourcing, and grid co-optimization.
    • Utility-connected infrastructure and regional statistical multiplexing help balance power demand.
  • Examples:
    • Partnering with utilities to co-locate training clusters near clean energy generation or balancing power draw with regional demand.
  • Terminology:
    • gigawatt-scale
    • clean energy
    • statistical multiplexing
    • grid integration
    • power availability
  • Why it matters: Energy availability and sustainability are the ultimate physical bottlenecks governing the physical footprint of future AI advances.

Key Points

Hardware-Software Co-Design

  • Explanation: Optimizing both compiler/runtime software and custom silicon together yields multiplicative performance benefits.
  • Evidence: Google's decade-long co-development of TPUs alongside TensorFlow, JAX, and Gemini models.
  • Practical implication: Companies relying solely on commodity hardware risk missing major efficiency multipliers achievable through vertical integration.

Goodput vs FLOPS

  • Explanation: Raw theoretical compute capacity is meaningless if system failures, networking bottlenecks, and checkpointing overhead stall execution.
  • Evidence: At 100,000+ accelerator scale, component failures happen constantly, making fault tolerance and rapid recovery paramount.
  • Practical implication: Infrastructure engineers must optimize for sustained end-to-end goodput rather than peak benchmark FLOPS.

Optical Networking Transition

  • Explanation: Electrical packet switching cannot scale indefinitely due to physical impedance and wiring limits; optical circuit switching is mandatory.
  • Evidence: Google's deployment of optical circuit switching and MEMS mirror technology for inter-rack communication.
  • Practical implication: Future data center architectures will rely heavily on optical interconnects to reduce latency and power consumption.

Frameworks, Models & Processes

The Co-Optimization Stack

  • How it works: Simultaneously designing hardware architecture, compilation software, model topology, and data center facilities to eliminate bottlenecks.
  • Components:
    • Custom Silicon (TPUs/GPUs)
    • Compilers and Runtimes
    • Model Architecture (Transformers)
    • Networking and Optical Switching
    • Power and Cooling Facilities
  • When to use: When building hyperscale AI infrastructure where general-purpose hardware limits performance and energy efficiency.

Examples & Case Studies

Google's 2013 decision to build custom inference silicon (TPU v1) despite prevailing industry wisdom against custom ASICs.

  • Illustrates: The value of taking contrarian bets on specialized hardware to bypass Moore's Law slowdowns.
  • Lesson: Specialized co-designed hardware can unlock orders-of-magnitude improvements for specific high-demand workloads like deep learning.

Deploying optical circuit switching with MEMS micro-mirrors to dynamically reconfigure data center network spines.

  • Illustrates: Overcoming electrical packet switching bottlenecks and transceiver power limits in hyperscale clusters.
  • Lesson: Shifting from electrical to optical routing enables massive bandwidth scaling without physical recabling.

Actionable Takeaways

  • Immediate:
    • Prioritize goodput and system resilience over peak theoretical FLOPS.
    • Invest in hardware-software co-design to maximize AI workload efficiency.
    • Incorporate optical networking technologies to alleviate intra-datacenter bandwidth bottlenecks.
  • Strategic:
    • Plan data center expansions around long-term power availability and clean energy integration.
    • Maintain flexibility by supporting open standards alongside proprietary hardware stacks.
    • Anticipate gigawatt-scale power requirements by partnering early with energy utilities.
  • Questions to investigate:
    • How will next-generation optical circuit switching replace traditional electrical switching at scale?
    • What are the most effective strategies for managing gigawatt-scale power distribution and thermal dissipation?
    • How can software abstraction layers maintain performance across heterogeneous accelerator architectures?

Claims Worth Verifying

  • Google was among the first to introduce wave division multiplexing and optical circuit switching within data centers. (historical technological fact)
  • Power is the single most fundamental constraint facing frontier AI infrastructure expansion over the next decade. (industry projection)

Notable Quotes

"In hardware, again, as you know, there is this opportunity where the more you specialize to a particular workload, the less flexible it is, the faster, the more power efficient the hardware is going to be." (at 0:00) "Power is the single most fundamental constraint that we face." (at 71:42)

Compressed Summary

  • AI infrastructure requires deep hardware-software co-design and vertical integration.
  • Goodput and failure recovery matter far more at scale than peak theoretical FLOPS.
  • Optical circuit switching is essential to bypass electrical networking bottlenecks.
  • Power availability and thermal constraints are the ultimate physical limits for future AI data centers.
  • Keywords: ai infrastructure, tpu, optical circuit switching, goodput, power constraints
  • Core insight: Scaling frontier AI requires overcoming severe physical and economic constraints in power, thermal management, and optical networking through relentless hardware-software co-optimization.

Core insights

6
Failure Modemedium noveltystrong evidence

Optimize for sustained goodput, not peak FLOPS: at 100,000+ accelerator scale, a single component failure synchronously halts the entire training job, so the real throughput ceiling is set by failure detection, checkpointing, and restore speed rather than raw compute.

Why it matters

It relocates the dominant cost of large-scale compute from arithmetic to fault handling, meaning reliability engineering and checkpoint/restore mechanics — not more chips — determine effective training speed, and the same logic applies to any system where many workers must synchronize on a barrier.

Generalization

In any tightly-synchronized fan-out system, goodput degrades with failure rate times worker count; resilience machinery (fast detection, cheap restart, frequent checkpointing) becomes a first-class performance feature rather than an operational afterthought.

FLOPS is a vanity metric; goodput and workload performance are what actually matter.
Open source video
A single chip failure in a synchronous training job can bring the entire 100,000-chip computation to a halt.
Open source video
Detecting, recovering from, and checkpointing failures continuously is like finding a needle in a haystack over hours, days, or weeks.
Open source video
Architecturehigh noveltymoderate evidence

Electrical packet switching is hitting hard physical ceilings (power, wiring density, impedance mismatch across layers), so inter-rack connectivity moves to optical circuit switching where data is transmitted and switched entirely in the optical domain with no electrical conversion, and MEMS mirrors reconfigure the topology dynamically without physically moving fibers.

Why it matters

It changes what the network is: from a fixed-topology packet fabric to a software-reconfigurable circuit layer, shifting responsibility for placement, topology selection, and congestion control into the runtime/scheduler.

Generalization

When a layer's per-unit cost scales worse than the workload's demand, the layer is replaced by a differently-physicalized substrate — and the new substrate's reconfiguration semantics become a new scheduling constraint software must manage.

Traditional electrical packet switching hits scaling limits due to power, wiring density, and impedance mismatch across layers.
Open source video
Optical circuit switching transmits and moves data entirely in the optical domain without electrical conversion.
Open source video
Using MEMS micro-electrical mirrors enables dynamic reconfiguration of network topologies without moving fibers.
Open source video
Mental Modelmedium noveltymoderate evidence

Power is framed as the binding physical constraint: gigawatt-scale sites require regional power capacity, clean-energy sourcing, and grid co-optimization, with regional statistical multiplexing used to balance draw rather than treating demand as a fixed point load.

Why it matters

It makes energy availability and temporal demand shaping part of the system design surface (siting, scheduling, workload shifting) rather than a procurement afterthought — the same pattern applies to any resource-constrained serving fleet.

Generalization

When a resource is capacity-constrained and its supply is variable, the architectural move is to design workload scheduling against supply variability (statistical multiplexing, deferral, co-location), not just to provision for peak.

Power is the single most fundamental constraint in scaling AI data center infrastructure.
Open source video
Utility-connected infrastructure and regional statistical multiplexing help balance power demand.
Open source video
Mechanismmedium noveltymoderate evidence

Co-design operates below the ISA: hardware and software are jointly optimized at the granularity of matrix-multiply units and remote memory access, with sparse-core accelerators shaped directly by transformer architectures.

Why it matters

It shows the abstraction boundary moving — instead of stable general-purpose primitives, the interface itself is tuned per model family, which is the only way to capture multiplicative (not incremental) efficiency gains.

Generalization

Multiplicative efficiency comes from co-optimizing across an abstraction boundary rather than within one layer; a stack that owns both sides of an interface can specialize the interface itself.

Co-design between hardware and software allows optimization at the level of matrix multiply units and remote memory access.
Open source video
Developing specialized matrix multiplication units and sparse core accelerators tuned directly to transformer model architectures.
Open source video
Predictionmedium noveltymoderate evidence

Accelerator rack power density is on a trajectory from 10–40 kW storage racks to hundreds of kilowatts today and potentially megawatts, which invalidates standard data-center cooling and power-distribution designs.

Why it matters

Power density per unit footprint becomes a primary architectural constraint on hardware packaging and facility design, with second-order effects on interconnect (optingics) and deployment topology.

Generalization

When a single component's power density grows by 1–2 orders of magnitude, facility-level assumptions (cooling medium, distribution, placement) stop scaling smoothly and force a discontinuously different design.

A storage rack using 10-40 kilowatts versus an AI accelerator rack easily hitting hundreds of kilowatts today and potentially megawatts in the future.
Open source video
The fundamental difference is specialization: AI infrastructure requires purpose-built co-design of hardware, cooling, and power distribution.
Open source video
Practicemedium noveltymoderate evidence

Google's custom silicon program began in 2013 as an explicitly contrarian bet against Moore's Law limits, starting with inference and only later training — sequencing specialization to the workload's maturity rather than to the workload's glamour.

Why it matters

It suggests a concrete ordering principle for building specialized infrastructure: target the most stable, highest-volume workload first and expand specialization as the dominant model architecture becomes known.

Generalization

Specialization should be sequenced by workload stability and volume, not by perceived importance; betting early on a still-shifting target risks an accelerator tuned to a transient architecture.

Google started custom silicon in 2013 as a contrarian bet against Moore's Law limitations.
Open source video
The first chip focused on inference; the second on training; transformers revolutionized the TPU program.
Open source video

Deep dives

4

Goodput and failure management at 100,000+ accelerator scale

Research question

How does sustained goodput degrade with cluster size due to failure and recovery overhead, and what checkpointing and failure-detection strategies maximize it?

Why

It shifts the optimization target from raw compute to resilience engineering, which determines real training speed and cost at scale.

FLOPS is a vanity metric; goodput and workload performance are what actually matter.
Open source video
A single chip failure in a synchronous training job can bring the entire 100,000-chip computation to a halt.
Open source video
Detecting, recovering from, and checkpointing failures continuously is like finding a needle in a haystack over hours, days, or weeks.
Open source video
Source video

Optical circuit switching for reconfigurable data center networks

Research question

What reconfiguration latencies and traffic patterns make optical circuit switching superior to electrical packet switching, and how should schedulers exploit dynamic topology?

Why

It redefines the network as a software-controlled resource, introducing new scheduling and control-plane challenges.

the future of networking with optical circuit switching
Open source video
optical circuit switching
Open source video
Source video

Power as the binding constraint and regional statistical multiplexing

Research question

Can regional statistical multiplexing of power demand absorb gigawatt-scale, correlated training workloads without additional peaking capacity?

Why

It makes energy availability and temporal demand shaping part of system design, affecting siting and scheduling.

the physics of power and cooling constraints
Open source video
power constraints
Open source video
Source video

Hardware-software co-design at the matrix-multiply and memory-access level

Research question

What efficiency multipliers are achievable by co-designing matrix-multiply units and sparse cores directly to transformer architectures, and what portability costs does this incur?

Why

It captures multiplicative gains by moving the abstraction boundary, but creates specialization risk as architectures evolve.

hardware-software co-optimization
Open source video
the deep hardware-software co-design required to scale TPU and GPU clusters
Open source video
AI infrastructure requires purpose-built co-design of hardware, cooling, and power distribution.
Open source video
Source video

Article ideas

4

Goodput, Not FLOPS: The Real Constraint on Frontier AI

The AI industry's focus on peak FLOPS is misleading; the true bottleneck is sustained goodput, which is governed by failure recovery and checkpointing at scale, not raw compute.

Angle

Argue that infrastructure investments should prioritize resilience engineering and fast checkpoint/restore over more accelerators.

Source video

When the Network Becomes a Scheduler: Optical Circuit Switching and the Future of AI Data Centers

The shift to optical circuit switching turns network topology from a fixed design artifact into a runtime-controlled resource, forcing schedulers to co-plan placement and reconfiguration.

Angle

Explain the physical limits of electrical packet switching and the new control-plane responsibilities introduced by optical switching.

Source video

Power Is the New Moore's Law: Why AI Infrastructure Is a Grid Problem

As power becomes the binding constraint, AI scaling depends more on regional energy availability and demand shaping than on chip performance.

Angle

Analyze the implications of gigawatt-scale data centers for siting, clean energy sourcing, and statistical multiplexing.

Source video

Specialization's Sequencing: What Google's TPU Program Teaches About Betting on Workloads

Effective hardware specialization follows workload maturity—start with the most stable, high-volume workload (inference) before expanding to training and architecture-specific units.

Angle

Derive a general principle for sequencing specialization from Google's 2013 inference-first bet.

Source video

Project ideas

4

GoodputLens

new

At 100,000+ accelerator scale, sustained goodput is inversely proportional to failure rate and recovery time, and can be improved by 30% through adaptive checkpointing and failure-domain isolation rather than adding more accelerators.

Proof of concept

Build an instrumentation layer that measures time-to-detect, time-to-restore, and checkpoint overhead in a simulated or real cluster of 10,000+ GPUs/TPUs, and implement adaptive checkpointing policies.

Measurement

Goodput as a fraction of peak FLOPS, decomposed into failure, restart, and stall components, compared across static vs adaptive checkpointing.

Source video

CircuitSched

new

A topology-aware scheduler that co-plans optical circuit reconfiguration with job placement can improve training goodput by >15% compared to static topologies for bursty training workloads.

Proof of concept

Simulate a reconfigurable optical network with MEMS mirrors and implement a scheduler that decides when and how to reconfigure circuits based on job communication patterns.

Measurement

Net goodput improvement, reconfiguration cost amortization, and latency impact under training, inference, and storage traffic mixes.

Source video

PowerMux

new

Regional statistical multiplexing can flatten aggregate power demand for gigawatt-scale AI data centers even with correlated training workloads, by exploiting temporal deferral and demand shaping.

Proof of concept

Build a simulator that models power demand profiles of co-located training fleets and evaluates multiplexing strategies against grid capacity constraints.

Measurement

Peak-to-average ratio reduction, required peaking capacity, and feasibility of clean energy sourcing.

Source video

CoDesignBench

new

Narrow co-design surfaces (kernel fusion, memory layout, runtime scheduling) on commodity accelerators can capture at least 50% of the efficiency gains of vertically integrated co-designed silicon for transformer training.

Proof of concept

Implement a set of co-designed optimizations on commodity GPUs and measure training throughput and energy efficiency against a baseline and against published TPU performance.

Measurement

Efficiency multiple (throughput per watt) relative to baseline and to vertically integrated systems.

Source video

Architectural implications

3

Failure detection, checkpointing, and recovery are described as continuous infrastructure concerns at 100k+ chip scale, not application-level error handling.

Before

Checkpointing and restart logic live in the training application or framework, tuned per job.

After

Recovery becomes a platform-level primitive with its own observability, provisioning, and performance budget, since every workload benefits and every workload pays the cost.

Consequence

Platform teams need goodput-style SLIs (time-to-detect, time-to-restore, effective utilization) rather than hardware availability percentages.

Source video

Network topology becomes dynamically reconfigurable via MEMS mirrors rather than fixed cabling.

Before

Topology is a static design-time artifact; the scheduler places work onto a fixed fabric.

After

Topology is a runtime-controlled resource the scheduler and placement layer can reshape.

Consequence

New control-plane concerns appear: how often to reconfigure, who owns the reconfiguration decision, and how reconfiguration latency interacts with job scheduling.

Source video

Hardware/software optimization reaches down to matrix-multiply units, sparse cores, and remote memory access.

Before

Software targets a stable ISA/runtime contract; hardware is a general-purpose substrate.

After

The interface itself is co-designed against a specific model family's compute and memory-access pattern.

Consequence

Portability becomes a deliberate cost; teams must decide which layers to specialize and how to keep a fallback path when architectures shift.

Source video

Tradeoffs and failure modes

5

Synchronous training at extreme scale

Benefit

Aggregating 100,000+ accelerators yields computation no smaller cluster can perform.

Cost or risk

Any single component failure halts the entire job, and recovery is characterized as a needle-in-a-haystack problem spanning hours to weeks.

A single chip failure in a synchronous training job can bring the entire 100,000-chip computation to a halt.
Open source video
Source video

Optical circuit switching vs electrical packet switching

Benefit

Avoids electrical conversion and the power, wiring-density, and impedance limits of packet switching, enabling dynamic spine reconfiguration without moving fibers.

Cost or risk

Introduces a physical reconfiguration mechanism (MEMS mirrors) and topology control that must be managed, rather than a transparent packet fabric.

Using MEMS micro-electrical mirrors enables dynamic reconfiguration of network topologies without moving fibers.
Open source video
Source video

Vertical integration of silicon and software

Benefit

Captures multiplicative efficiency gains from co-designed matrix-multiply units, sparse cores, and remote memory access.

Cost or risk

Requires a long, contrarian commitment made before the dominant workload is known; commodity-hardware shops forgo the multipliers but avoid the bet.

Companies relying solely on commodity hardware risk missing major efficiency multipliers achievable through vertical integration.
Open source video
Source video

Specialization vs generality in accelerator design

Benefit

Specialized units tuned to transformer architectures deliver large efficiency gains.

Cost or risk

Specialization is a bet on the durability of the target architecture; the summary notes that transformers, not the original design intent, ended up reshaping the TPU program.

The first chip focused on inference; the second on training; transformers revolutionized the TPU program.
Open source video
Source video

Gigawatt-scale siting near clean energy

Benefit

Co-locating training near clean generation and balancing draw regionally can make enormous loads feasible.

Cost or risk

Ties infrastructure to regional grid capacity and utility coordination, making deployment rate depend on external power availability rather than hardware supply alone.

A gigawatt-scale data center requires massive regional power capacity, clean energy sourcing, and grid co-optimization.
Open source video
Source video

Open questions

5

What are the measured time-to-detect, time-to-restore, and checkpoint overhead at 100,000+ accelerator scale, and how does the resulting goodput fraction scale as clusters grow?

Why unresolved

The summary asserts that failures happen constantly and recovery is hard, but provides no failure rates, recovery times, or checkpoint frequencies.

Research direction

Instrument large training clusters to report goodput as a fraction of peak, decomposed into failure, restart, and stall components, and model how that fraction changes with cluster size.

Source video

For which traffic patterns does optical circuit switching beat electrical packet switching, given circuit reconfiguration latency versus burstiness?

Why unresolved

The summary states the physical limits of packet switching and the mechanism of optical switching, but gives no reconfiguration latency, bandwidth, or workload mix where each wins.

Research direction

Benchmark reconfigurable optical topologies against packet fabrics under training, inference, and storage traffic mixes, measuring reconfiguration cost amortization.

Source video

Does regional statistical multiplexing of power demand actually hold for gigawatt-scale, correlated training workloads, or does synchronization drive correlated peaks that defeat multiplexing?

Why unresolved

The summary lists statistical multiplexing as a balancing approach without evidence about correlation between training jobs' power profiles.

Research direction

Measure temporal power-demand correlation across co-located training fleets and simulate whether deferral and demand shaping flatten aggregate load.

Source video

Which co-design principles transfer to teams that do not own their silicon, model, and compiler stack?

Why unresolved

The summary attributes multiplicative gains to vertical integration without describing a partial-adoption path for commodity-hardware users.

Research direction

Test whether narrower co-design surfaces (kernel fusion, memory layout, runtime scheduling) capture a meaningful fraction of vertically integrated gains on commodity accelerators.

Source video

How should a scheduler treat a reconfigurable optical topology — as a static resource to be partitioned, or as a dynamic resource to be time-shared?

Why unresolved

The summary describes dynamic reconfiguration as a capability but not the control policy or its interaction with job placement.

Research direction

Prototype topology-aware schedulers that co-plan circuit reconfiguration with job placement and measure the net goodput effect.

Source video

Key claims

7
opinionVerification needed

FLOPS is a vanity metric and goodput/workload performance is what actually matters.

Evidence

FLOPS is a vanity metric; goodput and workload performance are what actually matter.

Question

Does goodput (useful work per unit time) correlate weakly with peak FLOPS across large training and serving workloads, and by how much do they diverge at scale?

Source video
factualVerification needed

At 100,000 accelerators, hardware components fail constantly, so continuous detection, recovery, and checkpointing are required.

Evidence

At 100,000 accelerator scale, hardware components fail constantly.

Question

What is the observed MTBF per component and per job at 100,000-accelerator scale, and what fraction of wall-clock time is lost to failures and restarts?

Source video
causalVerification needed

A single chip failure in a synchronous training job halts the entire 100,000-chip computation.

Evidence

A single chip failure in a synchronous training job can bring the entire 100,000-chip computation to a halt.

Question

Is collective synchronization across the whole job actually required, or can failure domains be isolated by model/optimizer state partitioning?

Source video
causalVerification needed

Traditional electrical packet switching hits scaling limits due to power, wiring density, and impedance mismatch across layers.

Evidence

Traditional electrical packet switching hits scaling limits due to power, wiring density, and impedance mismatch across layers.

Question

At what port count and bandwidth per link do electrical packet switches become power- or density-infeasible relative to optical circuit switches?

Source video
factualVerification needed

Google's custom silicon effort started in 2013 as a contrarian bet against Moore's Law limitations.

Evidence

Google started custom silicon in 2013 as a contrarian bet against Moore's Law limitations.

Question

What was the projected performance-per-watt trajectory of general-purpose CPUs/GPUs at the time that justified a custom-silicon bet?

Source video
causalVerification needed

Utility-connected infrastructure and regional statistical multiplexing help balance AI power demand.

Evidence

Utility-connected infrastructure and regional statistical multiplexing help balance power demand.

Question

To what extent can regional demand multiplexing absorb gigawatt-scale correlated training load without additional peaking capacity?

Source video
comparativeVerification needed

Companies relying solely on commodity hardware risk missing major efficiency multipliers achievable through vertical integration.

Evidence

Companies relying solely on commodity hardware risk missing major efficiency multipliers achievable through vertical integration.

Question

What is the measured efficiency multiple of co-designed silicon plus compiler over commodity accelerators for equivalent model training or inference workloads?

Source video

Connections

5