GOTO Conferences · Published 2026-08-20

Modern Computer Architecture & Organization • Jim Ledin & Maciej «MJ» Jedrzejewski • GOTO 2026

privateReview: unreviewedConfidence: high
computer architectureGPUsllmsmemory bandwidthassemblycomputer-architecturememory-bandwidthhbmgpu-architecturetpuhardware-software-codesigntransformer-architecturedatacenter-scale
Open on YouTube ↗

Summary

Overview

  • Speaker: Jim Ledin & Maciej «MJ» Jedrzejewski
  • Channel: GOTO Conferences
  • Main topic: Modern Computer Architecture, Organization, GPUs, and Large Language Models
  • Purpose: To explore modern computer architecture principles, hardware-software co-design, GPU internals, tensor processing, and memory bandwidth challenges in the era of LLMs. Jim Ledin discusses the core themes of his book 'Modern Computer Architecture and Organization' (3rd edition), highlighting the evolution of hardware from the Commodore 64 to modern GPUs and TPUs, the role of assembly and hardware understanding in software engineering, and the memory bandwidth bottlenecks powering modern AI and large language models.

Topic Map

Introduction and Commodore 64 origins

  • Explanation: Jim Ledin shares his background in embedded systems, assembly, and how learning Commodore 64 BASIC and 6502 assembly code shaped his career.
  • Key claims:
    • Early exposure to hardware assembly via Commodore 64 inspired a lifelong interest in computer architecture.
    • Understanding low-level hardware informs better high-level software engineering decisions.
  • Examples:
    • Writing a Commodore 64 joystick screen-drawing program in BASIC, then speeding it up drastically with 6502 assembly language.
  • Terminology:
    • commodore 64
    • 6502 assembly
    • embedded systems
    • hardware-software boundary
  • Why it matters: Demonstrates how low-level hardware curiosity translates into deep systems engineering intuition.

GPU and LLM Architecture in Book 3rd Edition

  • Explanation: Discussion of the new chapters added to the third edition covering GPUs and Large Language Models (LLMs).
  • Key claims:
    • GPU and LLM architectures evolve rapidly, yet fundamental hardware principles remain stable.
    • Understanding transformer architecture mechanics helps developers reason about AI tooling as a mechanic.
  • Examples:
    • Analyzing NVIDIA GPUs and the transformer architecture underlying models like GPT-2.
  • Terminology:
    • gpu
    • llm
    • transformer architecture
    • tensor processing unit
  • Why it matters: Bridges traditional computer architecture with modern artificial intelligence hardware requirements.

Why GPUs Became AI Hardware by Accident

  • Explanation: Exploration of how graphics processing units designed for video games became the primary hardware backbone for AI workloads.
  • Key claims:
    • GPUs were not originally built for neural networks, but share massive matrix multiplication requirements with graphics rendering.
    • Neural networks rely heavily on tensor multiplications and parallel matrix operations.
  • Examples:
    • Comparing 3D rendering lighting calculations to artificial neural network weight multiplications.
  • Terminology:
    • matrix multiplication
    • tensor
    • parallel processing
    • graphics pipeline
  • Why it matters: Explains the historical accident of silicon design convergence that powered the AI boom.

Training vs. Inference: Same Chip, Different Job

  • Explanation: Comparing the computational workflows of training large language models versus running inference.
  • Key claims:
    • Training and inference share significant overlap, but training requires feedback loops to adjust neural weights.
    • TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.
  • Examples:
    • Feeding forward inference results back into neural network weights during training backpropagation.
  • Terminology:
    • training
    • inference
    • backpropagation
    • tpu
  • Why it matters: Clarifies the hardware workload differences between developing AI models and serving them to users.

Data Center as the Computer and Memory Bandwidth Bottlenecks

  • Explanation: Analyzing data center scale compute demands, high-bandwidth memory (HBM), and the primary hardware bottlenecks in modern systems.
  • Key claims:
    • At massive scale, the data center functions as a single giant computer.
    • Memory bandwidth—specifically the speed of reading and writing data to memory—is the primary bottleneck for LLMs, not raw clock speed.
  • Examples:
    • Comparing DDR5 RAM price increases over a year alongside the adoption of stacked High Bandwidth Memory (HBM).
  • Terminology:
    • high bandwidth memory
    • hbm
    • ddr5
    • cache hierarchy
    • pipeline
  • Why it matters: Identifies memory constraints as the primary physical and economic limitation scaling AI systems.

Key Points

Low-level knowledge empowers high-level software engineering

  • Explanation: Knowing how underlying processors, pipelines, and caches execute instructions allows developers to write significantly more efficient code.
  • Evidence: Assembly language optimization experience on early home computers.
  • Practical implication: Software engineers should understand hardware execution models to optimize application performance.

Memory bandwidth is the real AI bottleneck

  • Explanation: As large language models scale with billions of parameters, moving tensor data in and out of memory limits performance more than arithmetic compute speed.
  • Evidence: Rising costs and architectural shifts toward High Bandwidth Memory (HBM) in server-grade hardware.
  • Practical implication: System architects must prioritize memory bandwidth and caching strategies when deploying high-scale AI infrastructure.

Frameworks, Models & Processes

Hardware-Software Co-Design Principles

  • How it works: Aligning software algorithms and data structures with underlying CPU/GPU pipeline and cache hierarchies to maximize execution speed.
  • Components:
    • cache levels
    • pipeline execution
    • memory bandwidth
    • parallelism
  • When to use: When optimizing high-performance applications, embedded systems, or large-scale AI workloads.

Examples & Case Studies

Jim Ledin writing a Commodore 64 BASIC joystick screen-drawing program that ran extremely slowly, prompting him to learn 6502 assembly code.

  • Illustrates: The performance difference between high-level interpreted code and low-level assembly execution.
  • Lesson: Understanding hardware limits transforms how engineers approach software performance.

Actionable Takeaways

  • Immediate:
    • Understand how memory caches and instruction pipelines impact software performance.
    • Recognize the distinction between GPU workloads for graphics versus tensor operations for AI.
  • Strategic:
    • Account for memory bandwidth limitations when architecting large-scale data center applications.
    • Bridge the gap between software engineering and hardware architecture.
  • Questions to investigate:
    • How will future memory architectures overcome the HBM bandwidth bottleneck?
    • What new hardware paradigms will emerge beyond traditional transformer accelerators?

Claims Worth Verifying

  • 80 to 90 percent of AI model processing comes down to matrix multiplications. (technical)
  • High Bandwidth Memory (HBM) stacks 16 or more memory chips inside a single integrated circuit package. (hardware engineering)

Notable Quotes

"Rather than focusing too much on specific products or language models or the current state of the art, it's really concentrated more on the architectural principles underlying all of that." "The data center is really a giant computer with an architecture that is critical that it be done in a way that maximizes the value and minimizes the costs."

Compressed Summary

  • Hardware-software boundaries shape software performance.
  • GPUs became AI engines due to shared matrix multiplication needs.
  • Memory bandwidth is the primary bottleneck for large language models.
  • Keywords: computer architecture, gpus, llms, memory bandwidth, assembly
  • Core insight: Memory bandwidth limitations, rather than raw compute speed, constitute the fundamental bottleneck in modern computer architecture and AI scalability.

Core insights

6
Tradeoffmedium noveltymoderate evidence

For LLM workloads, memory bandwidth (rate of reading/writing tensor data to memory) — not raw clock/compute speed — is the dominant scaling bottleneck, and the industry response is architecturally visible in the shift to stacked High Bandwidth Memory.

Why it matters

Redefines the optimization target for AI infrastructure: FLOPS-focused sizing decisions misallocate capital if the real constraint is data movement between memory and compute units.

Generalization

For any data-intensive workload (inference serving, vector search, feature stores), optimize for data movement across the memory hierarchy before optimizing arithmetic throughput.

Memory bandwidth—specifically the speed of reading and writing data to memory—is the primary bottleneck for LLMs, not raw clock speed.
Open source video
moving tensor data in and out of memory limits performance more than arithmetic compute speed.
Open source video
Mental Modelmedium noveltymoderate evidence

GPU dominance in AI is a historical accident of silicon convergence: the matrix/tensor multiplication primitives required by neural networks overlap with those required by 3D graphics rendering, so gaming hardware became AI hardware without being designed for it.

Why it matters

Explains why current AI infrastructure has properties (SIMT parallelism, memory hierarchy, precision formats) inherited from graphics rather than chosen for learning, which predicts where current accelerators will be awkward.

Generalization

When evaluating whether an existing hardware or runtime substrate can absorb a new workload class, look for shared primitive operations rather than shared domain purpose.

GPUs were not originally built for neural networks, but share massive matrix multiplication requirements with graphics rendering.
Open source video
Comparing 3D rendering lighting calculations to artificial neural network weight multiplications.
Open source video
Architecturelow noveltymoderate evidence

TPUs are a specialization bet: strip out general-purpose compute and dedicate silicon to tensor multiplication to eliminate wasted cycles, in contrast to repurposed GPUs that carry graphics-legacy generality.

Why it matters

Frames the accelerator design axis as generality vs. tensor-op density, which determines which model families a given chip will serve efficiently as architectures change.

Generalization

Specialized execution substrates (TPU/ASIC, or in software, narrow runtimes) buy efficiency at the cost of being brittle to workload drift; generality is insurance against architecture churn.

TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.
Open source video
Architecturemedium noveltyweak evidence

At sufficient scale the unit of architecture stops being the server and becomes the data center itself — treated as a single giant computer.

Why it matters

Moves the abstraction boundary for orchestration, scheduling, and fault handling from single-machine concerns to datacenter-scale concerns, which is where AI training and serving economics actually live.

Generalization

When a workload exceeds a single node, the correct abstraction level shifts one tier up; designs that keep node-level boundaries become the bottleneck.

At massive scale, the data center functions as a single giant computer.
Open source video
Mechanismlow noveltymoderate evidence

Training and inference run on largely the same hardware but differ semantically: training closes a feedback loop that feeds results back to adjust weights, whereas inference does not.

Why it matters

Same-chip claims mask divergent operational requirements — training is bandwidth- and synchronization-heavy with iterative weight updates; inference is latency- and throughput-sensitive. Infrastructure sizing must treat them as separate workload classes.

Generalization

Shared silicon does not imply shared operational profile; separate the workload contract (feedback vs. feed-forward) from the execution substrate when designing capacity and scheduling.

Training and inference share significant overlap, but training requires feedback loops to adjust neural weights.
Open source video
Feeding forward inference results back into neural network weights during training backpropagation.
Open source video
Practicelow noveltymoderate evidence

Hardware-software co-design is framed as aligning algorithms and data structures to the actual pipeline and cache hierarchy, rather than treating the hardware as an opaque execution engine.

Why it matters

Gives a concrete optimization method (match data layout and access patterns to cache levels and pipeline behavior) rather than generic 'write efficient code' advice, and is testable via cache-aware benchmark comparisons.

Generalization

Performance engineering generalizes to any layered system: align the producer's data structures to the consumer's access hierarchy instead of optimizing each layer in isolation.

Aligning software algorithms and data structures with underlying CPU/GPU pipeline and cache hierarchies to maximize execution speed.
Open source video
Understanding low-level hardware informs better high-level software engineering decisions.
Open source video

Deep dives

5

Memory-bound vs compute-bound crossover in LLM workloads

Research question

At what batch sizes, sequence lengths, and precisions do LLM training and inference transition from being memory-bandwidth-bound to compute-bound, and how does that crossover differ between the two?

Why

Informs whether to invest in HBM capacity/bandwidth or in compute FLOPs, and predicts where accelerator designs will be bottlenecked.

Memory bandwidth—specifically the speed of reading and writing data to memory—is the primary bottleneck for LLMs, not raw clock speed.
Open source video
moving tensor data in and out of memory limits performance more than arithmetic compute speed.
Open source video
Source video

Mapping graphics pipeline primitives to tensor operations

Research question

Which specific 3D graphics pipeline operations (e.g., lighting, transformation, shading) map onto the same matrix multiplication and tensor primitives used in neural network training and inference?

Why

Determines which GPU architectural features are essential for AI and which are legacy graphics overhead, guiding future accelerator design and software optimization.

GPUs were not originally built for neural networks, but share massive matrix multiplication requirements with graphics rendering.
Open source video
Comparing 3D rendering lighting calculations to artificial neural network weight multiplications.
Open source video
Source video

TPU specialization brittleness under model architecture drift

Research question

How does the efficiency advantage of tensor-specialized accelerators (TPU-style) degrade when model architectures shift away from dense transformer matrix multiplications (e.g., to sparse, mixture-of-experts, or state-space models)?

Why

Quantifies the risk of specialization bets and informs whether to invest in general-purpose GPUs or ASICs for a given model roadmap.

TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.
Open source video
Source video

Separating training and inference workload contracts on shared silicon

Research question

How should capacity planning, batching, and memory strategies differ between training (feedback loop, weight updates) and inference (feed-forward, latency-sensitive) when using the same accelerator hardware?

Why

Prevents mis-sized clusters and poor utilization from assuming one profile fits both.

Training and inference share significant overlap, but training requires feedback loops to adjust neural weights.
Open source video
Feeding forward inference results back into neural network weights during training backpropagation.
Open source video
Source video

Datacenter as a single computer: abstraction boundary consequences

Research question

What are the concrete engineering consequences (scheduling, fault tolerance, data movement) of treating the datacenter as a single computer, and where do node-local optimizations break down?

Why

Provides actionable guidance for infrastructure design at scale, where node-level thinking becomes the bottleneck.

At massive scale, the data center functions as a single giant computer.
Open source video
At datacenter scale, inter-node and memory data movement rather than per-node clock speed determines system-level performance.
Open source video
Source video

Article ideas

4

The FLOPS Fallacy: Why Memory Bandwidth, Not Compute, Will Determine the Next Decade of AI Infrastructure

The AI industry's obsession with FLOPS is misdirected; memory bandwidth is the binding constraint, and the firms that optimize data movement across the memory hierarchy will win, while those that buy compute for its own sake will waste capital.

Angle

Argues that the HBM supply chain, packaging, and bandwidth-per-watt are the new strategic bottlenecks, and provides a roofline-based decision framework for accelerator procurement.

Source video

GPUs Didn't Win AI: Matrix Multiplication Did

GPU dominance in AI is a historical accident of shared primitives, not superior design for learning; understanding this explains current accelerator limitations and predicts that future AI hardware will diverge from graphics-derived architectures.

Angle

Traces the graphics-to-AI pipeline convergence, identifies which GPU features are load-bearing for AI versus vestigial, and forecasts where new architectures (TPU, wafer-scale, in-memory compute) will break away.

Source video

The Specialization Tax: Why TPUs and Other AI ASICs Are Brittle Bets

Tensor-specialized silicon buys efficiency at the cost of architectural flexibility; as model architectures drift, the specialization tax may outweigh the efficiency gains, making general-purpose accelerators the safer long-term bet for most organizations.

Angle

Quantifies the trade-off using historical examples of domain-specific processors and provides a decision framework based on model roadmap stability.

Source video

One Datacenter, One Computer: Relearning Systems Design at Fabric Scale

At AI scale, the datacenter becomes the unit of architecture, and engineers who cling to node-level abstractions will build systems that are bottlenecked by inter-node data movement and misaligned fault domains.

Angle

Explores concrete consequences for scheduling, storage, and reliability, with examples from large-scale training clusters.

Source video

Project ideas

4

Roofline Crossover Mapper

movement-lab

For LLM inference, the transition from memory-bandwidth-bound to compute-bound occurs at a predictable batch size and sequence length threshold that can be modeled with a roofline analysis and validated on commodity GPUs.

Proof of concept

Build a micro-benchmark suite that runs a small transformer (e.g., GPT-2 small) across a grid of batch sizes (1–64) and sequence lengths (128–4096) on a single GPU, measuring achieved FLOPS and memory bandwidth utilization; compare against a roofline model to identify the crossover point.

Measurement

Root-mean-square error between predicted and measured crossover points across models and precisions; sensitivity of crossover to HBM bandwidth.

Source video

Matrix Multiplication Share Auditor

new

The claim that 80–90% of AI model processing reduces to matrix multiplications holds for dense transformer models but fails for architectures with heavy attention or elementwise operations at long context lengths.

Proof of concept

Instrument PyTorch models (GPT-2, BERT, LLaMA) with profilers to decompose FLOPs and wall-clock time into matrix multiplication versus non-matmul operations across varying batch sizes and sequence lengths.

Measurement

Percentage of FLOPs and wall-clock time attributable to matrix multiplication, with confidence intervals across model families and configurations; deviation from the 80–90% claim.

Source video

Cache-Aware Data Layout Benchmark for LLM Serving

movement-lab

Aligning tensor data layouts and access patterns to GPU cache hierarchies (e.g., tiling, memory coalescing) yields measurable throughput gains in LLM inference that exceed gains from increasing arithmetic throughput alone.

Proof of concept

Implement a simple LLM inference kernel (e.g., matrix-vector multiply for generation) with naive and cache-optimized data layouts, and benchmark on a GPU with varying batch sizes.

Measurement

Tokens per second and achieved memory bandwidth for naive vs. optimized layouts; speedup ratio and L2/L1 hit rates.

Source video

HBM Bandwidth-per-Dollar Tracker

gatehouse

The effective cost of memory bandwidth for AI accelerators (bandwidth per dollar and per watt) is improving slower than compute cost, making bandwidth the economic bottleneck; stacked HBM generations will show diminishing returns relative to cost.

Proof of concept

Compile a public dataset of accelerator specs (HBM generation, capacity, bandwidth, TDP, price) and compute bandwidth-per-dollar and bandwidth-per-watt trends over time; validate against DDR5 price trends.

Measurement

Year-over-year improvement rates for bandwidth-per-dollar and bandwidth-per-watt versus FLOPS-per-dollar; correlation with listed HBM costs.

Source video

Architectural implications

3

The binding constraint on LLM scaling is data movement, not arithmetic throughput.

Before

Scale by adding compute and higher clock speeds; measure success in FLOPS.

After

Design around memory bandwidth and cache hierarchy, adopting stacked High Bandwidth Memory and bandwidth-aware data layouts.

Consequence

HBM supply, packaging complexity, and memory cost — not raw compute availability — become the dominant scaling economics and the likely chokepoint for AI capacity.

Source video

The relevant machine boundary expands from the individual server to the datacenter.

Before

Orchestration and fault domains defined per host.

After

Treat the datacenter as one giant computer with datacenter-scale scheduling and data movement.

Consequence

Reliability, latency, and cost models must be reasoned about at fabric level; node-local optimizations yield diminishing returns.

Source video

Training and inference share silicon but not workload semantics.

Before

Assume one accelerator profile serves both model development and model serving.

After

Separate capacity planning, batching, and memory strategies for feedback-loop training versus feed-forward inference.

Consequence

Misapplied shared-hardware assumptions produce mis-sized clusters and poor utilization.

Source video

Tradeoffs and failure modes

3

Tensor-specialized silicon (TPU-style) versus general-purpose accelerators

Benefit

Eliminates wasted general-purpose compute by dedicating silicon to tensor multiplications.

Cost or risk

Specialization reduces flexibility: efficiency is tied to the tensor-op pattern current models use, making the substrate brittle if model architectures shift.

TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.
Open source video
Source video

High Bandwidth Memory adoption for AI workloads

Benefit

Provides the memory bandwidth that is the primary LLM bottleneck.

Cost or risk

Stacked HBM inside a single IC package raises hardware cost and complexity, visible in rising DDR5 RAM prices referenced alongside HBM adoption.

Comparing DDR5 RAM price increases over a year alongside the adoption of stacked High Bandwidth Memory (HBM).
Open source video
Source video

Optimizing at the low-level hardware boundary

Benefit

Understanding pipelines and caches allows developers to write significantly more efficient code.

Cost or risk

Requires the ability to reason about hardware execution models; abstractions that hide the memory hierarchy can silently defeat optimization.

Knowing how underlying processors, pipelines, and caches execute instructions allows developers to write significantly more efficient code.
Open source video
Source video

Open questions

4

How will future memory architectures overcome the HBM bandwidth bottleneck?

Why unresolved

The summary names memory bandwidth as the primary constraint and notes HBM adoption and DDR5 price pressure, but offers no candidate post-HBM mechanisms or timelines.

Research direction

Survey emerging memory technologies and near-memory/in-memory compute proposals, then benchmark bandwidth-per-dollar and bandwidth-per-watt against HBM baselines.

Source video

What new hardware paradigms will emerge beyond traditional transformer accelerators?

Why unresolved

The summary states that GPU/LLM architectures evolve rapidly while fundamental principles remain stable, but does not characterize any successor paradigm.

Research direction

Track non-transformer accelerator designs and specialized tensor-op alternatives, evaluating which fundamental principles actually persist.

Source video

Is the claim that 80-90 percent of AI model processing reduces to matrix multiplications empirically robust across model families?

Why unresolved

The summary lists this figure under claims worth verifying as a technical claim rather than a demonstrated measurement.

Research direction

Profile representative models and decompose runtime into matrix multiplication versus non-matmul operations (attention, normalization, elementwise) across architectures and batch sizes.

Source video

Which AI workloads are actually compute-bound versus memory-bandwidth-bound, and how does that mix change with batching and context length?

Why unresolved

The summary asserts memory bandwidth is the primary bottleneck generally, without qualifying by workload or deployment configuration.

Research direction

Build a roofline-style characterization of training and inference across batch sizes and sequence lengths to map where the bottleneck crosses over.

Source video

Key claims

7
factualVerification needed

80 to 90 percent of AI model processing comes down to matrix multiplications.

Evidence

80 to 90 percent of AI model processing comes down to matrix multiplications.

Question

What fraction of FLOPs and wall-clock time in representative training and inference workloads is attributable to matrix multiplication versus other operations?

Source video
factualVerification needed

High Bandwidth Memory (HBM) stacks 16 or more memory chips inside a single integrated circuit package.

Evidence

High Bandwidth Memory (HBM) stacks 16 or more memory chips inside a single integrated circuit package.

Question

What is the actual die-stack count and capacity per stack in current HBM generations and vendor specifications?

Source video
causalVerification needed

Memory bandwidth, not raw clock speed, is the primary bottleneck for LLMs.

Evidence

Memory bandwidth—specifically the speed of reading and writing data to memory—is the primary bottleneck for LLMs, not raw clock speed.

Question

Under what batch sizes, sequence lengths, and precisions does LLM inference become memory-bandwidth-bound rather than compute-bound?

Source video
causalVerification needed

GPUs share massive matrix multiplication requirements with graphics rendering, which is why they became AI hardware.

Evidence

GPUs were not originally built for neural networks, but share massive matrix multiplication requirements with graphics rendering.

Question

Which specific graphics pipeline operations map onto the same primitive operations used in neural network tensor math?

Source video
factualVerification needed

TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.

Evidence

TPUs are specialized hardware optimized specifically for tensor multiplications to eliminate wasted general-purpose compute.

Question

What measured efficiency gain do tensor-specialized accelerators achieve over general-purpose GPUs on comparable tensor workloads?

Source video
opinionVerification not requested

At massive scale, the data center functions as a single giant computer.

Evidence

At massive scale, the data center functions as a single giant computer.

Source video
comparativeVerification not requested

Training and inference share significant overlap, but training requires feedback loops to adjust neural weights.

Evidence

Training and inference share significant overlap, but training requires feedback loops to adjust neural weights.

Source video

Connections

4