Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Overview

• Speaker: Harshul Jain & Tanmay Sah
• Channel: AI Engineer
• Main topic: LLM Inference at Scale
• Purpose: To help engineers and developers understand the fundamental economics, memory dynamics, hardware bottlenecks, and serving optimizations of running large language models at scale. A comprehensive two-hour workshop diving deep into LLM inference at scale from first principles, covering memory consumption, time to first token (TTFT), throughput, GPU memory equations, model optimizations like quantization and grouped query attention, serving engines like vLLM and SGLang, and advanced caching and speculative decoding techniques.

Topic Map

Introduction and Problem Statement

• Explanation: Introduction of speakers Harshul Jain and Tanmay Sah, outlining the workshop agenda and framing the soaring costs and pain points of LLM inference in production.
• Key claims:
  • Every interaction with AI is an inference call.
  • Inference costs now exceed training costs at scale.
  • Query cost must be less than 0.5 cents to keep search profitable.
• Examples:
  • GPT-3 training was a one-time cost of $4.6M, while inference costs scale with every user and token.
• Terminology:
  • LLM inference
  • Token hunger games
  • Inference cost
  • Prefill
  • Decode
• Why it matters: Hardware is limited and compute is expensive; understanding inference economics is critical for building profitable AI products.

GPU Memory Equation and Fundamentals

• Explanation: Deriving model memory size and KV cache memory from first principles using model configuration parameters.
• Key claims:
  • GPU memory = model weights + KV memory + overhead.
  • KV memory is variable and grows with every user and context length.
  • Weights are fixed; KV cache is the variable that kills you.
• Examples:
  • Mistral-7B with 16-bit precision takes 14.5 GB for weights, and 131 KB per token for KV cache.
• Terminology:
  • KV cache
  • Model weights
  • SRAM
  • HBM
  • Arithmetic intensity
• Why it matters: The memory equation governs how many users fit on a GPU, which GPU to buy, and what a token costs.

The Roofline Model: Prefill vs. Decode

• Explanation: Explaining the two phases of LLM inference: prefill (compute-bound) and decode (memory-bound, bottlenecked by memory bandwidth).
• Key claims:
  • Prefill is fast and parallel; decode is slow and sequential.
  • Decode is memory-bound, not compute-bound.
  • Every step of decode streams model weights from HBM to the chip.
• Examples:
  • Mistral-7B decode step streams 14.5 GB from HBM to SRAM on every step.
• Terminology:
  • Prefill
  • Decode
  • Roofline model
  • TTFT
  • ITL
  • Bandwidth wall
• Why it matters: Decode performance is limited by memory bandwidth, meaning faster chips don't help without optimization.

Model Optimizations: Quantization and Attention Mechanisms

• Explanation: Exploring ways to shrink model size and KV cache through quantization (FP16, INT8, INT4/NF4) and attention mechanism improvements (GQA, MLA, FlashAttention).
• Key claims:
  • Quantization shrinks weights and frees memory for more users.
  • Grouped Query Attention (GQA) and Multi-Latent Attention (MLA) reduce KV cache memory without significant quality loss.
  • FlashAttention eliminates redundant HBM reads/writes.
• Examples:
  • Quantizing Mistral-7B from FP16 to INT4 reduces weight memory from 14.5 GB to 3.6 GB.
• Terminology:
  • Quantization
  • Ostrich algorithm
  • Grouped Query Attention (GQA)
  • Multi-Head Attention (MHA)
  • Multi-Latent Attention (MLA)
  • FlashAttention
• Why it matters: Smaller models and optimized attention mechanisms allow packing more users onto a single GPU and lowering per-token costs.

Serving Engines and Optimizations

• Explanation: Comparing serving engines like vLLM and SGLang and their core optimizations: PagedAttention, continuous batching, and prefix caching.
• Key claims:
  • PagedAttention eliminates memory fragmentation in the KV cache.
  • Continuous batching prevents GPU slots from remaining idle.
  • Prefix caching allows sharing prompts across users to save compute.
• Examples:
  • vLLM achieves significant speedups over HuggingFace baseline through paged attention and continuous batching.
• Terminology:
  • vLLM
  • SGLang
  • TensorRT-LLM
  • PagedAttention
  • Continuous batching
  • Prefix caching
  • RadixAttention
• Why it matters: Production serving engines abstract away low-level memory management and scheduling to maximize GPU throughput.

Key Points

Inference costs scale with usage

• Explanation: Unlike training which is a one-time capital cost, inference is an operating cost that scales with every user, token, and session.
• Evidence: Analysis showing search queries with LLMs require massive profit margins and low cost per token.
• Practical implication: Engineering teams must optimize inference serving to maintain profitability.

Decode is memory-bandwidth bound

• Explanation: During token generation (decode), the GPU spends most of its time reading model weights and KV cache from HBM rather than performing arithmetic.
• Evidence: Roofline model analysis showing low arithmetic intensity during the decode phase.
• Practical implication: Hardware memory bandwidth is the primary constraint on token generation speed.

Quantization saves VRAM and increases user capacity

• Explanation: Reducing numerical precision of model weights from FP16 to INT8 or INT4 dramatically reduces VRAM usage with minimal quality loss.
• Evidence: Benchmarking results showing Mistral-7B VRAM drops from 14.5 GB to 3.6 GB under INT4 quantization.
• Practical implication: Quantization enables running larger models on cheaper or fewer GPUs.

PagedAttention solves memory fragmentation

• Explanation: Treating KV cache memory like virtual memory pages in operating systems eliminates internal and external fragmentation.
• Evidence: vLLM paged attention architecture increasing KV cache utilization from ~20% to over 95%.
• Practical implication: Significantly higher concurrent user capacity without increasing hardware budget.

Frameworks, Models & Processes

GPU Memory Equation

• How it works: Calculates total GPU memory consumption as the sum of model weights, KV memory for active users, and system overhead.
• Components:
  • Model Weights
  • KV Memory
  • Overhead
• When to use: Capacity planning and GPU instance selection for LLM deployments.

Roofline Model

• How it works: Visualizes hardware performance limits based on arithmetic intensity, separating compute-bound prefill from memory-bound decode.
• Components:
  • Arithmetic Intensity
  • FLOPs/byte
  • Memory Bandwidth
  • Compute Peak
• When to use: Analyzing inference bottlenecks and understanding hardware utilization.

Trade-off Triangle (Latency, Quality, Throughput)

• How it works: Demonstrates the fundamental architectural trade-off where you can pick two out of three dimensions.
• Components:
  • Latency
  • Quality
  • Throughput
• When to use: Designing serving architecture for specific use cases like premium chat vs. offline batch processing.

Examples & Case Studies

Loading Mistral-7B in a Jupyter notebook via Marimo and measuring GPU memory consumption.

• Illustrates: How memory grows linearly with context length and number of users.
• Lesson: KV cache size quickly dominates memory as context and user concurrency increase.

Benchmarking HuggingFace baseline against vLLM with PagedAttention and continuous batching.

• Illustrates: The massive throughput and latency improvements delivered by optimized serving engines.
• Lesson: Production serving engines are essential for efficient scaling.

Actionable Takeaways

• Immediate:
  • Always calculate your KV cache memory requirements before deploying models.
  • Use serving engines like vLLM or SGLang instead of raw PyTorch/HuggingFace loops.
  • Apply quantization (INT8/INT4) to save VRAM and fit larger models or more users.
• Strategic:
  • Understand the memory-bandwidth bottleneck in the decode phase when optimizing cost.
  • Choose your serving engine based on your workload (e.g., SGLang for agentic prefix caching, vLLM for general production).
  • Leverage prefix caching to eliminate redundant computation for shared system prompts.
• Questions to investigate:
  • How do emerging multi-node distributed inference architectures scale across clusters?
  • What are the precise trade-offs of advanced attention mechanisms like MLA in production quality?
  • How can speculative decoding be effectively tuned for specific domain workloads?

Claims Worth Verifying

• Replacing Google search queries with LLMs requires a $36B profit drain. (economic projection)
• PagedAttention increases KV cache utilization from ~20% to over 95%. (architectural performance claim)

Notable Quotes

"Every interaction with AI is an inference call." "Weights are fixed. The KV cache is the variable that kills you." "Decode is bottlenecked by how fast weights move, not how fast the GPU computes."

Compressed Summary

• LLM inference costs scale with every token and user, making inference optimization critical.
• GPU memory consists of fixed model weights and variable KV cache that grows with context and users.
• Prefill phase is compute-bound, while decode phase is memory-bandwidth bound.
• Quantization and attention optimizations (GQA, MLA, FlashAttention) reduce memory and increase throughput.
• Serving engines like vLLM and SGLang provide PagedAttention, continuous batching, and prefix caching out of the box.
• Keywords: inference, kv cache, roofline, quantization, vllm
• Core insight: LLM inference is primarily constrained by memory bandwidth during the decode phase, making KV cache management and memory optimization the key levers for scaling throughput and reducing costs.