AI Engineer · Published 2026-08-27

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

privateReview: unreviewedConfidence: high
multigpucudaParallelKittensParallelKernelBenchnvlinkmulti-gpucommunication-overlapcuda-kernelsllm-code-generationhardware-abstractionsbenchmarking-agents
Open on YouTube ↗

Summary

Overview

  • Speaker: Simran Arora
  • Channel: AI Engineer
  • Main topic: Simplifying the development of multi-GPU AI kernels and evaluating frontier LLM capabilities on ParallelKernelBench
  • Purpose: To analyze the challenges in multi-GPU kernel development, introduce the ParallelKernelBench evaluation suite and ParallelKittens framework, and assess the reasoning capabilities of foundation models in hardware optimization. Simran Arora, principal scientist at Together AI, discusses simplifying the development of multi-GPU AI kernels. The talk covers hardware architectures, multi-GPU communication bottlenecks, a newly developed benchmark called ParallelKernelBench (PKB), the ParallelKittens programming framework, and evaluates how well frontier large language models (LLMs) perform at writing efficient multi-GPU kernels.

Topic Map

Motivation & Multi-GPU Networking Bottlenecks

  • Explanation: Explains why multi-GPU networking is the new bottleneck as AI models grow larger, shifting computation bottlenecks to inter-GPU and intra-node communication.
  • Key claims:
    • Communication can consume over 50% of execution time in large language model workloads.
    • Communication hardware improvements have lagged far behind compute and memory improvements.
  • Examples:
    • Comparing NVIDIA A100 to B200 tensor core speedups versus inter-node and intra-node communication speedups.
  • Terminology:
    • FlashAttention
    • Mamba
    • TileLang
    • NVLink
    • NVSwitch
  • Why it matters: Understanding communication bottlenecks is essential for maximizing hardware utilization in distributed AI training and inference.

Hardware Fundamentals & Interconnects

  • Explanation: Reviews GPU hardware anatomy, memory hierarchy (from registers and shared memory to L2 cache and HBM), and multi-GPU networking fabrics like NVLink, NVSwitch, InfiniBand, and AMD XGMI.
  • Key claims:
    • Closer memory relative to compute units is much faster but limited in volume.
    • Multi-GPU systems require a hierarchy of interconnects ranging from PCIe to NVLink fabrics and InfiniBand.
  • Examples:
    • NVIDIA H100 and B200 GPU architectures with streaming multiprocessors and HBM stacks.
  • Terminology:
    • Streaming Multiprocessor
    • L2 Cache
    • HBM
    • NVLink
    • NVSwitch
    • InfiniBand
  • Why it matters: Kernel developers must reason about hardware topography and memory latency to write high-performance distributed code.

Tradeoffs in Multi-GPU Kernel Development

  • Explanation: Outlines three key tradeoffs in multi-GPU programming: transfer mechanism choices (Copy Engine vs. TMA vs. register instructions), overlapping schedules (intra-SM vs. inter-SM), and levels of abstraction (NCCL/NVSHMEM vs. low-level code).
  • Key claims:
    • Copy engine is good for large bulk transfers; TMA and register instructions enable fine-grained device-initiated communication.
    • Intra-SM overlapping requires precise synchronization and alignment; inter-SM overlapping offers more flexibility.
  • Examples:
    • Observed memory bandwidth utilization for 1 GB peer-to-peer transfer over NVLink between two H100s.
  • Terminology:
    • Copy Engine
    • TMA
    • Register Instructions
    • Intra-SM Overlapping
    • Inter-SM Overlapping
  • Why it matters: Engineers must navigate these tradeoffs to achieve peak performance in custom communication primitives.

ParallelKittens Framework

  • Explanation: Introduces ParallelKittens, a framework developed at Together AI and Hazy Research that provides a small set of minimal primitives and templates to simplify writing multi-GPU AI kernels.
  • Key claims:
    • A small set of primitives and design patterns is sufficient to write performant multi-GPU kernels across diverse parallelism schemes.
    • ParallelKittens adds roughly a dozen lines of code over single-GPU kernels to insert multi-GPU primitives.
  • Examples:
    • Data parallelism, tensor parallelism, sequence parallelism, and expert parallelism implementations.
  • Terminology:
    • ParallelKittens
    • Loader
    • Consumer
    • Storer
    • Communicator
  • Why it matters: Simplifies multi-GPU kernel authoring without sacrificing peak hardware performance.

ParallelKernelBench & LLM Evaluation

  • Explanation: Presents ParallelKernelBench (PKB), a benchmark covering 87 multi-GPU AI problems across various parallelism schemes, used to evaluate frontier foundation models.
  • Key claims:
    • Single-shot LLMs struggle on PKB, with the best models solving only a fraction of problems faster than baselines.
    • LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.
  • Examples:
    • Evaluating GPT-4.5, Claude 4.7, Gemini 3 Pro, GLM 5.1, and DeepSeek V4 Pro.
  • Terminology:
    • ParallelKernelBench
    • Pass@k
    • Fast1@k
    • Agentic Loops
  • Why it matters: Highlights current limitations of LLMs in hardware-level reasoning and distributed code generation.

Key Points

Communication is the primary bottleneck in modern LLM workloads

  • Explanation: In distributed training and inference, network communication across multiple GPUs consumes more than half of runtime, resulting in low model FLOP utilization.
  • Evidence: Analysis of runtime profiles showing communication gaps versus compute and memory timelines.
  • Practical implication: Kernel optimization efforts must shift focus from single-GPU compute to efficient multi-GPU communication overlapping.

Hardware interconnects are diversifying

  • Explanation: Vendors use different interconnect strategies such as NVLink/NVSwitch for NVIDIA, 3D torus and optical interconnects for TPUs, and XGMI for AMD.
  • Evidence: Comparative architectural diagrams of NVIDIA, TPU, and AMD multi-GPU node configurations.
  • Practical implication: Writing portable, high-performance kernels requires targeting diverse networking topologies.

Frontier LLMs struggle with multi-GPU algorithmic reasoning

  • Explanation: While LLMs can generate syntax-correct code via agentic loops, they fail to reason through complex hardware tradeoffs like collective ordering, tensor partitioning, and overlapping schedules.
  • Evidence: Results from ParallelKernelBench showing success rates dropping rapidly as speedup thresholds increase.
  • Practical implication: Automated code generation tools need domain-specific abstractions like ParallelKittens to assist LLMs effectively.

Frameworks, Models & Processes

ParallelKittens

  • How it works: Encapsulates synchronization and buffering into a small set of programming primitives and modular components (Loader, Communicator, Consumer, Storer) for multi-GPU kernels.
  • Components:
    • Loader
    • Communicator
    • Consumer
    • Storer
    • Shared Memory (SMEM)
    • Register Tile
  • When to use: When developing high-performance multi-GPU kernels across data, tensor, sequence, and expert parallelism.

ParallelKernelBench (PKB)

  • How it works: A distributed benchmark featuring 87 problems paired with PyTorch reference implementations and multi-GPU hardware topologies to evaluate LLM-generated CUDA code.
  • Components:
    • Task instruction
    • Language model
    • Custom kernel generation
    • Evaluator (Correctness, Performance, Communication)
  • When to use: For benchmarking and evaluating code-generation LLMs on distributed hardware tasks.

Examples & Case Studies

Evaluating single-shot and agentic LLMs (like Gemini 3 Pro) on ParallelKernelBench problems.

  • Illustrates: Agentic loops help correct syntax and shape errors, but do not help models innovate algorithmically.
  • Lesson: LLMs require proper abstractions and domain-specific context to successfully optimize hardware-level code.

Actionable Takeaways

  • Immediate:
    • Explore ParallelKittens on GitHub for writing cleaner multi-GPU CUDA kernels.
    • Account for communication overhead when profiling distributed AI workloads.
  • Strategic:
    • Design AI systems and frameworks that bridge the gap between high-level abstractions and low-level hardware communication primitives.
    • Recognize that current LLMs require structured benchmarks like PKB to guide improvements in hardware-aware code generation.
  • Questions to investigate:
    • How can agentic frameworks be adapted to reason about hardware tradeoffs?
    • What new compilation techniques can bridge Triton to multi-GPU fabrics effectively?

Claims Worth Verifying

  • Communication can consume over 50% of execution time in large language model workloads. (empirical)
  • ParallelKittens adds roughly a dozen lines of code over single-GPU kernels for multi-GPU primitives. (technical)

Notable Quotes

"Communication can consume over 50% of execution time in large language model workloads, leaving GPU compute idle." "Agentic loops help correct syntax and shape errors, but does not help the model innovate algorithmically."

Compressed Summary

  • Multi-GPU communication is the primary bottleneck in modern LLM workloads.
  • ParallelKittens provides minimal primitives to simplify multi-GPU kernel development.
  • ParallelKernelBench evaluates LLMs on 87 distributed multi-GPU coding tasks.
  • Frontier LLMs struggle with hardware tradeoff reasoning despite agentic refinement.
  • Keywords: multigpu, cuda, parallelkittens, parallelkernelbench, nvlink
  • Core insight: While hardware communication bottlenecks dominate modern multi-GPU workloads, current LLMs struggle to reason through complex hardware tradeoffs without structured programming abstractions like ParallelKittens.

Core insights

6
Empirical Resultmedium noveltystrong evidence

Communication, not single-GPU compute, is now the dominant bottleneck in distributed LLM workloads: it can account for more than 50% of execution time. Since interconnect bandwidth has improved much slower than compute or memory, the payoff for optimizing compute kernels is limited unless communication is overlapped with computation.

Why it matters

Directs engineering effort away from single-GPU kernel tuning and toward intra-kernel communication overlap, transfer-mechanism selection, and scheduling that hides network stalls behind compute.

Generalization

Performance engineering for distributed compute should start with the communication timeline, not the compute timeline, especially as compute and memory scale faster than interconnects.

Communication can consume over 50% of execution time in large language model workloads.
Open source video
Communication hardware improvements have lagged far behind compute and memory improvements.
Open source video
Empirical Resulthigh noveltystrong evidence

Frontier LLMs can produce syntactically and shape-correct multi-GPU kernels through agentic loops, but they fail at the algorithmic hardware tradeoffs—collective ordering, tensor partitioning, and overlapping schedules—so their performance success rate collapses as speedup thresholds rise.

Why it matters

Tool-builders cannot assume that agentic error-correction is sufficient to make LLMs useful for high-performance kernel generation; raw code generation still lacks the mental model needed to reason about multi-GPU hardware.

Generalization

For any domain where runtime performance matters, an agentic loop that fixes compilation errors is far weaker than one that can also be told about performance and hardware tradeoffs.

LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.
Open source video
success rates dropping rapidly as speedup thresholds increase.
Open source video
Architecturehigh noveltystrong evidence

A small, composable set of primitives can encapsulate synchronization and buffering for distributed GPU kernels: ParallelKittens expresses data, tensor, sequence, and expert parallelism using Loader, Communicator, Consumer, and Storer patterns, adding only about a dozen lines over a single-GPU kernel.

Why it matters

Shows that the lowest-level communication complexity can be pushed into a thin, curated abstraction layer instead of being scattered across kernel code. This is an architectural pattern both for human kernel developers and for LLM-based generators.

Generalization

High-performance parallel programming benefits when the varied collective operations of distributed execution are compressed into a few, composable components so that optimization is localized rather than interleaved with business logic.

A small set of primitives and design patterns is sufficient to write performant multi-GPU kernels across diverse parallelism schemes.
Open source video
ParallelKittens adds roughly a dozen lines of code over single-GPU kernels to insert multi-GPU primitives.
Open source video
Mechanismmedium noveltymoderate evidence

The right way to initiate multi-GPU transfer depends on data size and device-initiation granularity: Copy Engine is best for large bulk transfers, while TMA and register instructions are needed for fine-grained device-initiated communication. Similarly, overlap scheduling can be placed within an SM or across SMs, each with different synchronization constraints.

Why it matters

These are concrete, compiler-level choices that materially affect memory bandwidth utilization. A framework or agent that hides them too coarsely risks sacrificing performance; one that exposes them lets authors tune transfer granularity.

Generalization

When designing parallel abstractions, the abstraction should preserve granularity choices for data movement (bulk vs fine-grained) instead of imposing one transfer mechanism for all cases.

Copy engine is good for large bulk transfers; TMA and register instructions enable fine-grained device-initiated communication.
Open source video
Intra-SM overlapping requires precise synchronization and alignment; inter-SM overlapping offers more flexibility.
Open source video
Predictionmedium noveltymoderate evidence

Multi-GPU hardware interconnects are diversifying by vendor—NVLink/NVSwitch for NVIDIA, 3D torus and optical for TPUs, XGMI for AMD—so writing portable high-performance kernels is becoming a problem of targeting diverse, topology-specific communication fabrics.

Why it matters

A kernel framework, benchmark suite, or code-generation tool built for one network topology may not transfer to another. Abstraction layers must accommodate topology-dependence or they will fail at the portability task.

Generalization

Distributed programming systems should make topology explicit and strategy-aware rather than assuming one global interconnect model.

Vendors use different interconnect strategies such as NVLink/NVSwitch for NVIDIA, 3D torus and optical interconnects for TPUs, and XGMI for AMD.
Open source video
Practicemedium noveltystrong evidence

A code-generation benchmark aimed at 'fastness' rather than just correctness changes what it means for an LLM to 'solve' a problem. ParallelKernelBench's speedup-threshold results show that correctness-based pass rates overestimate capability to generate performant distributed kernels.

Why it matters

Evaluation methodology for agentic code generation must distinguish 'compiles and produces correct output' from 'runs faster than a baseline'. Fast@k-style metrics need to become standard when optimizing for latency or throughput.

Generalization

Agentic-code benchmarks should always include thresholded performance metrics, a reference baseline, and realistic hardware rather than only unit tests, otherwise model progress may be illusory.

Single-shot LLMs struggle on PKB, with the best models solving only a fraction of problems faster than baselines.
Open source video
Single-shot LLMs struggle on PKB, with the best models solving only a fraction of problems faster than baselines.
Open source video

Deep dives

4

Does constraining LLMs to ParallelKittens-style primitives raise their Fast1@k on multi-GPU kernel generation?

Research question

When an LLM agent is required to express multi-GPU communication using a Loader/Communicator/Consumer/Storer library instead of writing raw CUDA multi-GPU code, does its solve rate at meaningful speedup thresholds increase materially?

Why

If the main LLM failure is algorithmic hardware reasoning, adding a compact abstraction that makes collective ordering, tensor partitioning, and synchronization explicit may be the cheapest way to improve generation. This determines whether framework investment dominates model/training investment for high-performance kernel synthesis.

A small set of primitives and design patterns is sufficient to write performant multi-GPU kernels across diverse parallelism schemes.
Open source video
ParallelKittens adds roughly a dozen lines of code over single-GPU kernels to insert multi-GPU primitives.
Open source video
Source video

Which feedback modality teaches agentic loops to optimize hardware-level kernel tradeoffs?

Research question

Can an agentic loop improve more on ParallelKernelBench when feedback shifts from compilation/shape errors to memory-bandwidth traces, overlap reports, or speedup-threshold pass/fail signals?

Why

Agentic loops currently fix only syntax and shape errors; if the missing component is performance feedback, then building profilers into the loop could close the gap between pass@k and Fast1@k without new model training.

LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.
Open source video
Source video

Is the over-50% communication bottleneck stable across model architectures and interconnect generations?

Research question

Does communication time dominate execution for FlashAttention, Mamba, and dense Transformer kernels at similar arithmetic intensities, and does the ratio persist on NVLink, NVSwitch, and newer multi-GPU fabrics?

Why

The communication-wall claim is a strategic pointer for where to spend optimization effort. If the ratio is workload- or vendor-specific, performance frameworks need conditional schedules and cannot assume communication dominates everywhere.

Communication can consume over 50% of execution time in large language model workloads.
Open source video
Communication hardware improvements have lagged far behind compute and memory improvements.
Open source video
Source video

Portable performance-threshold benchmarks across NVLink, XGMI, and TPU fabrics

Research question

Can ParallelKernelBench-style speedup-threshold evaluation be generalized to non-NVIDIA interconnects while keeping thresholds meaningful and rankings stable across topologies?

Why

Interconnect diversity is making any vendor-specific benchmark less representative; a portable benchmark must parameterize topology or risk producing illusory portability of agent-generated kernels.

Vendors use different interconnect strategies such as NVLink/NVSwitch for NVIDIA, 3D torus and optical interconnects for TPUs, and XGMI for AMD.
Open source video
Source video

Article ideas

4

Fast@k Is the Truth Serum for LLM Code Benchmarks

Correctness-based pass@k overstates what frontier LLMs can do for performance-sensitive code; PKB's speedup-threshold results show that solve rates collapse as speedup requirements rise, so standard code-agent evaluations should report Fast@k or admit they are not measuring practical capability.

Angle

A critique of code-generation evaluation through the ParallelKernelBench lens.

Source video

Stop Making Agents Compile Kernels: Give Them ParallelKittens Instead

Spending agentic loop budget on syntax and shape fixes is the wrong bottleneck; defining a dozen-line Loader/Communicator/Consumer/Storer layer turns multi-GPU kernel generation into composition over tested primitives and gives LLMs a better chance of making the hardware-level choices that matter.

Angle

Abstraction design as the missing interface for LLM systems programmers.

Source video

The Communication Wall Is the New Compute Wall in AI Infrastructure

When communication consumes more than half of LLM workload time, distributed kernel engineering should be reorganized around overlap scheduling, transfer-mechanism selection, and collective ordering rather than single-GPU micro-optimization.

Angle

A performance-engineering manifesto for multi-GPU systems.

Source video

Portability Is a Topology Problem, Not an API Problem

With NVLink/NVSwitch, TPU 3D torus, and AMD XGMI as divergent fabrics, portable high-performance kernel software must treat interconnect topology as a first-class compilation target; abstractions that hide topology will leave performance on the table.

Angle

Multi-vendor interconnect-aware design for portable kernel frameworks.

Source video

Project ideas

3

ParallelKittens Constraint Study for PKB

beyond-evals

On a fixed subset of ParallelKernelBench problems, an LLM agent forced to express multi-GPU data movement through a small composable primitive library will achieve at least 15 percentage points higher Fast1@k at a 2.0x speedup threshold than an equivalent raw-CUDA agent that only receives compiler-fix feedback.

Proof of concept

Build two harnesses over the same PKB subset and budget: one arm lets the model write raw CUDA with compile/run feedback; the other arm constrains generation to Loader/Communicator/Consumer/Storer-style templates while keeping the same base model and retry budget. Run both arms across representative kernels on a 2-4 GPU NVLink node.

Measurement

Fast1@k at 1.0x, 1.5x, and 2.0x thresholds; median achieved kernel speedup; compilation success rate.

Source video

Feedback-Modal Ablation on Kernel Agents

beyond-evals

Supplementing agentic-loop feedback with per-kernel memory-bandwidth utilization and overlap reports improves Fast1@k on ParallelKernelBench by at least 10 percentage points over compiler-error-only feedback at a 1.25x speedup threshold.

Proof of concept

For a fixed model and PKB sample, compare agent runs under three feedback protocols: compile errors only; speedup pass/fail only; and speedup plus bandwidth/overlap traces computed from actual kernel executions. Keep retry budgets and prompts constant.

Measurement

Delta Fast1@k across feedback conditions; number of retries needed to reach threshold; variance across model instances.

Source video

Multi-Workload Communication Share Study

new

For representative attention, FlashAttention, and Mamba kernels on a 2-GPU NVLink node, communication fraction remains above 50% for each workload when arithmetic intensity per GPU is held fixed, confirming communication dominance is workload-level and not an artifact of one benchmark.

Proof of concept

Profile reference implementations and their ParallelKittens-style distributed variants on an H100 node, measuring kernel time, NVLink transfer time and idle time; make total per-GPU arithmetic intensity comparable by scaling sequence/tensor parallelism.

Measurement

Communication fraction of total execution time; achieved NVLink bandwidth; overlap utilization across each workload.

Source video

Architectural implications

3

LLM agentic loops can iterate on syntax and shape errors but do not improve hardware-level algorithmic reasoning. A raw code-generation harness therefore has a ceiling when applied to high-performance systems.

Before

Agent harnesses for CUDA kernels generally compile, run tests, output-pass or fail, then retry with compilation error feedback.

After

A harness for kernel generation should additionally provide performance feedback, speedup thresholds, and likely a library of domain-specific parallel primitives that constrain the search space.

Consequence

The engineering burden shifts from 'make the compiler happy' to 'represent the hardware/performance tradeoffs as accessible feedback and abstractions'.

Source video

ParallelKittens suggests that a dozen lines of communication pattern can be layered onto single-GPU kernels using modular components.

Before

Multi-GPU kernels are typically written as separate low-level code with explicit distributed collectives and synchronization at every relevant point.

After

Parallel patterns are expressed as a pipeline of Loader, Communicator, Consumer, and Storer, so multi-GPU behavior is inserted in a small, fixed set of places.

Consequence

The abstraction boundary becomes about scheduling and buffer ownership; humans LLMs can reason about module composition instead of raw low-level calls.

Source video

Interconnect choice and topology heavily influence what communication strategy is best, yet high-level convenience layers often abstract topology away.

Before

Portable distributed code targets a generic collective library and hopes the network is fast enough.

After

Kernels, frameworks, and benchmarks explicitly parameterize by GPU vendor and fabric, treating topology as a first-class compilation target.

Consequence

Performance portability becomes a difficult optimization challenge, but one that is exposed to the developer rather than hidden.

Source video

Tradeoffs and failure modes

4

Transfer mechanism selection

Benefit

Copy Engine handles large bulk transfers with high utilization and less software control.

Cost or risk

Fine-grained device-initiated transfer requires TMA or register instructions, increasing programming complexity and reducing the convenience of simple copies.

Copy engine is good for large bulk transfers; TMA and register instructions enable fine-grained device-initiated communication.
Open source video
Source video

Overlap scheduling location

Benefit

Inter-SM overlap is more flexible to schedule and less constrained than intra-SM.

Cost or risk

Intra-SM overlap demands precise synchronization and alignment, making it brittle if not all data movement and compute dependencies are visible to the scheduler.

Intra-SM overlapping requires precise synchronization and alignment; inter-SM overlapping offers more flexibility.
Open source video
Source video

Agentic code-fix loops

Benefit

LLM agentic loops can eliminate many syntax and shape errors in generated multi-GPU kernels.

Cost or risk

These loops consume time and tokens while failing to improve the deeper inefficiencies: collective ordering, tensor partitions, and overlapping schedules remain unsolved.

LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.
Open source video
Source video

Level of abstraction for multi-GPU communication

Benefit

High-level tools like NCCL/NVSHMEM and ParallelKittens simplify multi-GPU kernel development and make performance portable.

Cost or risk

Too much abstraction can hide transfer granularity and topology choices that are needed to approach peak bandwidth, while full low-level code is costly to write and maintain.

Outlines three key tradeoffs in multi-GPU programming: ... levels of abstraction (NCCL/NVSHMEM vs. low-level code).
Open source video
Source video

Open questions

4

If frontier LLMs are given ParallelKittens-style domain-specific primitives instead of writing raw CUDA multi-GPU code, does their Fast1@k rate improve substantially?

Why unresolved

ParallelKernelBench evidence only reports that LLMs struggle on raw multi-GPU kernel problems; no LLM-plus-abstraction experimental result is in the summary.

Research direction

Augment PKB with tool/abstraction conditions and measure Fast1@k for ParallelKittens implementations vs raw CUDA implementations.

Source video

What feedback signal—execution traces, memory bandwidth charts, overlap reports, or simulated interconnect latency—would improve an agentic loop's ability to optimize multi-GPU kernels?

Why unresolved

Agentic loops are observed to help only syntax and shape, not hardware tradeoffs, so the kind of feedback that would teach hardware reasoning is not yet specified.

Research direction

Compare agentic loops with different feedback channels on PKB and measure how much each closes the gap between pass@k and fast1@k.

Source video

Does the >50% communication time figure hold across model architectures (e.g., Transformer vs Mamba) and across interconnect generations?

Why unresolved

The summary gives one strong measurement but does not include architecture- or generation-specific variance.

Research direction

Create a profile suite covering multiple LLM workloads, hardware nodes, and interconnect families to map where communication- vs compute-bound behavior changes.

Source video

Can a benchmark like ParallelKernelBench be designed so it is portable across NVIDIA, TPU, and AMD topologies without losing the precise speedup thresholds that make it informative?

Why unresolved

Vendors are diversifying interconnects, while PKB appears tied to the specific multi-GPU CUDA world and associated software stack.

Research direction

Develop a topology-parameterization layer and standard speedup measurement across platforms so that cross-vendor comparisons are meaningful.

Source video

Key claims

7
factualVerification needed

Communication can consume over 50% of execution time in large language model workloads.

Evidence

Communication can consume over 50% of execution time in large language model workloads.

Question

Does this hold across representative LLM training and inference workloads in more than one hardware generation?

Source video
comparativeVerification needed

Communication hardware improvements have lagged far behind compute and memory improvements.

Evidence

Communication hardware improvements have lagged far behind compute and memory improvements.

Question

What is the ratio of annual interconnect vs compute/memory scaling in recent generations?

Source video
causalVerification needed

A small set of primitives and design patterns is sufficient to write performant multi-GPU kernels across diverse parallelism schemes.

Evidence

A small set of primitives and design patterns is sufficient to write performant multi-GPU kernels across diverse parallelism schemes.

Question

Can this set of primitives cover all common kernel patterns, or is there a hidden class of parallelism that does not fit?

Source video
comparativeVerification needed

LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.

Evidence

LLMs can handle syntax and shape errors using agentic loops, but struggle to reason through algorithmic hardware tradeoffs.

Question

Which agentic feedback mechanisms, if any, succeed at improving the algorithmic hardware decisions on PKB?

Source video
factualVerification needed

Copy engine is good for large bulk transfers; TMA and register instructions enable fine-grained device-initiated communication.

Evidence

Copy engine is good for large bulk transfers; TMA and register instructions enable fine-grained device-initiated communication.

Question

What are the measured bandwidth/latency boundaries between copy-engine and TMA operation in modern multi-GPU kernels?

Source video
comparativeVerification needed

Intra-SM overlapping requires precise synchronization and alignment; inter-SM overlapping offers more flexibility.

Evidence

Intra-SM overlapping requires precise synchronization and alignment; inter-SM overlapping offers more flexibility.

Question

What effect does each overlap style have on achieved speedup in a variety of kernel designs?

Source video
factualVerification needed

Frontier LLMs fail to reason through complex hardware tradeoffs like collective ordering, tensor partitioning, and overlapping schedules.

Evidence

While LLMs can generate syntax-correct code via agentic loops, they fail to reason through complex hardware tradeoffs like collective ordering, tensor partitioning, and overlapping schedules.

Question

Would this conclusion change with larger models, more feedback, or specialized multi-GPU examples in the prompt?

Source video

Connections

5

Fast1@kpass@k

Fast1@k is a performance-thresholded contrast to syntax-oriented pass@k and better reflects real-world usability.

Source video