AI Engineer · Published 2026-08-03

Building Turbopuffer: Gergely Orosz (@pragmaticengineer ) × Simon Eskildsen (CEO)

Open on YouTube ↗

Summary

Overview

  • Speaker: Gergely Orosz and Simon Eskildsen
  • Channel: AI Engineer
  • Main topic: Building Turbopuffer, database engineering, infrastructure scaling, and startup economics
  • Purpose: Provide deep technical and operational insights into infrastructure scaling, database architecture, engineering culture, and startup growth from the perspective of an experienced database engineer and founder. In this candid and technical fireside chat at the AI Engineer World's Fair, Gergely Orosz interviews Simon Eskildsen, founder and CEO of Turbopuffer. They discuss Simon's early programming journey, his eight-year tenure scaling infrastructure at Shopify through massive growth phases, the engineering challenges of managing database state and caching, building Turbopuffer, navigating venture capital, and the realities of running a high-performance vector database company in the AI era.

Topic Map

Early Programming and Learning Path

  • Explanation: Simon Eskildsen shares how he fell in love with computers via PowerPoint, learned HTML, PHP, and JavaScript, and dived into competitive programming and international informatics olympiads without a formal computer science degree.
  • Key claims:
    • Self-taught programming through front-end experiments, HTML, PHP, and Danish programming advice.
    • Crash course in computer science through competitive programming and the International Olympiad in Informatics.
  • Examples:
    • Transitioning from PowerPoint diagrams to building dynamic web pages and discovering international informatics competitions.
  • Terminology:
    • HTML
    • PHP
    • Dreamweaver
    • Front-end
  • Why it matters: Illustrates how curiosity-driven, self-taught engineering can build a foundation for massive infrastructure scaling roles later.

Shopify Infrastructure Scaling

  • Explanation: Simon discusses his 8-year stint at Shopify (2013-2021) scaling infrastructure through hyper-growth, Black Friday traffic spikes, containerization with Docker, sharding, and database caching strategies.
  • Key claims:
    • Shopify grew at 120-140% year-over-year during his tenure.
    • Database scaling required moving beyond single shards just before Black Friday peaks.
    • Simplicity and understanding low-level drivers (MySQL, Rails, storage layers) are critical for robust systems.
  • Examples:
    • Scaling a Redis server with 128GB of RAM that nobody knew the contents of.
    • Implementing sharding just a week before Black Friday.
  • Terminology:
    • Sharding
    • Redis
    • MySQL
    • Rails
    • Docker
    • Cache rights
  • Why it matters: Demonstrates the realities of scaling a major SaaS platform through rapid growth phases and infrastructure bottlenecks.

Turbopuffer and Vector Database Engineering

  • Explanation: Simon explains the genesis of Turbopuffer, a vector database designed for high performance and cost efficiency, addressing the limitations of existing storage solutions and LLM memory constraints.
  • Key claims:
    • Vector databases face severe cost and latency challenges when relying solely on traditional cloud infrastructure.
    • S3 and object storage provide great durability but high latency (hundreds of milliseconds), requiring clever caching and reverse proxies.
    • Simplicity is the core engineering principle: keeping architecture simple beats overly complex distributed layers.
  • Examples:
    • Napkin math calculations for memory bandwidth, S3 round trips, NVMe SSDs, and EBS volumes.
    • Using Nginx and reverse proxies with object storage caching to dramatically reduce database costs.
  • Terminology:
    • Vector database
    • S3
    • NVMe
    • EBS
    • Reverse proxy
    • P99 latency
    • Clustering
    • Centroids
  • Why it matters: Provides a masterclass in evaluating cost-performance tradeoffs in cloud storage and vector search infrastructure for AI workloads.

Venture Capital, Hiring, and Remote Culture

  • Explanation: Simon and Gergely debate the reasons for raising venture capital, the importance of aligning funding with R&D and employee liquidity, and the dynamics of building remote-first teams.
  • Key claims:
    • Raising venture capital should primarily fund R&D and provide employee liquidity rather than fuel ego or status.
    • Remote culture works exceptionally well for high-speed infrastructure teams when built on strong engineering trust and asynchronous communication.
  • Examples:
    • Raising capital to fund early R&D and bring on top engineering talent like Boyan and Morgan.
    • Hosting Turbopuffer campfires and gatherings in San Francisco and across North America.
  • Terminology:
    • Venture capital
    • R&D
    • Liquidity
    • Remote culture
    • Campfires
  • Why it matters: Offers pragmatic advice on startup financing, team building, and maintaining high engineering standards without unnecessary corporate bloat.

Key Points

Simplicity over Complexity

  • Explanation: Simple architectures that leverage commodity object storage (S3) combined with smart caching often outperform complex distributed database topologies.
  • Evidence: Simon's experience building reverse proxies in Nginx over S3 to achieve massive cost reductions.
  • Practical implication: Engineers should question complex distributed systems layers before exhausting simpler storage caching approaches.

State Management is the Core Hard Problem

  • Explanation: Most gnarly engineering problems in large-scale systems boil down to state management and database consistency.
  • Evidence: Database scaling challenges at Shopify and Turbopuffer regarding memory, caching, and failover handling.
  • Practical implication: Investing early in deep database and storage layer understanding pays massive dividends.

Pragmatic Capital Allocation

  • Explanation: Raising venture capital should be tied to clear operational goals like accelerating R&D and rewarding early employees, avoiding vanity metrics.
  • Evidence: Turbopuffer's deliberate approach to capital raising and unit economics.
  • Practical implication: Founders must maintain disciplined unit economics even when scaling rapidly.

Frameworks, Models & Processes

Napkin Math Infrastructure Evaluation

  • How it works: Calculate theoretical limits of hardware (memory bandwidth, disk IOPS, network latency) before committing to complex architectures.
  • Components:
    • Memory bandwidth per core
    • S3 round-trip cost and latency
    • NVMe/EBS throughput limits
    • Query plan complexity
  • When to use: When designing high-performance databases, search engines, or distributed storage systems.

Examples & Case Studies

Simon sharded Shopify's database just a week before Black Friday.

  • Illustrates: Executing high-stakes infrastructure changes under extreme time pressure.
  • Lesson: High-performing engineering teams can execute radical changes quickly when forced by circumstances.

Building a vector search caching layer over S3 that reduced customer costs by 95%.

  • Illustrates: Leveraging object storage economics with intelligent caching for AI workloads.
  • Lesson: Challenging conventional cloud storage assumptions yields massive cost and performance advantages.

Actionable Takeaways

  • Immediate:
    • Evaluate hardware limits using napkin math before building complex distributed abstractions.
    • Prioritize architectural simplicity over unnecessary framework complexity.
  • Strategic:
    • Align startup financing directly with R&D acceleration and team retention.
    • Embrace remote-first engineering cultures backed by high trust and asynchronous execution.
  • Questions to investigate:
    • How do modern vector databases balance S3 latency with ultra-low latency index lookups at scale?
    • What are the optimal hardware skews (CPU vs. GPU vs. RAM) for inference workloads in 2024 and beyond?

Claims Worth Verifying

  • Shopify grew at 120-140% year-over-year during Simon's tenure. (historical business metric)
  • Turbopuffer reduced storage and vector search costs by 95% compared to traditional setups. (performance/cost claim)

Notable Quotes

"Simplicity above everything." "If everything goes south, we can always pivot into vapes."

Compressed Summary

  • Simon Eskildsen shares his journey from self-taught PHP developer to Shopify infrastructure leader and Turbopuffer CEO.
  • Emphasizes architectural simplicity, low-level database understanding, and cost-efficient vector search over S3.
  • Discusses pragmatic venture capital raising, remote engineering culture, and hardware optimization.
  • Keywords: turbopuffer, databases, infrastructure, scaling, vector search, engineering
  • Core insight: Achieving extreme performance and cost efficiency in modern database engineering requires architectural simplicity, rigorous hardware napkin math, and a relentless focus on state management.

Core insights

6
Architecturehigh noveltymoderate evidence

A vector database can be built as a thin query/clustering layer over commodity object storage (S3) plus an aggressive reverse-proxy cache (Nginx) in front of it, rather than as a purpose-built distributed storage engine — with claimed ~95% customer cost reduction.

Why it matters

It relocates the hard engineering from storage replication/consensus into cache design and latency hiding, a very different set of failure modes and responsibilities than a classic distributed database.

Generalization

For read-heavy AI workloads, the storage tier can be commoditized and the value-add moved into a caching/coordination tier above it; the same pattern applies to embedding stores, feature stores, and artifact caches for model weights.

Using Nginx and reverse proxies with object storage caching to dramatically reduce database costs.
Open source video
Building a vector search caching layer over S3 that reduced customer costs by 95%.
Open source video
Practicemedium noveltymoderate evidence

Do napkin math on hardware limits (memory bandwidth per core, S3 round-trip latency and cost, NVMe/EBS throughput, query plan complexity) as an explicit gate before committing to a complex distributed architecture.

Why it matters

It turns architecture choice into a quantitative comparison against physical limits instead of an appeal to fashion or scale anxiety, which is exactly the decision most teams get wrong early and pay for later.

Generalization

Any infrastructure choice (vector index, KV cache, batch inference pipeline, agent memory backend) can be pre-screened by computing whether the naive hardware-bound design already satisfies the latency/cost budget.

Calculate theoretical limits of hardware (memory bandwidth, disk IOPS, network latency) before committing to complex architectures.
Open source video
Napkin math calculations for memory bandwidth, S3 round trips, NVMe SSDs, and EBS volumes.
Open source video
Mental Modelmedium noveltymoderate evidence

Simplicity of the storage topology is treated as the primary engineering principle, with 'simple S3 + smart caching' explicitly claimed to outperform complex distributed database topologies.

Why it matters

It inverts the default instinct to add distributed layers (consensus, replication, custom storage nodes) when facing scale, and instead pushes complexity into a smaller, more testable surface.

Generalization

When a system is on the wrong side of a cost or latency cliff, first re-examine the storage/transport substrate before adding new distributed machinery.

Simple architectures that leverage commodity object storage (S3) combined with smart caching often outperform complex distributed database topologies.
Open source video
Simplicity is the core engineering principle: keeping architecture simple beats overly complex distributed layers.
Open source video
Mental Modellow noveltymoderate evidence

State management — memory contents, caching, consistency, and failover — is identified as the recurring core difficulty in large-scale systems, spanning both Shopify's MySQL/Redis/Rails stack and Turbopuffer.

Why it matters

It argues that investment priorities should go to state-layer depth (drivers, storage internals, cache semantics) rather than to higher-level application architecture.

Generalization

For AI systems, the analogous core difficulty is agent/session state: context stores, cache consistency, and recovery semantics are likely where the operational pain concentrates.

Most gnarly engineering problems in large-scale systems boil down to state management and database consistency.
Open source video
Investing early in deep database and storage layer understanding pays massive dividends.
Open source video
Empirical Resultmedium noveltymoderate evidence

Simplicity in architecture preserves the ability to execute radical infrastructure changes under extreme time pressure — sharding was implemented a week before Black Friday and it worked.

Why it matters

It suggests 'change velocity under deadline' should be treated as an architectural property to design for, not just an organizational trait, since it depends on how few moving parts must be touched.

Generalization

Systems whose critical path is small can be re-partitioned or re-sharded close to a traffic event; systems with deep distributed coupling cannot, which raises the cost of any late-stage architectural correction.

High-performing engineering teams can execute radical changes quickly when forced by circumstances.
Open source video
Implementing sharding just a week before Black Friday.
Open source video
Practicelow noveltyweak evidence

Venture capital is framed as a tool tied to specific operational goals — funding R&D and providing employee liquidity — explicitly not to fuel ego or status.

Why it matters

It reframes the funding decision as an engineering-resource and unit-economics decision, which affects how much headcount and infrastructure burn an infra team can sustain.

Generalization

Capital decisions in technical companies should be justified by concrete capability gains (hiring, R&D runway) rather than by competitive signaling.

Raising venture capital should primarily fund R&D and provide employee liquidity rather than fuel ego or status.
Open source video
Founders must maintain disciplined unit economics even when scaling rapidly.
Open source video

Deep dives

5

Cache coherence and invalidation over object storage as the durable source of truth

Research question

When the durable tier is S3 and the serving tier is a reverse-proxy cache, what invalidation protocol bounds staleness, and how is that staleness observed and alerted on under index updates and cold starts?

Why

The claimed 95% cost reduction is purchased with a cache layer whose correctness semantics are unspecified; if staleness is unbounded or unobservable, the architecture trades a known storage cost for an unknown correctness risk, which is the worst kind of trade for retrieval systems where index freshness affects answer quality.

S3 and object storage provide great durability but high latency (hundreds of milliseconds), requiring clever caching and reverse proxies.
Open source video
Building a vector search caching layer over S3 that reduced customer costs by 95%.
Open source video
Source video

Workload envelope where object-storage-plus-cache loses to a distributed database

Research question

Along which axes — write ratio, working-set-to-cache ratio, and query locality skew — does the S3-plus-cache topology stop outperforming a distributed database, and is the crossover sharp enough to serve as a decision rule?

Why

The simplicity claim is stated as 'often outperform' with no boundary condition, which is exactly the kind of unbounded generalization that leads teams to adopt an architecture outside its valid regime; locating the crossover converts a slogan into an engineering constraint.

Simple architectures that leverage commodity object storage (S3) combined with smart caching often outperform complex distributed database topologies.
Open source video
Simplicity is the core engineering principle: keeping architecture simple beats overly complex distributed layers.
Open source video
Source video

Calibration of napkin-math predictions against measured latency and cost

Research question

Do napkin-math estimates for memory bandwidth, S3 round trips, and NVMe/EBS throughput predict measured P50 and P99 latency and per-query cost within an acceptable factor, or do they systematically diverge in one direction?

Why

Napkin math is only useful as a gate if its error is bounded and known; if predictions are consistently optimistic on tail-sensitive axes, teams will approve architectures the hardware cannot actually serve, and the failure will surface during a traffic event rather than at design time.

Calculate theoretical limits of hardware (memory bandwidth, disk IOPS, network latency) before committing to complex architectures.
Open source video
Napkin math calculations for memory bandwidth, S3 round trips, NVMe SSDs, and EBS volumes.
Open source video
Source video

Tail-latency control when the durable tier is inherently slow

Research question

Which combination of prefetching, speculative fetch, and cache warming reduces cold-cache P99 in an object-store-backed retrieval system most effectively per unit of memory, without simply provisioning the entire working set in RAM?

Why

Cache memory is the obvious lever and the expensive one; if a prefetch or warming strategy can reduce P99 at lower memory cost, it changes the unit economics that made the architecture attractive in the first place, and it addresses the tail that determines perceived quality.

S3 and object storage provide great durability but high latency (hundreds of milliseconds), requiring clever caching and reverse proxies.
Open source video
Vector databases face severe cost and latency challenges when relying solely on traditional cloud infrastructure.
Open source video
Source video

Cache state observability as the concrete face of state management

Research question

What minimum instrumentation — key-space census, hit-rate by shard, eviction and TTL distributions, dependency graph of what populates each key — is needed so that no production cache can become an undocumented 128GB liability?

Why

State management is identified as the recurring core difficulty, and the Redis anecdote is its most concrete manifestation; making cache contents a first-class observable turns failover, scaling, and debugging from archaeology into routine operations.

Scaling a Redis server with 128GB of RAM that nobody knew the contents of.
Open source video
Most gnarly engineering problems in large-scale systems boil down to state management and database consistency.
Open source video
Source video

Article ideas

4

Your Vector Database Is a Cache Problem, Not a Storage Problem

For read-heavy retrieval workloads, the durable tier can be commoditized to object storage, which means the engineering differentiator, the operational risk, and the on-call burden all migrate into cache coherence and tail latency — so teams adopting this pattern should staff, instrument, and hire for the cache tier, not the storage engine.

Angle

Architectural inversion: what changes about ownership, failure modes, and unit economics when the storage engine stops being the product.

Source video

Napkin Math Is an Architecture Gate, Not a Vibe Check

Computing memory bandwidth, object-store round trips, and NVMe/EBS throughput before selecting a topology is the cheapest available way to reject complexity, and the discipline only works if teams log predicted values against measured ones so the estimate error stays visible and bounded.

Angle

A pre-design checklist with worked numbers, plus a running predicted-versus-observed ledger.

Source video

Change Velocity Is an Architectural Property

The ability to re-shard or re-partition a system a week before peak traffic is determined by how few moving parts sit on the critical path, not by organizational heroics — which means late-stage optionality should be treated as an explicit design output of simplicity.

Angle

Simplicity as insurance: measuring the cost of a late architectural correction as a function of distributed coupling.

Source video

The 128GB Cache Nobody Understood

An unobserved cache is state management debt wearing a performance costume; if a team cannot enumerate what a cache holds, why each key is there, and how it is invalidated, the cache is a latent incident rather than an optimization.

Angle

Incident-narrative structure: the cache that absorbs load by day and blocks failover by night.

Source video

Project ideas

3

Cachefront Vector Store

gatehouse

An Nginx reverse-proxy cache in front of an S3-resident vector index sustains P99 under 50ms at at least 10x lower cost per million queries than a provisioned vector database for read-heavy workloads with a stable working set, and degrades predictably rather than catastrophically when the index is invalidated.

Proof of concept

Build an S3-backed HNSW/IVF index with an Nginx caching reverse proxy in front, load a fixed query set with a skewed distribution, then measure latency, hit ratio, and cost while injecting index updates to force cache invalidation.

Measurement

P50/P99 latency, cache hit ratio, dollars per 1M queries, and staleness window after an index update, compared against an in-memory vector database baseline.

Source video

Napkin Divergence Harness

movement-lab

Napkin-math estimates for memory bandwidth, S3 round-trip latency, and NVMe/EBS throughput diverge from measured values by more than 2x on at least one axis for a realistic vector workload, making them usable only as order-of-magnitude filters rather than as precise design gates.

Proof of concept

Record predicted values from a written napkin-math checklist before building, then instrument the built system to capture observed memory-bandwidth utilization, object-store round trips, and storage throughput, and diff the two per axis.

Measurement

Predicted-to-observed ratio per hardware axis, with variance across workload shapes and percentile levels (P50 vs P99).

Source video

Crossover Point Sweep

movement-lab

The object-storage-plus-cache topology ceases to outperform a distributed database beyond a definable write-ratio threshold and working-set-to-cache-capacity ratio, and the crossover is sharp enough to be encoded as a deployment decision rule.

Proof of concept

Parameter-sweep a single retrieval workload across write ratio (1 to 100 percent), working-set-to-cache-capacity ratio, and query locality skew, running each configuration against both the S3-plus-cache design and a distributed database baseline.

Measurement

Crossover thresholds for P99 latency and cost parity, and the steepness of degradation on either side of each threshold.

Source video

Architectural implications

2

The durability/latency split between object storage and query serving is addressed by inserting a caching tier rather than by changing the storage engine.

Before

Assume a vector database must own its storage layer to meet latency targets on cloud infrastructure.

After

Treat S3 as the durable source of truth and put reverse proxies (Nginx) plus caching in front of it to absorb latency.

Consequence

Cache invalidation, warm-up, and P99 tail behavior become the primary operational risks, and cost scales with object-storage economics rather than provisioned DB capacity.

Source video

Napkin math is used as a pre-design filter on hardware limits.

Before

Default to a distributed architecture when data or query volume is expected to be large.

After

First compute memory bandwidth, S3 round trips, and NVMe/EBS throughput to see whether a simpler design already clears the bar.

Consequence

Architecture reviews become quantitative and reject complexity that hardware numbers don't justify.

Source video

Tradeoffs and failure modes

4

Object storage as the substrate for vector search

Benefit

High durability and dramatically lower cost, with a claimed 95% reduction in customer cost via a caching layer over S3.

Cost or risk

S3 round trips are slow (hundreds of milliseconds), so correctness and latency now depend on a clever caching/reverse-proxy layer that must be right.

S3 and object storage provide great durability but high latency (hundreds of milliseconds), requiring clever caching and reverse proxies.
Open source video
Source video

Undocumented cached state in production

Benefit

Large in-memory caches (e.g., a 128GB Redis server) can absorb load cheaply.

Cost or risk

When nobody knows what the cache contains, the system carries an unobservable dependency that complicates scaling, failover, and debugging.

Scaling a Redis server with 128GB of RAM that nobody knew the contents of.
Open source video
Source video

Late-stage infrastructure changes under deadline

Benefit

Sharding deployed one week before Black Friday succeeded and let the system survive a traffic peak it otherwise could not have handled.

Cost or risk

Operating an untested topology change against a peak event risks catastrophic failure with no rollback window.

Database scaling required moving beyond single shards just before Black Friday peaks.
Open source video
Source video

Remote-first engineering culture

Benefit

Works exceptionally well for high-speed infrastructure teams when built on strong engineering trust and asynchronous communication.

Cost or risk

Requires trust and async discipline that the summary does not define or measure, so the conditions for success remain unspecified.

Remote culture works exceptionally well for high-speed infrastructure teams when built on strong engineering trust and asynchronous communication.
Open source video
Source video

Open questions

4

What is the consistency and invalidation model for a cache layer sitting over object storage that is the durable source of truth — and how is staleness bounded and observed?

Why unresolved

The summary describes the caching architecture and its cost benefit but says nothing about invalidation, coherence, or staleness handling.

Research direction

Prototype an S3-backed index with a reverse-proxy cache, then measure staleness windows, hit rates, and P99 latency under invalidation and cold-start conditions.

Source video

Do the napkin-math predictions (memory bandwidth, S3 round trips, NVMe/EBS throughput) actually match measured P99 latency and cost in a production vector workload?

Why unresolved

The summary presents napkin math as a design framework and separately claims a 95% cost reduction, but does not compare predicted versus measured numbers.

Research direction

Build a benchmark harness that records predicted vs. observed latency/cost for candidate architectures and tracks the divergence.

Source video

Under what access patterns does an object-storage-plus-cache design stop outperforming a distributed database — e.g., high write rates, heavy mutation, or low cache locality?

Why unresolved

The claim is that simple architectures 'often' outperform, but no boundary condition or workload envelope is given.

Research direction

Sweep workloads along write ratio, working-set size, and query distribution to locate the crossover point where the simple topology loses.

Source video

How do you systematically reduce tail latency (P99) when the durable tier is inherently slow, without simply over-provisioning cache memory?

Why unresolved

The summary lists P99 latency as a concern for vector databases and mentions caching and clustering/centroids, but does not describe a tail-latency strategy.

Research direction

Compare prefetching, speculative fetch, and cache-warming strategies against cold-cache tail latency in a controlled experiment.

Source video

Key claims

8
factualVerification needed

A caching layer over S3 reduced customer costs by 95% for vector search.

Evidence

Building a vector search caching layer over S3 that reduced customer costs by 95%.

Question

What was the baseline, the workload shape, and the measured latency change when this 95% cost reduction was achieved?

Source video
factualVerification needed

S3 and object storage provide great durability but high latency in the hundreds of milliseconds.

Evidence

S3 and object storage provide great durability but high latency (hundreds of milliseconds), requiring clever caching and reverse proxies.

Question

What round-trip latency distribution was measured for the specific object-storage access pattern used?

Source video
comparativeVerification needed

Simple architectures using commodity object storage plus smart caching often outperform complex distributed database topologies.

Evidence

Simple architectures that leverage commodity object storage (S3) combined with smart caching often outperform complex distributed database topologies.

Question

On which benchmarks and workloads does the simple S3-plus-cache topology beat a distributed database, and where does it lose?

Source video
opinionVerification not requested

Most hard problems in large-scale systems reduce to state management and database consistency.

Evidence

Most gnarly engineering problems in large-scale systems boil down to state management and database consistency.

Source video
causalVerification needed

Shopify's database scaling required moving beyond single shards just before Black Friday peaks.

Evidence

Database scaling required moving beyond single shards just before Black Friday peaks.

Question

What load signal forced the sharding decision, and what was the observed failure mode had sharding not been deployed?

Source video
causalVerification needed

Vector databases face severe cost and latency challenges when relying solely on traditional cloud infrastructure.

Evidence

Vector databases face severe cost and latency challenges when relying solely on traditional cloud infrastructure.

Question

Which specific cost and latency components dominate for a vector workload on a conventional cloud database?

Source video
opinionVerification needed

Remote culture works exceptionally well for high-speed infrastructure teams built on engineering trust and asynchronous communication.

Evidence

Remote culture works exceptionally well for high-speed infrastructure teams when built on strong engineering trust and asynchronous communication.

Question

What measurable outcomes (delivery speed, incident rates, attrition) support remote-first for infrastructure teams?

Source video
opinionVerification not requested

Raising venture capital should primarily fund R&D and provide employee liquidity rather than fuel ego or status.

Evidence

Raising venture capital should primarily fund R&D and provide employee liquidity rather than fuel ego or status.

Source video

Connections

4