Lenny's Podcast · Published 2026-09-24

How to run your product team like a research lab | Dan Shipper (Every)

privateReview: unreviewedConfidence: high
labs teamproduct managementai frontierresearch pipelinedogfoodinglabs-teamresearch-pipelineexplore-exploitorg-designpromotion-gatesevaluation-biasprototyping
Open on YouTube ↗

Summary

Overview

  • Speaker: Dan Shipper
  • Channel: Lenny's Podcast
  • Main topic: Running a product organization with research lab methodologies during technological revolutions
  • Purpose: To provide product leaders with a framework for balancing roadmap execution with frontier exploration during rapid technological revolutions. Dan Shipper, co-founder and CEO of Every, explores how product teams can adapt to rapid technological change by running part of their product organization like a research lab. He contrasts exploration and execution, introduces the concept of labs teams versus product teams, and explains pipeline management, team composition, and continuous feedback loops.

Topic Map

The Frontier Challenge

  • Explanation: Navigating the tension between executing a roadmap and exploring a rapidly moving technological frontier.
  • Key claims:
    • Product teams face the unreasonable inefficiency of business-as-usual during technology revolutions.
    • Execution and exploration are opposing ways of working that pull teams in opposite directions.
    • Never make any major life decisions within 30 days of a meditation retreat, psychedelic experience, or first encounter with a frontier model.
  • Examples:
    • Battle of Waterloo 3D historical reproduction generated with a single AI prompt.
    • Agent simulation with 1,000 agents running on a computer.
  • Terminology:
    • frontier model
    • exploration
    • execution
  • Why it matters: Product leaders must reconcile the need to execute established roadmaps with the imperative to explore unpredictable new capabilities.

The Labs Team Framework

  • Explanation: Separating concerns by creating a dedicated labs team to explore capabilities while product teams focus on execution and scaling.
  • Key claims:
    • Making a labs team separates concerns between improving existing work and exploring what comes next.
    • AI allows for a labs team of one with high leverage and minimal resource requirements.
    • Labs teams expect to dispose of 90 percent of what they make, while product teams adopt about 10 percent.
  • Examples:
    • Anthropic Labs and Claude Code originating from a small experimental team.
    • OpenAI desktop app scaling into ChatGPT.
  • Terminology:
    • labs team
    • embedded explorer
    • rotating assignment
  • Why it matters: It allows organizations to capture the upside of new technology without distracting the core product delivery team.

The Research Pipeline

  • Explanation: Managing ideas from lab-only experiments through internal use and early customers to scaling.
  • Key claims:
    • Ideas start on the left as lab-only experiments and move progressively right toward product teams.
    • Reviewing the pipeline regularly in all-hands meetings keeps lab and product teams in sync.
    • Defining clear decision criteria prevents shiny object syndrome.
  • Examples:
    • Automating Kate Lee's copy editing workflow using custom AI agents (KateBench).
    • Bringing in an architect like Yannick to build dashboards and metrics for copy edits.
  • Terminology:
    • research pipeline
    • dogfooding
    • KateBench
    • all-hands
  • Why it matters: It establishes a rigorous, repeatable process for turning raw AI experiments into scaled product features.

Key Points

Separate exploration from execution

  • Explanation: Trying to do both simultaneously causes organizational whovian friction because exploration is divergent and execution is convergent.
  • Evidence: Organizational tree diagrams showing split percentages between exploration and execution.
  • Practical implication: Create a distinct labs team with different expectations of success and disposal rates.

Keep labs teams very small

  • Explanation: In the AI era, a one-to-two person 'slice team' (pirates and architects) moves faster and avoids coordination overhead.
  • Evidence: Jeff Bezos's two-pizza team concept updated to the AI-era two-slice team.
  • Practical implication: Assign one or two versatile people to explore new model drops rather than forming large cross-functional squads.

Establish tight feedback loops via dogfooding

  • Explanation: The fastest way to know if an experiment is good is to use it yourself or test it with early adopters.
  • Evidence: Dan Shipper automating his editor Kate Lee's copy edits over three years.
  • Practical implication: Build internal tools for your own team first before exposing them to customers.

Frameworks, Models & Processes

The Labs Team vs. Product Team Framework

  • How it works: Splits an organization into a labs team focused on parallel exploration and high disposal rates, and product teams focused on scaling proven capabilities for customers.
  • Components:
    • Labs Team (Exploration, 90% disposal rate, small team)
    • Product Team (Execution, 10% adoption rate, scale-focused)
  • When to use: When operating on a fast-moving technological frontier where roadmap execution conflicts with exploratory R&D.

The Research Pipeline

  • How it works: A linear progression pipeline moving ideas from raw experiments to internal use, early customer testing, and final product scaling.
  • Components:
    • Lab-only
    • Internal use
    • Early customers
    • Ready to scale
  • When to use: To systematically filter and transition AI experiments into production products without organizational chaos.

Examples & Case Studies

Dan Shipper attempted to automate his editor-in-chief Kate Lee's copy editing workflow using custom AI tools.

  • Illustrates: The progression of an experiment from a private lab project (KateBench) to internal tool usage and measurable ROI.
  • Lesson: Dogfooding internal workflows creates tight feedback loops and proves value before customer rollout.

Anthropic Labs incubated Claude Code as an experimental project outside the main app before it became a major product.

  • Illustrates: How labs teams spawn breakout productivity tools.
  • Lesson: Small experimental groups can produce world-class products when given freedom to explore.

OpenAI experimented with desktop form factors before merging code-assist capabilities into ChatGPT.

  • Illustrates: Iterating across form factors in a labs pipeline.
  • Lesson: Rapid prototyping across multiple form factors leads to foundational product breakthroughs.

Actionable Takeaways

  • Immediate:
    • Set aside dedicated time for at least one person to explore new AI model drops.
    • Build internal tools for your own team's workflows first.
  • Strategic:
    • Structure your product org with distinct exploration and execution functions.
    • Implement a research pipeline with clear decision criteria and weekly pipeline reviews.
  • Questions to investigate:
    • Who on your team are the natural 'pirates' and 'architects' suited for a labs team?
    • What internal workflows are ripe for automated agent exploration right now?

Claims Worth Verifying

  • Codex was launched in February 2021 by a small team at OpenAI and later merged into ChatGPT. (historical fact)
  • Anthropic Labs created Claude Code as an internal experiment. (company practice)

Notable Quotes

"Rule number one is never make any major life decisions within 30 days of a meditation retreat, a psychedelic experience, or your first encounter with a frontier model." "The key question to ask yourself if you're running a product org is how do I harness the early adopters on my team on my team and maybe even some of my customers who are early adopters without distracting everybody else?" "On a labs team, your job is you're going to explore capabilities, especially when new models drop... and the expectation for a labs team is very different. On a labs team, you are going to expect to dispose of like 90% of what you make."

Compressed Summary

  • Separate exploration from execution by establishing dedicated labs teams.
  • Keep labs teams extremely small (one to two people: pirates and architects).
  • Manage ideas through a four-stage research pipeline: lab-only, internal use, early customers, and ready to scale.
  • Dogfood experiments internally to create tight feedback loops and prove ROI.
  • Keywords: labs team, product management, ai frontier, research pipeline, dogfooding
  • Core insight: To build products on a moving technological frontier without going crazy, product organizations must separate exploration from execution by running dedicated, small labs teams with high experimental disposal rates.

Core insights

5
Architecturemedium noveltystrong evidence

Exploration and execution are opposing work modes (divergent vs convergent) that should be split into separate organizational units with explicitly different success metrics: labs teams expect ~90% disposal of what they build, product teams adopt only ~10% of it. Success is not one metric applied uniformly, but a per-unit metric tuned to the unit's function.

Why it matters

Assigning a single success/quality metric across exploratory and production work corrupts both: explorers get punished for correctly discarding dead ends, and production teams get distracted by unproven artifacts. Defining a high-disposal metric for the exploration side legitimizes failed experiments as expected output.

Generalization

Any system that both searches a large option space and ships a stable artifact should separate the search path from the serving path and instrument them with different KPIs (learning rate vs reliability/uptime).

Labs teams expect to dispose of 90 percent of what they make, while product teams adopt about 10 percent.
Open source video
Trying to do both simultaneously causes organizational whovian friction because exploration is divergent and execution is convergent.
Open source video
Mechanismhigh noveltymoderate evidence

AI leverage collapses the minimum viable exploration team to one or two people ('slice team' of pirates and architects), superseding the two-pizza team heuristic. The binding constraint shifts from headcount to coordination overhead, so smaller units move faster on new model drops.

Why it matters

If one or two people can run a high-leverage experiment, the cost of spinning up a parallel exploration workstream drops dramatically, making many small parallel bets affordable where one large squad previously was. It also implies the scarce resource is versatile generalists, not staffing volume.

Generalization

When per-agent leverage rises, optimal team size falls until coordination cost exceeds marginal capacity; orchestration boundaries should be drawn to minimize cross-agent synchronization.

AI allows for a labs team of one with high leverage and minimal resource requirements.
Open source video
In the AI era, a one-to-two person 'slice team' (pirates and architects) moves faster and avoids coordination overhead.
Open source video
Mechanismmedium noveltystrong evidence

Ideas advance through an explicit stage-gated pipeline (lab-only -> internal use -> early customers -> ready to scale), with predefined decision criteria and regular review at all-hands to keep the exploration and execution units synchronized.

Why it matters

A named, ordered set of promotion stages with explicit criteria converts ad-hoc 'this demo looked cool' judgments into a repeatable gate, and a recurring sync ritual prevents drift between the units that produce and the units that consume the artifacts.

Generalization

Promotion between environments (prototype -> internal -> canary -> GA) should be gated by fixed criteria and reviewed on a cadence, not decided by whoever is most excited.

Ideas start on the left as lab-only experiments and move progressively right toward product teams.
Open source video
Defining clear decision criteria prevents shiny object syndrome.
Open source video
Reviewing the pipeline regularly in all-hands meetings keeps lab and product teams in sync.
Open source video
Practicemedium noveltymoderate evidence

Internal dogfooding is the cheapest and fastest validation stage: build the tool for your own team's workflow first, run it on a real recurring internal task, and only then expose it to customers. The Kate Lee copy-editing automation progressed over three years from private project to instrumented internal tool.

Why it matters

Internal use generates ground-truth signal (does it actually help someone do the job) far cheaper than a customer rollout, and it surfaces failure modes under realistic use without external blast radius. The progression also shows value must be instrumented (dashboards/metrics) to justify promotion.

Generalization

Maintain a private/staging deployment of a new capability against a real internal workload before user-facing exposure; instrument it to produce promotion evidence.

The fastest way to know if an experiment is good is to use it yourself or test it with early adopters.
Open source video
Build internal tools for your own team first before exposing them to customers.
Open source video
Mental Modelhigh noveltyweak evidence

First exposure to a new frontier model produces unreliable judgments, analogous to post-retreat or post-psychedelic euphoria: the speaker's rule is to avoid major decisions within 30 days of a first encounter with a frontier model. Novelty itself distorts capability assessment.

Why it matters

It identifies a concrete evaluation bias: capability assessments made at first contact systematically overestimate what the model can do reliably, leading to premature architectural commitments. A cooling-off window before decisions is a cheap debiasing mechanism.

Generalization

Evaluation protocols should separate 'wow' exploration from commitment decisions, requiring repeated measurements over time before locking in dependencies or re-architecting around a new capability.

Never make any major life decisions within 30 days of a meditation retreat, psychedelic experience, or first encounter with a frontier model.
Open source video

Deep dives

5

Promotion-gate rubrics and telemetry for the lab-to-product pipeline

Research question

Which pre-declared, written criteria and instrumented signals reliably predict that a lab experiment deserves promotion from lab-only to internal use to early customers to scale?

Why

Without written criteria, promotion defaults to whoever is most excited; with them, promotion becomes auditable and repeatable and both producing and consuming teams share one view of what is in flight. This is the single missing mechanism that makes the whole labs/product split operational rather than rhetorical.

Defining clear decision criteria prevents shiny object syndrome.
Open source video
Ideas start on the left as lab-only experiments and move progressively right toward product teams.
Open source video
Reviewing the pipeline regularly in all-hands meetings keeps lab and product teams in sync.
Open source video
Source video

Instrumenting the 90/10 disposal ratio with an explicit denominator

Research question

On what denominator (ideas, prototypes, shipped experiment-weeks) and over what time window should the ~90% disposal rate be counted, and does the measured ratio predict delivered value?

Why

The 90/10 figure is stated as an expectation but the summary gives no counting rule, making the ratio unverifiable as stated. Until it is instrumented it cannot function as an exploration budget or as a guardrail against sunk cost.

Labs teams expect to dispose of 90 percent of what they make, while product teams adopt about 10 percent.
Open source video
Ideas start on the left as lab-only experiments and move progressively right toward product teams.
Open source video
Source video

Whether the 1-2 person slice team remains optimal as per-agent leverage grows

Research question

Does the one-to-two person slice team sustain higher throughput than a 4-8 person squad on matched exploration tasks as model generations advance, or does coordination tooling change the optimum?

Why

If the optimum team size is a function of per-agent leverage, then org design must be re-derived each model generation rather than fixed. This determines how many parallel exploration bets an organization can afford.

AI allows for a labs team of one with high leverage and minimal resource requirements.
Open source video
Anthropic Labs and Claude Code originating from a small experimental team.
Open source video
Source video

First-contact novelty bias in frontier-model capability assessment

Research question

How much do capability assessments made at first exposure to a new frontier model overestimate sustained task reliability compared with repeated measurements after a cooling-off window?

Why

It identifies a concrete, cheap-to-control evaluation bias: premature architectural commitments made on day-zero enthusiasm do not survive sustained use. A cooling-off rule only helps if the size of the bias is measured.

Never make any major life decisions within 30 days of a meditation retreat, psychedelic experience, or first encounter with a frontier model.
Open source video
Source video

The promotion path from internal-first dogfood tool to customer product

Research question

When an internal-first tool outgrows the organization's own workflow, what fraction of internal assumptions hold when it is exposed to external customers?

Why

Internal use is the cheapest validation stage but it is validated against one workflow with one set of users; the summary stops the KateBench example at internal dashboards and metrics, leaving the internal-to-customer transition path unspecified.

Build internal tools for your own team first before exposing them to customers.
Open source video
Automating Kate Lee's copy editing workflow using custom AI agents (KateBench).
Open source video
Bringing in an architect like Yannick to build dashboards and metrics for copy edits.
Open source video
Source video

Article ideas

4

The 90% Disposal Rate Is a Metric, Not a Culture Statement

The lab disposal rate only works as an exploration budget if it has a declared denominator, a counting rule, and instrumentation; until then it is a slogan that legitimizes waste without producing learning-rate signal.

Angle

Measurement critique of a widely repeated org-design heuristic

Source video

Your First Week With a New Frontier Model Is the Worst Time to Re-Architect

Capability assessments taken at first contact with a frontier model systematically overestimate reliable capability, so commitment decisions should require repeated measurement across a cooling-off window, not a compelling demo.

Angle

Evaluation bias and the architectural commitments it causes

Source video

Two-Pizza Teams Are Obsolete, and the Replacement Constraint Is Coordination, Not Headcount

When per-person leverage rises, coordination overhead becomes the binding constraint on exploration throughput, so the optimal exploration unit shrinks to one or two generalists until coordination cost exceeds marginal capacity.

Angle

Org-design argument with a stated boundary condition for when larger teams become preferable again

Source video

Dogfooding Is a Canary Deployment With a Real Workload

Running a new capability against your own team's recurring workflow is the cheapest validation stage available, but it only counts as promotion evidence if the internal usage is instrumented to produce measurable signal.

Angle

Reframing internal tooling as a staged deployment rather than a perk

Source video

Project ideas

4

Gatehouse Ledger

gatehouse

Teams that pre-declare per-stage promotion criteria and log every artifact's stage transitions promote fewer experiments that are rolled back or abandoned within 30 days of customer exposure than teams deciding promotion case-by-case.

Proof of concept

A lightweight artifact registry with four named stages (lab-only, internal, early customer, scale), mandatory written criteria fields for each transition, and an all-hands review view that shows stage age for every artifact in flight.

Measurement

Rate of artifacts reaching the early-customer stage that are abandoned or rolled back within 30 days, compared against a criteria-free baseline; median time-in-stage per artifact before promotion.

Source video

Cooling-Off Eval

beyond-evals

Capability scores for a newly released frontier model measured on day 0 overestimate sustained task reliability relative to scores measured on the same tasks at day 30.

Proof of concept

Run a fixed task suite with blind grading against a newly released model at day 0, 7, 14, and 30, holding prompts and tasks constant, and record both pass rate and time-to-completion.

Measurement

Delta in pass rate and time-to-completion between day-0 and day-30 runs on identical tasks, and correlation of the day-0 score with subsequent production incident rate for features built on that model.

Source video

Disposal Ledger

new

Making the 90/10 ratio computable at artifact level (artifacts created versus artifacts absorbed per period) changes which experiments get funded and predicts delivered value, whereas an unmeasured disposal budget does not.

Proof of concept

Automated lifecycle tracking that counts each experiment artifact at creation, at each stage transition, and at disposal, rolled up into a weekly created-versus-absorbed ratio per team.

Measurement

Correlation between the measured disposal ratio and period-over-period delivered value, and whether the ratio's presence changes funding decisions in a before/after comparison.

Source video

Slice Team Bench

movement-lab

On matched exploration tasks, one-to-two person slice teams reach a first working demo faster than four-to-eight person teams, and the gap widens across successive model generations.

Proof of concept

Pair matched exploration tasks and assign one to a 1-2 person slice team and one to a 4-8 person squad, repeated across two model generations with the same task framing.

Measurement

Time-to-first-working-demo and demo-to-promotion rate per team size, tracked across generations to test whether the gap widens or narrows.

Source video

Architectural implications

3

Labs artifacts are expected to be thrown away at ~90%, so the exploration surface must be cheap to build and cheap to discard, while the production surface must be stable and instrumented.

Before

One shared codebase/environment where experiments and production features compete for the same reliability guarantees and the same success metrics.

After

Two surfaces: a disposable exploration path optimized for iteration speed, and a hardened serving path that receives only artifacts that passed explicit promotion criteria.

Consequence

Reduces the blast radius of failed experiments and prevents experimental code from accruing the maintenance burden of production code.

Source video

The pipeline is a promotion ladder with four named stages, which is structurally the same shape as prototype -> internal -> canary -> GA.

Before

Implicit promotion decided case-by-case, driven by enthusiasm ('shiny object syndrome').

After

Fixed stage definitions plus written criteria for advancing between them, reviewed on a regular cadence with both producing and consuming teams present.

Consequence

Promotion decisions become auditable and repeatable, and both sides share a single view of what is in flight and why it is or isn't advancing.

Source video

Small slice teams (1-2 people, pirates plus architects) are the exploration unit; larger cross-functional squads carry coordination overhead that dominates on frontier work.

Before

Cross-functional squad assembled per initiative regardless of scope.

After

Minimal team of generalists (an explorer and a builder) chartered per exploration thread, scaled up only after promotion.

Consequence

More parallel threads become affordable, but the organization must tolerate low-legibility work in progress and depends on a scarce generalist profile.

Source video

Tradeoffs and failure modes

4

Splitting exploration from execution

Benefit

Protects the roadmapped delivery team from exploratory churn and lets the labs team optimize for learning velocity with a high disposal rate.

Cost or risk

Coordination overhead and drift between the two units; without a regular sync (all-hands pipeline review) the labs team produces artifacts the product team cannot absorb or does not want.

Reviewing the pipeline regularly in all-hands meetings keeps lab and product teams in sync.
Open source video
Source video

Very small labs teams

Benefit

Faster movement and lower coordination overhead; a single person can achieve high leverage on new model capabilities.

Cost or risk

Coverage is limited and the organization becomes dependent on individual generalists ('pirates and architects'), concentrating knowledge and creating key-person risk.

In the AI era, a one-to-two person 'slice team' (pirates and architects) moves faster and avoids coordination overhead.
Open source video
Source video

High tolerable disposal rate (~90%)

Benefit

Encourages wide parallel experimentation and makes abandoning dead ends a normal outcome rather than a failure.

Cost or risk

Sunk cost accrues in discarded work, and a 90% discard rate requires cultural tolerance for apparent waste that many organizations cannot sustain; the 10% that survives must carry all the value.

Labs teams expect to dispose of 90 percent of what they make, while product teams adopt about 10 percent.
Open source video
Source video

Judging new frontier models at first contact

Benefit

Fast response to genuinely large capability jumps; early experiments capture upside before competitors.

Cost or risk

Novelty distorts assessment, producing premature architectural commitments and inflated expectations that do not survive sustained use.

Never make any major life decisions within 30 days of a meditation retreat, psychedelic experience, or first encounter with a frontier model.
Open source video
Source video

Open questions

4

What concrete criteria promote an artifact from lab-only to internal use to early customers to ready-to-scale?

Why unresolved

The summary asserts that clear decision criteria prevent shiny object syndrome but does not state what the criteria are or how they are measured.

Research direction

Define and test a written promotion rubric per stage (e.g., recurring internal usage frequency, measured time saved, reliability under load) and compare promotion outcomes against a criteria-free control.

Source video

How is a ~90% disposal rate actually measured and on what denominator (ideas, prototypes, shipped experiment-weeks)?

Why unresolved

The 90/10 figure is stated as an expectation but the summary gives no counting rule or instrumentation, making the ratio unverifiable as stated.

Research direction

Instrument experiment lifecycle tracking to count artifacts created versus artifacts absorbed per period, and test whether the ratio predicts delivered value.

Source video

Does the one-to-two person slice team remain optimal as the frontier model capability curve flattens or as coordination tooling for larger agent fleets improves?

Why unresolved

The claim is tied to the AI era's current leverage profile; the summary offers no boundary condition at which larger teams become preferable again.

Research direction

Compare throughput of 1-2 person teams versus 4-8 person teams on matched exploration tasks across successive model generations.

Source video

How should the pipeline be sequenced when an internal dogfood tool outgrows the organization's own workflow and must be productized for external customers?

Why unresolved

The KateBench example stops at internal use with dashboards and metrics; the transition path from internal tool to customer product is not described.

Research direction

Study cases where an internal-first tool was promoted to customer-facing status and measure what fraction of internal assumptions held externally.

Source video

Key claims

7
opinionVerification needed

Labs teams expect to dispose of roughly 90% of what they make, while product teams adopt roughly 10%.

Evidence

Labs teams expect to dispose of 90 percent of what they make, while product teams adopt about 10 percent.

Question

What is the denominator and time window for the 90/10 ratio, and does it hold across organizations that adopt this split?

Source video
causalVerification needed

A labs team of one is viable in the AI era because individual leverage is high and resource requirements are minimal.

Evidence

AI allows for a labs team of one with high leverage and minimal resource requirements.

Question

What tasks can a one-person labs team complete that previously required a full cross-functional squad, and at what quality and reliability?

Source video
factualVerification needed

Anthropic Labs incubated Claude Code as an experimental project outside the main app before it became a major product.

Evidence

Anthropic Labs incubated Claude Code as an experimental project outside the main app before it became a major product.

Question

Was Claude Code originally developed as an internal experiment outside Anthropic's main product surface, and what was the timeline?

Source video
factualVerification needed

Codex was launched in February 2021 by a small team at OpenAI and later merged into ChatGPT.

Evidence

Codex was launched in February 2021 by a small team at OpenAI and later merged into ChatGPT.

Question

What was the actual launch date of Codex and how did its capabilities merge into ChatGPT?

Source video
causalVerification needed

Trying to run exploration and execution simultaneously within the same unit causes organizational friction because the two modes are divergent and convergent respectively.

Evidence

Trying to do both simultaneously causes organizational whovian friction because exploration is divergent and execution is convergent.

Question

Does separating the two functions measurably improve delivery velocity or learning rate versus a combined team?

Source video
causalVerification needed

Reviewing the research pipeline regularly in all-hands meetings keeps lab and product teams in sync.

Evidence

Reviewing the pipeline regularly in all-hands meetings keeps lab and product teams in sync.

Question

What review cadence and artifact format most effectively reduce handoff failures between exploration and execution teams?

Source video
causalVerification needed

Defining clear decision criteria prevents shiny object syndrome in the research pipeline.

Evidence

Defining clear decision criteria prevents shiny object syndrome.

Question

Do pre-declared promotion criteria reduce premature scaling of experiments compared with case-by-case judgment?

Source video

Connections

4