The Knowledge Project Podcast · Published 2026-09-15

Shopify CEO: Most Businesses Are Using AI Wrong

privateReview: unreviewedConfidence: high
aiagentsshopifymetricspruningagent-architectureevaluation-metricsgoodharts-lawtechnical-debtHuman-in-the-loopGuardrailscontext-management
Open on YouTube ↗

Summary

Overview

  • Speaker: Tobi Lutke and Shane Parrish
  • Channel: The Knowledge Project Podcast
  • Main topic: AI implementation in business, software evolution, and decision-making frameworks
  • Purpose: To explore how AI transforms software architecture, business operations, and organizational decision-making. Shopify CEO Tobi Lutke discusses how modern businesses are misapplying AI, advocating for the concept of AI agents over traditional software applications. He explores software history, file systems, organizational decision-making, intuition versus feedback loops, and why pruning and rebuilding are essential for company longevity.

Topic Map

AI Agents vs Applications

  • Explanation: Modern AI usage in software is shifting from static tools to interactive agentic systems that act as colleagues.
  • Key claims:
    • Modern applications should be agents.
    • AI agents act as colleagues rather than static software tools.
    • Traditional software requires complex configuration files, whereas agents interact via natural language.
  • Examples:
    • Shopify's internal AI agent 'River' participating in meetings and checking code.
    • Microsoft's Sydney chatbot as an early milestone.
  • Terminology:
    • AI agent
    • sub-agent
    • LLM
  • Why it matters: Understanding agents changes how companies build software and interface with technology.

The Danger of Over-Fitting to Metrics

  • Explanation: Companies fail when they optimize too narrowly for quarterly stock prices or singular proxies of success.
  • Key claims:
    • Over-fitting to proxies destroys long-term viability.
    • Goodheart's Law states that when a measure becomes a target, it ceases to be a good measure.
    • Metrics are heuristics, not goals in themselves.
  • Examples:
    • Enron's collapse due to extreme metric optimization.
    • Shopify's Ruby poetry mode and focus on craftsmanship over short-term output.
  • Terminology:
    • Goodheart's Law
    • proxy
    • heuristic
    • over-fitting
  • Why it matters: Helps leaders avoid the trap of optimizing the wrong metrics at the expense of organizational health.

Pruning and Rebuilding Systems

  • Explanation: Systems, codebases, and companies need periodic pruning and rebuilding to stay efficient and avoid bloat.
  • Key claims:
    • You cannot make things better by simply adding stuff; you must prune and rebuild.
    • SpaceX's Raptor engine evolution demonstrates how removing unnecessary parts yields massive performance gains.
  • Examples:
    • SpaceX Raptor 1 vs Raptor 3 engine redesign.
    • Unix file system evolution replacing older computing paradigms.
  • Terminology:
    • pruning
    • rebuild
    • technical debt
  • Why it matters: Explains why successful organizations must ruthlessly prune legacy processes and code.

Key Points

AI as a Colleague, Not a Tool

  • Explanation: AI agents like River function as active collaborators with access to company code, sandboxes, and context.
  • Evidence: Tobi Lutke's description of how Shopify employees interact with AI agents daily.
  • Practical implication: Companies should design workflows around collaborative AI agents rather than rigid software interfaces.

The Illusion of Certainty

  • Explanation: Humans seek complex, certain answers to complex problems, but true understanding requires embracing complexity and intuition built from repetitions.
  • Evidence: Discussion on how business books oversimplify decisions into binary rights and wrongs.
  • Practical implication: Leaders must develop intuition through high-volume repetitions rather than relying solely on rigid frameworks.

The Slop Grenade Phenomenon

  • Explanation: When AI increases output without accountability, organizations get flooded with low-quality content and code, termed 'slop grenades'.
  • Evidence: Observations on Microsoft's early chatbot experience and organizational communication overload.
  • Practical implication: Guardrails and human accountability are necessary when scaling AI-generated outputs.

Frameworks, Models & Processes

Human-in-the-Loop Decision Surfaces

  • How it works: Creating organizational structures where AI assists with analysis and synthesis, but humans retain ultimate responsibility and judgment.
  • Components:
    • Autonomous AI agents for data gathering and synthesis
    • Cross-functional expert councils for deliberation
    • Human accountability for final decisions
  • When to use: When making complex, ambiguous strategic decisions in software and business.

Examples & Case Studies

Shopify's internal AI agent 'River' participates in Slack channels and code reviews.

  • Illustrates: The practical application of AI agents as interactive colleagues.
  • Lesson: AI can be integrated into everyday communication and development workflows.

SpaceX's Raptor 3 engine removed unnecessary pipes and complexity compared to Raptor 1.

  • Illustrates: The power of pruning and redesigning from first principles.
  • Lesson: Simplicity and removal of redundant components yield superior performance.

Actionable Takeaways

  • Immediate:
    • Experiment with AI agents as collaborative partners rather than static tools.
    • Evaluate whether your company's metrics are proxies or true goals.
    • Prune redundant software and processes instead of layering on complexity.
  • Strategic:
    • Build organizational resilience by fostering truth-seeking and long-term thinking.
    • Embrace the complexity of problem spaces rather than seeking false binaries.
    • Ensure human accountability remains central as AI automation scales.
  • Questions to investigate:
    • How are we measuring success, and is our metric a proxy or a goal?
    • Where are we adding complexity instead of pruning?
    • How can we integrate AI agents into our daily operational workflows?

Claims Worth Verifying

  • 30 million people use HeyGen for avatar generation. (statistic)
  • Goodheart's Law applies to corporate metric optimization. (economic principle)

Notable Quotes

"Things need to be pruned. You cannot make things better and better by adding stuff." (at 0:00) "The modern application is an agent." "Failure is never a problem unless it is catastrophic." (at 98:39)

Compressed Summary

  • AI agents should function as interactive colleagues rather than static tools.
  • Goodheart's Law warns against optimizing proxy metrics at the expense of real value.
  • Pruning and rebuilding from first principles drive superior performance and longevity.
  • Keywords: ai, agents, shopify, metrics, pruning
  • Core insight: Businesses must move beyond treating AI as a static tool and instead deploy accountable AI agents while avoiding the trap of proxy metric over-fitting.

Core insights

6
Architecturehigh noveltymoderate evidence

The unit of software is shifting from the configured application to the agent: instead of users editing configuration files, an agent is addressed in natural language and acts as a colleague with standing access to code, sandboxes, and organizational context (e.g., Shopify's 'River' in Slack channels and code reviews).

Why it matters

If the agent, not the app, is the interface, then the engineer's primary artifacts become context, tool/permission surfaces, and sandboxed execution environments rather than static UI and config schemas; access control and audit must be redesigned for non-human actors.

Generalization

For any internal tool, ask whether the deliverable is a deterministic UI/API or an agent holding identity, context, and sandboxed execution rights on shared systems.

The modern application is an agent.
Open source video
AI agents act as colleagues rather than static software tools.
Open source video
Traditional software requires complex configuration files, whereas agents interact via natural language.
Open source video
Shopify's internal AI agent 'River' participates in Slack channels and code reviews.
Open source video
Failure Modemedium noveltymoderate evidence

Scaling AI output without a matching increase in accountability produces 'slop grenades': organizations get flooded with low-quality content and code because generation capacity outruns review and ownership capacity.

Why it matters

It identifies throughput mismatch, not model quality, as the failure: the binding constraint moves from generation to review, ownership, and guardrails, which must be provisioned whenever generation is scaled.

Generalization

Any pipeline that raises generated artifact volume must scale verification and named ownership proportionally, or net artifact quality falls.

When AI increases output without accountability, organizations get flooded with low-quality content and code, termed 'slop grenades'.
Open source video
Guardrails and human accountability are necessary when scaling AI-generated outputs.
Open source video
Mental Modelmedium noveltymoderate evidence

Goodhart's Law is applied directly to operating metrics: a measure that becomes a target ceases to be a good measure, so metrics function as heuristics rather than goals. Extreme proxy optimization is linked to Enron's collapse.

Why it matters

This is the same dynamic that governs agent evaluation: once a benchmark or proxy score becomes the optimization target, it stops measuring the underlying capability, so eval design must anticipate self-degradation.

Generalization

Evaluation suites for agents and models are Goodhart-vulnerable; treat scores as proxies and pair them with rotating, harder, or outcome-grounded measures.

Goodheart's Law states that when a measure becomes a target, it ceases to be a good measure.
Open source video
Metrics are heuristics, not goals in themselves.
Open source video
Over-fitting to proxies destroys long-term viability.
Open source video
Enron's collapse due to extreme metric optimization.
Open source video
Practicemedium noveltymoderate evidence

Progress comes from pruning and rebuilding, not accumulation: removing unnecessary parts (SpaceX Raptor 3 vs Raptor 1) yields larger gains than adding capability, and only architectural rewriting can achieve it.

Why it matters

It inverts the default engineering incentive to add—more tools, more context, more prompts, more layers—and legitimizes deletion and periodic rebuild as first-class maintenance work with measurable performance payoff.

Generalization

Applies to agent stacks, tool inventories, context payloads, and codebases: measure whether an added component earns its maintenance cost, and schedule rebuilds rather than incremental accretion.

Things need to be pruned. You cannot make things better and better by adding stuff.
Open source video
You cannot make things better by simply adding stuff; you must prune and rebuild.
Open source video
SpaceX's Raptor 3 engine removed unnecessary pipes and complexity compared to Raptor 1.
Open source video
Architecturemedium noveltymoderate evidence

Human-in-the-loop decision surfaces: autonomous agents handle data gathering and synthesis, cross-functional expert councils deliberate, and a named human holds final accountability for the decision.

Why it matters

It draws an explicit boundary between synthesis (delegable to agents) and judgment/accountability (which must remain human and attributable), giving a concrete division-of-responsibility pattern for agentic decision systems.

Generalization

In any agentic decision pipeline, separate the synthesis layer from the judgment layer and attach an auditable human owner to the latter.

Creating organizational structures where AI assists with analysis and synthesis, but humans retain ultimate responsibility and judgment.
Open source video
Autonomous AI agents for data gathering and synthesis
Open source video
Human accountability for final decisions.
Open source video
Mental Modellow noveltyweak evidence

Business books and decision frameworks oversimplify complex problems into binary rights and wrongs, while actual competence comes from intuition built through high-volume repetitions rather than rigid frameworks.

Why it matters

It is a caution against substituting static rule sets or one-shot heuristics for feedback-driven iteration, which maps onto how agent behavior should be tuned (repeated trials and feedback) rather than hard-coded decision trees.

Generalization

Prefer systems that accumulate competence from repeated feedback cycles over systems that encode a fixed decision framework up front.

Humans seek complex, certain answers to complex problems, but true understanding requires embracing complexity and intuition built from repetitions.
Open source video
Leaders must develop intuition through high-volume repetitions rather than relying solely on rigid frameworks.
Open source video

Deep dives

5

Detecting Goodharting in agent evaluation suites

Research question

How can we instrument an agent evaluation suite so that it detects a proxy metric decoupling from the underlying capability before the decoupling silently propagates into engineering decisions?

Why

The summary elevates Goodhart's Law from a management aphorism to a design constraint, but supplies no detection mechanism. Without one, teams optimizing agent benchmarks will keep reporting rising scores while capability stalls, and there is no artifact they can point to that proves it happened.

Goodheart's Law states that when a measure becomes a target, it ceases to be a good measure.
Open source video
Metrics are heuristics, not goals in themselves.
Open source video
Over-fitting to proxies destroys long-term viability.
Open source video
Source video

Telemetry that separates useful AI throughput from 'slop grenades'

Research question

Which production telemetry signals (review latency, revert rate, ownership coverage per artifact, downstream consumption) separate net-useful AI-generated output from volume that merely transfers cost to reviewers?

Why

The failure mode is named qualitatively in the talk but has no threshold, metric, or measurement approach attached. Generation capacity is trivially scalable; review, ownership, and guardrail capacity are not. A measurable definition of the mismatch is the precondition for any guardrail investment.

When AI increases output without accountability, organizations get flooded with low-quality content and code, termed 'slop grenades'.
Open source video
Guardrails and human accountability are necessary when scaling AI-generated outputs.
Open source video
Source video

Blast-radius granularity for agents holding code and sandbox access

Research question

What sandboxing granularity (filesystem scope, network egress, repo scope, approval gates) minimizes incident severity without collapsing agent task completion rate?

Why

The talk licenses agent failure as acceptable 'unless it is catastrophic', which makes the containment boundary load-bearing rather than incidental. The summary asserts sandboxes and failure tolerance but never specifies the boundary conditions that make the tolerance safe.

Failure is never a problem unless it is catastrophic.
Open source video
Shopify's internal AI agent 'River' participates in Slack channels and code reviews.
Open source video
Source video

A decision rule for prune-versus-keep in agent stacks

Research question

Can a per-component maintenance-cost-versus-measured-contribution ratio predict which prompt layers, tools, and context chunks should be removed before an ablation experiment is run?

Why

Pruning is endorsed with a single anecdote and no cost model. Engineering incentives default to addition — more tools, more context, more prompt layers — so without a decision rule, the pruning advice is unactionable inside a sprint.

You cannot make things better by simply adding stuff; you must prune and rebuild.
Open source video
SpaceX's Raptor 3 engine removed unnecessary pipes and complexity compared to Raptor 1.
Open source video
Source video

Audit records for the synthesis/judgment boundary

Research question

What record should link agent-generated synthesis, council deliberation inputs, and the named human decision owner so that accountability survives an agent-mediated decision pipeline?

Why

The human-in-the-loop pattern from the talk names human accountability as a layer but describes no artifact carrying it. An accountability layer with no record is indistinguishable from an unaccountable one after the fact.

Creating organizational structures where AI assists with analysis and synthesis, but humans retain ultimate responsibility and judgment.
Open source video
Human accountability for final decisions.
Open source video
Source video

Article ideas

4

Slop Grenades: When Generation Capacity Outruns Review Capacity

The failure of enterprise AI adoption is not model quality but a throughput mismatch: generation scales instantly while review, ownership, and guardrails scale linearly, so every AI rollout that does not budget a corresponding verification layer converts its own productivity gain into downstream cost.

Angle

Treat AI-generated artifacts as a pipeline with a binding downstream constraint, and show that quality metrics should be computed per reviewer-hour consumed rather than per artifact produced.

Source video

Goodhart's Law Is an Eval Design Requirement, Not a Warning

Any benchmark your agent team optimizes against will stop measuring capability, so eval suites must be designed for their own decay — held-out pools, rotation schedules, and outcome-grounded anchors are structural requirements, not hygiene.

Angle

Reframe eval engineering as adversarial maintenance: assume your headline score is already Goodharted and design the instrumentation that will reveal it.

Source video

The Interface Was the Product: Agents Make Configuration Schemas Obsolete

If users address agents in natural language instead of authoring configuration, then the artifacts engineers maintain shift from UI and schema to context, tool surfaces, permissions, and action audit — and teams that keep investing in configuration expressiveness are optimizing a surface that is disappearing.

Angle

Trace the migration of engineering effort from schema design to identity-and-context design, and argue that permissioning must be rebuilt for non-human principals.

Source video

Delete More: Why Pruning Beats Addition in Agent Stacks

Agent stacks accrete prompt layers, tools, and context until the accretion itself is the bottleneck; the performance gains from removal (Raptor 3 vs Raptor 1) are unavailable to teams whose roadmap only contains additions, so rebuild must be budgeted as ordinary maintenance.

Angle

Argue that any component which cannot demonstrate measured contribution should be a deletion candidate, and that systems must be designed to be disposable.

Source video

Project ideas

4

Goodhart Sentinel

beyond-evals

Agent teams optimizing a fixed headline eval will show rising proxy scores while performance on a rotating held-out set stays flat or declines, and the divergence is detectable within a few optimization cycles.

Proof of concept

Run two agent configurations against a fixed public-style eval and a rotating private held-out eval over repeated tuning iterations; log both score curves and flag divergence points automatically.

Measurement

Delta between proxy score slope and held-out score slope across iterations, plus lead time between divergence onset and human detection without the tool.

Source video

Slop Meter

gatehouse

Net artifact quality is a function of generation volume divided by review-and-ownership capacity, so holding reviewer capacity constant while raising generation volume measurably degrades downstream quality signals.

Proof of concept

Instrument a repo or content pipeline to capture per-generated-artifact review latency, revert/reject rate, and named-owner coverage, then correlate each against weekly generation volume.

Measurement

Correlation between generation volume and revert rate per reviewer-hour; threshold at which revert rate per reviewer-hour inflects upward.

Source video

Blast Radius Bench

new

Sandbox granularity and agent task completion rate trade off non-linearly, so there exists a containment configuration that keeps incident severity non-catastrophic at acceptable completion cost.

Proof of concept

Benchmark an agent with repo and code access across escalating containment levels (filesystem scope, network egress, approval gates) on a fixed task suite; induce failures deliberately and score incident severity.

Measurement

Task completion rate per containment level plotted against worst-case incident severity; identify the containment frontier before completion collapses.

Source video

Prune-or-Keep Ablator

movement-lab

Most tools, prompt layers, and context chunks in a mature agent stack contribute less than their measured maintenance cost, so ablation-ordered removal improves performance per maintenance unit.

Proof of concept

For a working agent stack, define a per-component maintenance cost proxy, run leave-one-out removal experiments on each component, and rank components by contribution-to-cost ratio.

Measurement

Change in task success and latency per component removed versus maintenance cost proxy; count of components whose ratio falls below the keep threshold.

Source video

Architectural implications

4

The user-facing surface of an application becomes a natural-language agent rather than a configured UI.

Before

Static application plus complex configuration files that users must author.

After

An agent addressed in natural language that acts on the user's behalf.

Consequence

Configuration, permissioning, and observability shift from schema validation to context management, tool exposure, and agent-action audit.

Source video

Agents are given standing access to company code, sandboxes, and context as collaborators.

Before

Software is invoked as a bounded tool by a human operator with credentials.

After

Non-human collaborators hold identity and access inside shared systems (Slack, code review) with sandboxed execution.

Consequence

Identity, least-privilege scoping, and action logging must be designed for agents as first-class principals.

Source video

Decision-making is split into agent synthesis and human judgment within a defined surface.

Before

Humans perform analysis and decide, or the framework decides and humans comply.

After

Agents gather and synthesize; councils deliberate; a human owns the final decision.

Consequence

Accountability becomes an explicit architectural layer requiring attribution and audit rather than an implicit property of the operator.

Source video

Sustained performance depends on periodic rebuild rather than incremental addition.

Before

Add components, prompt layers, tooling, and features to improve the system.

After

Prune and rebuild from first principles when accretion becomes load-bearing.

Consequence

Rewrite/removal must be budgeted as normal maintenance, and systems must be designed to be disposable.

Source video

Tradeoffs and failure modes

4

Failure tolerance for autonomous agents

Benefit

Treating non-catastrophic failure as acceptable enables fast iteration and higher autonomy.

Cost or risk

The rule depends on a crisp, enforced line between tolerable and catastrophic failure; if near-misses accumulate unexamined, tolerance normalizes small errors that compound into the catastrophic outcome it was meant to prevent.

Failure is never a problem unless it is catastrophic.
Open source video
Source video

Scaling AI-generated output

Benefit

AI raises content and code output volume.

Cost or risk

Without matching accountability, output degrades into low-quality 'slop grenades' that consume downstream review capacity.

When AI increases output without accountability, organizations get flooded with low-quality content and code, termed 'slop grenades'.
Open source video
Source video

Optimizing proxy metrics

Benefit

Proxies are tractable and align short-term effort.

Cost or risk

The proxy stops measuring what it proxied once it becomes a target, destroying long-term viability (Enron cited).

Goodheart's Law states that when a measure becomes a target, it ceases to be a good measure.
Open source video
Source video

Pruning and rebuilding systems

Benefit

Removing unnecessary parts yields outsized performance gains, as in Raptor 3 vs Raptor 1.

Cost or risk

Rebuilds discard accumulated working machinery and investment; the summary gives no criterion for when the cost is justified.

You cannot make things better by simply adding stuff; you must prune and rebuild.
Open source video
Source video

Open questions

5

How do you detect that an agent evaluation metric has been Goodharted before it silently stops measuring capability?

Why unresolved

The summary establishes Goodhart's Law as a principle but gives no detection mechanism or instrumentation for it.

Research direction

Prototype held-out, rotating, or outcome-grounded eval sets and test whether agent behavior degrades against them while proxy scores keep rising.

Source video

What telemetry distinguishes useful AI-generated throughput from 'slop grenades' in a production organization?

Why unresolved

The failure mode is described qualitatively (volume without accountability) with no metric, threshold, or measurement approach proposed.

Research direction

Instrument review latency, revert rate, and ownership coverage per generated artifact and correlate with generation volume.

Source video

What containment boundary should an agent with code access and sandbox privileges have so that failures stay non-catastrophic?

Why unresolved

The summary asserts failure is acceptable unless catastrophic and mentions sandboxes, but does not specify blast-radius constraints.

Research direction

Test sandboxing granularity (filesystem, network, repo scope, approval gates) against agent task completion rate and incident severity.

Source video

How do you decide when a component—prompt layer, tool, context chunk, or subsystem—should be pruned versus kept?

Why unresolved

Pruning is endorsed with an anecdote (Raptor 3) but no criterion, cost model, or decision rule is supplied.

Research direction

Define per-component cost-of-maintenance versus measured contribution and run ablation-style removal experiments on agent stacks.

Source video

How is accountability attributed and audited at the human/agent boundary in a decision surface?

Why unresolved

The framework names human accountability for final decisions but does not describe the mechanism, record, or review artifact that carries it.

Research direction

Prototype an audit record linking agent-generated synthesis, deliberation inputs, and the named human decision owner.

Source video

Key claims

8
predictionVerification needed

Modern applications should be agents rather than static tools.

Evidence

Modern applications should be agents.

Question

Do agent-shaped internal tools outperform equivalent configured applications on adoption, task completion, or maintenance cost?

Source video
opinionVerification not requested

AI agents act as colleagues rather than static software tools.

Evidence

AI agents act as colleagues rather than static software tools.

Source video
comparativeVerification needed

Traditional software requires complex configuration files, whereas agents interact via natural language.

Evidence

Traditional software requires complex configuration files, whereas agents interact via natural language.

Question

Does natural-language interaction reduce configuration and onboarding cost versus declarative config for equivalent task coverage?

Source video
causalVerification needed

AI output volume without accountability floods organizations with low-quality content and code.

Evidence

When AI increases output without accountability, organizations get flooded with low-quality content and code, termed 'slop grenades'.

Question

Does increased AI generation volume measurably reduce artifact quality when review and ownership capacity are held constant?

Source video
causalVerification needed

When a measure becomes a target, it ceases to be a good measure (Goodhart's Law), and this applies to corporate metric optimization.

Evidence

Goodheart's Law applies to corporate metric optimization.

Question

Which commonly used corporate or AI evaluation metrics demonstrably decouple from the underlying objective once optimized against?

Source video
causalVerification needed

You cannot make things better by adding; you must prune and rebuild.

Evidence

You cannot make things better by simply adding stuff; you must prune and rebuild.

Question

Do periodic rebuilds of a system outperform incremental addition on performance per maintenance unit over the same time horizon?

Source video
comparativeVerification needed

SpaceX's Raptor 3 removed unnecessary pipes and complexity relative to Raptor 1, yielding performance gains.

Evidence

SpaceX's Raptor 3 engine removed unnecessary pipes and complexity compared to Raptor 1.

Question

What portion of Raptor 3's gains is attributable to part removal versus other design changes?

Source video
factualVerification needed

30 million people use HeyGen for avatar generation.

Evidence

30 million people use HeyGen for avatar generation. (statistic)

Question

What is the verified user count for HeyGen as of the claim's date?

Source video

Connections

5