From Arc to Dia: Lessons learned building AI Browsers – Samir Mody, The Browser Company of New York

Overview

• Speaker: Samir Mody
• Channel: AI Engineer
• Main topic: Lessons learned in building AI-native browsers from Arc to Dia
• Purpose: To share engineering and product development lessons learned while building AI-native web browsers at The Browser Company. Samir Mody, Head of Engineering for AI at The Browser Company of New York, shares the journey of transitioning from building the Arc browser to building Dia, an AI-native browser. He discusses four key lessons learned: optimizing tools and processes for fast iteration, treating model behavior as a craft and discipline, AI security as an emergent property of product building, and adapting company culture and team structure alongside technology shifts.

Topic Map

Introduction and Background of The Browser Company

• Explanation: Introduction of the speaker Samir Mody and the founding mission of The Browser Company to rethink how people use the internet via the browser.
• Key claims:
  • The browser is one of the most important pieces of software in your life.
  • The way people use the browser has changed over decades, but the browser itself hasn't.
  • Arc was shipped in 2022 as an incremental step; Dia was shipped in early 2024 as an AI-native browser.
• Examples:
  • Arc browser launched in 2022 to make the internet more personal and organized.
  • Dia launched in early 2024 as an AI-native browser built from the ground up.
• Terminology:
  • Arc
  • Dia
  • The Internet Computer
  • LLMs
• Why it matters: Establishes the product vision and the motivation behind moving from a traditional browser (Arc) to an AI-native browser (Dia).

Lesson 1: Optimize your tools and process for fast iteration

• Explanation: Building AI-native products requires specialized internal tooling and processes to enable rapid prototyping, evaluation, data collection, and hill-climbing.
• Key claims:
  • Iteration speed is critical when building AI products.
  • Internal prototyping tools transformed from rudimentary prompt editors in dev builds to integrated product features.
  • Embedding tools into the product itself allowed everyone in the company (PMs, designers, ops) to ideate and iterate.
• Examples:
  • Building an internal prompt editor accessible only in dev builds.
  • Integrating prompt and parameter editing directly into Dia for all team members.
• Terminology:
  • Prototyping
  • Evals
  • Hill-climbing
  • Dogfooding
• Why it matters: Demonstrates how internal tooling accelerates product velocity and broadens who can contribute to AI product features.

GEPA: Evolve and optimize your prompts

• Explanation: Using reflective prompt mutation and Pareto selection (GEPA) to sample-efficiently improve LLM systems without relying solely on RL or fine-tuning.
• Key claims:
  • Reflective prompt mutation can outperform reinforcement learning for prompt optimization.
  • Pareto selection allows exploring more of the prompting space rather than a single avenue.
  • Automated hill-climbing on prompts is as important as initial ideation.
• Examples:
  • Seeding the system with an initial prompt, executing on tasks, scoring, and using an LLM to reflect and generate new prompts.
• Terminology:
  • GEPA
  • Reflective Prompt Mutation
  • Pareto Selection
  • Sample Efficiency
• Why it matters: Provides a practical automated approach to prompt engineering and optimization for small teams.

Lesson 2: Treat Model Behavior as a craft and discipline

• Explanation: Model behavior encompasses behavior design, measurement and training via evals, and model steering, moving beyond simple prompting.
• Key claims:
  • Model behavior defines, evaluates, and ships the desired behavior of models.
  • Just as web design evolved in the early 2000s from functional logic to product design, AI product design is evolving from functional prompts to agentic behaviors.
  • Building a dedicated model behavior team drastically improved product quality.
• Examples:
  • A strategy/ops team member rewriting prompts over a weekend using internal prompt tools to radically improve product capability.
• Terminology:
  • Model Behavior
  • Behavior Design
  • Model Steering
  • Agentic Behavior
• Why it matters: Highlights that managing model outputs requires treating AI behavior as a professional discipline and involving non-engineers in the process.

Lesson 3: AI Security is an emergent property of product building

• Explanation: Browsers sit at the intersection of private data, untrusted content, and external communication, making prompt injections a critical security challenge.
• Key claims:
  • Browsers sit at the center of the 'lethal trifecta': access to private data, exposure to untrusted content, and ability to externally communicate.
  • Prompt injections can cause data exfiltration, execution of malicious commands, or bypassing safety rules.
  • Technical mitigations like wrapping untrusted context in tags or role separation help, but user awareness, control, and confirmation steps are essential.
• Examples:
  • An LLM summarizing a malicious website that contains a hidden prompt injection directing it to extract personal data and embed it in a URL.
  • Autofill tool in Dia requiring user review and confirmation of plaintext data before submitting forms.
• Terminology:
  • Prompt Injection
  • Lethal Trifecta
  • Data Exfiltration
  • System/User Roles
• Why it matters: Addresses the critical security vulnerabilities inherent in AI-powered browsers and how to design products to maintain user trust.

Conclusion: Technology Shift to Product Evolution Company

• Explanation: A technology shift is ultimately a company shift that requires embracing new ways of hiring, communicating, and collaborating.
• Key claims:
  • Adapting to AI is not just a product evolution; it is a company evolution.
  • Technology shifts must be embraced with conviction.
• Examples:
  • The formation of cross-functional model behavior teams and integrated tooling across departments.
• Terminology:
  • Technology Shift
  • Product Evolution
  • Company Evolution
• Why it matters: Provides a high-level philosophical takeaway on organizational transformation during technological revolutions.

Key Points

Optimize internal tools for fast iteration

• Explanation: Building AI features requires internal prompt editors, eval runners, and data collection pipelines integrated directly into the product.
• Evidence: Internal tooling scaled iteration speed 10x and enabled non-engineers to create and refine AI features.
• Practical implication: Teams should invest in custom internal authoring tools for prompts and context management early.

Treat model behavior as a structured discipline

• Explanation: Model behavior encompasses behavior design, rigorous evaluations, and model steering.
• Evidence: Evolution from functional prompts (early 2020s) to agentic behaviors (today) mirrors the evolution of web design.
• Practical implication: Involve strategists, PMs, and operators in prompt authoring and model steering, not just engineers.

Mitigate prompt injection through user-in-the-loop design

• Explanation: Because browsers combine private data, untrusted web content, and external communication, prompt injection is a severe risk.
• Evidence: Browsers sit at the lethal trifecta; technical fixes like XML wrapping are insufficient without user confirmation steps.
• Practical implication: Always require explicit user review and confirmation before AI agents execute sensitive actions or fill forms.

Frameworks, Models & Processes

Model Behavior Framework

• How it works: A continuous iterative loop combining product requirements, solution identification, implementation (prompts, context, model switching), writing evals, testing evals, shipping, collecting feedback, and refining.
• Components:
  • Behavior Design
  • Measurement and Training
  • Model Steering
• When to use: When developing and refining complex LLM-powered assistant products.

GEPA (Reflective Prompt Evolution)

• How it works: Seeds the system with an initial candidate prompt, executes and scores on a task set, selects top prompts via Pareto selection, and uses an LLM to reflect and generate mutated prompts.
• Components:
  • Seed candidate prompt
  • Execute and score
  • Pareto selection
  • Reflective prompt generation
• When to use: To optimize complex system prompts and workflows without expensive fine-tuning or reinforcement learning.

Examples & Case Studies

A strategy and ops team member used internal prompt editing tools over a weekend to rewrite prompts across Dia.

• Illustrates: Empowering non-engineers with internal AI tooling unlocks massive creativity and product quality.
• Lesson: Build internal tools that allow everyone in the company to iterate on AI behaviors.

An LLM tasked with summarizing a web page encountered a hidden prompt injection instructing it to exfiltrate user data into a URL.

• Illustrates: The danger of prompt injection in AI browsers sitting at the 'lethal trifecta'.
• Lesson: Implement UI safeguards like plain-text user confirmation steps before sensitive actions occur.

Actionable Takeaways

• Immediate:
  • Build internal prompt and eval tools for your team.
  • Separate system instructions from untrusted user content using roles and randomized tags.
  • Require user confirmation steps before AI agents execute external actions.
• Strategic:
  • Treat model behavior as a core product discipline.
  • Empower non-engineers (designers, PMs, ops) to participate in prompt engineering.
  • Embrace technology shifts with organizational and cultural conviction.
• Questions to investigate:
  • How can we further automate prompt optimization using reflection and Pareto selection?
  • What are robust architectural defenses against indirect prompt injection in browser agents?

Claims Worth Verifying

• Reflective Prompt Evolution (GEPA) can outperform reinforcement learning for prompt optimization. (Research claim)
• Browsers sit at the 'lethal trifecta' of private data access, untrusted content exposure, and external communication. (Security framework)

Notable Quotes

"Simply put, the way we've used the browser has changed over the last couple decades, but the browser itself hadn't." (at 0:55) "AI Security is an emergent property of product building." (at 20:55) "When you recognize that technology shift, you have to embrace it and you have to embrace it with conviction." (at 28:44)

Compressed Summary

• Optimized internal tools enabled 10x faster iteration and enabled non-engineers to build AI features.
• Model behavior must be treated as a craft involving behavior design, evals, and steering.
• AI security in browsers requires defending against prompt injection through role separation and user-in-the-loop confirmations.
• Technological shifts require complete company and cultural evolution.
• Keywords: ai browser, prompt engineering, model behavior, prompt injection, internal tooling
• Core insight: Building AI-native products requires optimizing iteration tools, treating model behavior as a cross-functional craft, and engineering robust security against prompt injection.