Building secure agents for knowledge work

Overview

• Speaker: Katelyn Lesse, Willie Williams, Dan Shipper
• Channel: Claude
• Main topic: Building secure and effective AI agents for knowledge work using Claude and Every's AI co-worker platform
• Purpose: To explore how AI agents can transform knowledge work, the importance of security and isolation, the challenges of team-wide AI adoption, and how platform-level primitives enable better human-AI collaboration. A discussion between Anthropic's Head of Platform Engineering Katelyn Lesse, Every's Head of Platform Willie Williams, and Every's Co-Founder & CEO Dan Shipper on building secure AI agents for knowledge work, utilizing platform capabilities, and bridging the gap between generic AI benchmarks and practical, customized knowledge work workflows.

Topic Map

Changing Nature of Knowledge Work and AI Benchmarks

• Explanation: Discussion on how knowledge work tools are evolving and why pre-AI benchmarks do not accurately reflect real-world productivity gains.
• Key claims:
  • Knowledge work tools are changing rapidly.
  • Benchmarks measure pre-AI tasks and do not reflect individual job enhancement.
  • Engineering teams scale faster with AI agents, a pattern coming to general knowledge work.
• Examples:
  • Engineering teams using AI agents to pull forward roadmaps by two years.
• Terminology:
  • benchmark progress
  • knowledge work
  • pre-AI tasks
• Why it matters: Understanding how to measure AI's actual impact on human productivity versus abstract testing.

What is Every and the Every Agent

• Explanation: Introduction to Every as a media and technology company building subscriptions and AI co-workers like the Every Agent.
• Key claims:
  • Every builds subscriptions for staying at the edge of AI.
  • The Every Agent is designed to be the easiest way to AI-pill an entire company.
  • Deploying agents across an entire organization is much harder than single-user usage.
• Examples:
  • Open enrollment process handled through the Every Agent.
• Terminology:
  • Every agent
  • AI ambassador
  • open enrollment
  • media and technology company
• Why it matters: Highlights the transition from individual AI tool usage to company-wide AI agent orchestration.

Kate Bench and Automating Editorial Standards

• Explanation: How Every uses 'Kate Bench' (named after their editor-in-chief) to codify editorial taste and automate top edits.
• Key claims:
  • Kate Bench measures how well models match editorial standards.
  • Historical edits are turned into skills inside the Every agent.
  • Automating copy edits saves time while preserving human editorial standards.
• Examples:
  • Turning 30,000 historical edits into a skill inside the Every agent for Google Docs editing.
• Terminology:
  • Kate bench
  • top edits
  • editorial standards
  • skill
  • vibe checks
• Why it matters: Demonstrates how subjective human taste and editorial style can be encoded into AI workflows.

Infrastructure and Cloud Managed Agents

• Explanation: Why Every chose Cloud Managed Agents for building their agent platform to handle sandboxing, memory, and session management.
• Key claims:
  • Infrastructure is hard, and building multi-agent systems has hidden complexities.
  • Cloud Managed Agents provide sandboxes, memory, and session control out of the box.
  • Running agents on the same infrastructure as production speeds up development.
• Examples:
  • Isolating sandboxes and tool calls for testing experiments without impacting production.
• Terminology:
  • Cloud Managed Agents
  • sandboxes
  • session control
  • infra layer
  • primitives
• Why it matters: Emphasizes the engineering reality of building reliable agent infrastructure rather than reinventing core systems.

Security, Isolation, and Identity

• Explanation: Addressing the nuances of security, data isolation, and user identity when agents interact within corporate tools like Slack.
• Key claims:
  • Built-in isolation makes it easy to separate sandboxes and tool calls.
  • Agents acting in Slack need clear boundaries between internal and external data.
  • Session management ensures agents act with the correct user permissions and context.
• Examples:
  • Managing user requests in Slack so the agent acts on behalf of the user securely.
• Terminology:
  • built-in isolation
  • session management
  • tool calls
  • identity
  • sandboxes
• Why it matters: Security and authorization are critical when AI agents access sensitive company knowledge and tools.

Personality Design and Future of Knowledge Work

• Explanation: Exploring agent personality design, fidelity to user taste, and how knowledge work will transform over the next few years.
• Key claims:
  • Agents need personality design to feel like working with real colleagues.
  • Fidelity means the agent matches user taste and expectations over time.
  • Reaching the horizon of knowledge work opens up entirely new possibilities for productivity.
• Examples:
  • An agent announcing IT security requirements using the voice and tone of a company leader.
• Terminology:
  • personality design
  • fidelity
  • context management
  • hands experiment
  • bellwether
• Why it matters: Defines the future of human-AI collaboration where agents embody organizational norms and personal work styles.

Key Points

Company-wide AI adoption requires structured agents

• Explanation: Individual experimentation with open-source LLMs fades when maintenance is high; centralized agents standardize workflows.
• Evidence: Transition from individual open-claw tool usage to centralized Every Agent company-wide deployment.
• Practical implication: Companies should build standardized agent wrappers rather than relying on unstructured employee prompt engineering.

Codifying human taste into agent skills

• Explanation: By turning historical personal work output into agent skills, teams can automate complex qualitative tasks like editing.
• Evidence: Turning 30,000 historical editorial edits into Google Doc suggestion skills in Every Agent.
• Practical implication: Capture expert workflows by converting past work samples into evaluation rubrics and agent prompts.

Infrastructure-first agent development

• Explanation: Using cloud-managed agent primitives accelerates feature delivery and ensures proper security sandboxing.
• Evidence: Using Cloud Managed Agents for sandboxes, memory, and session control.
• Practical implication: Leverage existing managed infrastructure to avoid spending engineering bandwidth on custom sandbox orchestration.

Frameworks, Models & Processes

Kate Bench Framework

• How it works: Measures AI model output against a specific human expert's historical editing patterns and style preferences.
• Components:
  • Historical work samples
  • Vibe checks
  • Automated suggestion generation
  • Human acceptance/rejection tracking
• When to use: When evaluating whether an AI model can replicate qualitative knowledge work tasks like writing and editing.

Examples & Case Studies

Every used their agent to handle company-wide open enrollment paperwork.

• Illustrates: How complex algebraic PDF planning can be automated by agents that know the employee's context.
• Lesson: Agents eliminate tedious administrative paperwork by pairing document knowledge with user identity.

An internal announcement skill was created mimicking the company's Head of Operations' tone.

• Illustrates: Personality and tone fidelity in agent design.
• Lesson: Adding personality and voice to agents increases organizational trust and adoption.

Actionable Takeaways

• Immediate:
  • Codify historical work output into reusable agent skills.
  • Use managed agent infrastructure to handle sandboxing and security.
  • Deploy agents within existing communication tools like Slack where knowledge work happens.
• Strategic:
  • Shift the AI evaluation mindset from generic benchmarks to personalized reference checks and work trials.
  • Design agent personalities that reflect company culture and leadership tone.
  • Treat agents as organizational force multipliers rather than simple automation scripts.
• Questions to investigate:
  • How do we balance agent autonomy with data security and access permissions?
  • What is the best way to continuously compound agent skill improvements from human feedback?
  • How will agent personality configuration evolve across different organizational silos?

Claims Worth Verifying

• Engineering teams orchestrating teams of agents have pulled forward their roadmaps by two years. (empirical observation)
• Cloud-managed agents provide isolated sandboxes and memory primitives that drastically reduce infrastructure overhead. (technical architectural claim)

Notable Quotes

"See very excited to be chatting with you guys. I have said this to you before, but I think you guys are some of the most AG-I-pilled people out there." (at 0:07) "Every is a media and technology company. We build the only subscription you need to stay at the edge of AI." (at 3:44) "We built the Every Agent. It is an AI co-worker. It's an agent that is the easiest way to AI pill your whole company." (at 4:14) "Infrastructure is hard, you know, and a lot of what we want to focus on is like the future of work." (at 21:57)

Compressed Summary

• Building secure AI agents for knowledge work
• Transitioning from individual tool usage to company-wide agent orchestration
• Codifying human editorial taste into reproducible agent skills
• Leveraging cloud-managed agent infrastructure for security and sandboxing
• Keywords: agents, knowledge work, infrastructure, security, ai
• Core insight: Building secure AI agents for knowledge work requires combining robust cloud-managed infrastructure with personalized agent skills that reflect expert human taste and organizational context.