AI Engineer · Published 2026-10-12

Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

privateReview: unreviewedConfidence: high
agentscicdevalsmultitenancySandboxinginfrastructure-as-codecoordinator-patternagent-harness
Open on YouTube ↗

Summary

Overview

  • Speaker: Anthropic team members
  • Channel: AI Engineer
  • Main topic: Advanced deployment, CI/CD integration, multi-tenancy, evals, and sandboxing for Claude Managed Agents
  • Purpose: Address developer inquiries and clarify advanced operational aspects of building, testing, evaluating, and scaling Claude Managed Agents. In this final Q&A session from the Claude Managed Agents workshop at the AI Engineer World's Fair, Anthropic engineers answer developer questions regarding agent CI/CD pipelines, evaluation strategies and quality best practices, multi-tenancy isolation options, compute resource allocation and GPU environments, and orchestrating sub-agents across long-running sessions.

Topic Map

CI/CD and Agent Workflow Updates

  • Explanation: How developers can version agent definitions, prompt Yamls, and long-term resources in codebases and automatically update deployments via CLI tools.
  • Key claims:
    • Ant-CLI wraps Anthropic APIs to allow Terraform-like YAML configuration of agent resources.
    • CI/CD pipelines can automatically push updates whenever code repo resources change.
  • Examples:
    • Storing agent long-term resources in YAML within a codebase and syncing via CI/CD.
  • Terminology:
    • ant-cli
    • CI/CD
    • Terraform-like YAML
  • Why it matters: Enables automated, version-controlled agent management across engineering teams.

Evaluations Best Practices

  • Explanation: Approaches for running evals on new model releases and agent updates, highlighting the Garbage In, Garbage Out principle and iterative prompt tuning.
  • Key claims:
    • Anthropic runs continuous evals in cloud environments, accessing models early to update harness assumptions.
    • High-quality data is critical for accurate evaluation; poor benchmark design leads to uninformative metrics.
  • Examples:
    • Spinning up 10,000 parallel cloud sessions to evaluate prompts and assumptions against diverse data sets.
  • Terminology:
    • evals
    • harness
    • garbage in garbage out
  • Why it matters: Ensures reliability and prevents degradation of agent performance during prompt or model updates.

Multi-Tenancy and Data Segregation

  • Explanation: Structuring workspaces, API keys, and organizations to maintain strict tenant isolation and compliance for enterprise customers.
  • Key claims:
    • Agents are bound to workspaces; API keys within a workspace control resource access.
    • Workspace design or isolated organizations can achieve customer-level data segregation.
  • Examples:
    • Creating one workspace per customer to enforce strict data segregation.
  • Terminology:
    • workspace
    • organization
    • data segregation
    • multi-tenancy
  • Why it matters: Crucial for enterprise compliance and security when deploying shared agents across different clients.

Compute Environments, Memory, and GPUs

  • Explanation: Managing resource requirements like higher memory or GPU needs for agents using self-hosted sandboxes and integrations like Modal.
  • Key claims:
    • For custom resource allocations (GPUs or high memory), developers should use self-hosted sandboxes like Modal.
    • Anthropic provides cookbooks for first-party integrations with third-party sandbox providers.
  • Examples:
    • Hooking up Modal instances to run agent loops instead of default Anthropic sandboxes.
  • Terminology:
    • Modal
    • self-hosted sandboxes
    • cookbooks
  • Why it matters: Allows developers to scale compute-heavy agents beyond default platform constraints.

Long-Running Sessions and Coordinator Patterns

  • Explanation: Handling hours-long agent runs, tracking checkpoint status, and coordinating sub-agents without blocking the main thread.
  • Key claims:
    • Outcome-driven evaluation and exit criteria are best for very long tasks.
    • Coordinator-subagent patterns allow main agents to remain active while sub-agents execute parallel subtasks.
  • Examples:
    • Setting exit criteria for a task like 'rebuild blank' and letting Claude iterate until criteria are met.
  • Terminology:
    • coordinator pattern
    • sub-agents
    • exit criteria
    • session event stream
  • Why it matters: Enables robust execution of complex, multi-hour engineering tasks.

Key Points

CI/CD Integration for Agents

  • Explanation: Agent configurations can be treated like infrastructure-as-code using CLI wrappers.
  • Evidence: Mention of ant-cli and syncing YAML configurations on git push.
  • Practical implication: Teams can manage agent updates through standard pull request workflows.

Importance of Data Quality in Evals

  • Explanation: Evaluation quality depends entirely on the quality and specificity of the test data and prompts.
  • Evidence: Statement on garbage in, garbage out regarding initial evaluation assumptions.
  • Practical implication: Developers must invest time in refining historical sessions and scoring rubrics.

Workspace-Level Multi-Tenancy

  • Explanation: Isolation and access control are managed via workspaces and organization admin settings.
  • Evidence: Explanation that API keys are bound to workspaces and admin privileges govern resource partitioning.
  • Practical implication: Architecting customer-specific workspaces ensures data privacy.

Frameworks, Models & Processes

Coordinator-Subagent Pattern

  • How it works: A main coordinator agent delegates work to multiple sub-agents running across separate threads or sessions, gathering tool results and maintaining a central chat interface.
  • Components:
    • Main coordinator agent
    • Sub-agents
    • Session event stream API
  • When to use: When building complex applications requiring parallel subtasks or role-based delegation.

Examples & Case Studies

A developer asked about maintaining tenant isolation across 100 enterprise customers.

  • Illustrates: Multi-tenancy and data segregation best practices using workspaces.
  • Lesson: Creating per-customer workspaces or using separate organizational units provides the necessary isolation.

Actionable Takeaways

  • Immediate:
    • Use ant-cli to manage agent definitions as code in YAML.
    • Leverage workspace structures for multi-tenant data separation.
  • Strategic:
    • Define clear exit criteria and outcome metrics for long-running agent tasks.
    • Integrate self-hosted providers like Modal when agents require GPU or high-memory compute.
  • Questions to investigate:
    • How does fine-grained permission control evolve across larger enterprise workspaces?
    • What are the latency tradeoffs when scaling parallel sub-agent sessions?

Claims Worth Verifying

  • ant-cli allows Terraform-like YAML configuration for deploying agent resources (feature claim)
  • Self-hosted sandboxes like Modal can be integrated with Claude Managed Agents for GPU access (technical architecture claim)

Notable Quotes

"garbage in, garbage out" (at 6:47) "we don't have uh kind of tunneling today" (at 13:54)

Compressed Summary

  • Manage agent configurations via CLI and YAML version control
  • Ensure robust evaluation metrics with high-quality historical data
  • Achieve multi-tenant isolation using workspace and organization boundaries
  • Use Modal or custom sandboxes for GPU and high-memory workloads
  • Deploy coordinator-subagent patterns for long-running, multi-threaded tasks
  • Keywords: agents, cicd, evals, multitenancy, sandboxing
  • Core insight: Claude Managed Agents support enterprise-grade workflows through version-controlled YAML configuration, flexible workspace isolation, and scalable sandbox integrations.

Core insights

6
Architecturemedium noveltymoderate evidence

Agent definitions and long-term resources can be treated as declarative infrastructure: ant-cli wraps the Anthropic APIs to expose Terraform-like YAML configuration, and CI/CD pipelines push updates whenever repo resources change, making agent deployment a version-controlled, PR-driven operation rather than a console/manual task.

Why it matters

Moves agent lifecycle management into the same abstraction layer as cloud infrastructure (IaC), enabling review, rollback, and reproducible environments for agent state including prompt YAML and long-term resources.

Generalization

Any managed agent platform benefits from separating declarative agent/resource specs (versioned in git) from the runtime that reconciles them, mirroring Terraform's plan/apply model.

Ant-CLI wraps Anthropic APIs to allow Terraform-like YAML configuration of agent resources.
Open source video
CI/CD pipelines can automatically push updates whenever code repo resources change.
Open source video
Architecturemedium noveltystrong evidence

The workspace is the actual tenancy and access-control boundary: agents are bound to a workspace and API keys within a workspace govern resource access, so per-customer workspaces (or fully isolated organizations) are the mechanism for data segregation — not application-level filtering. Cross-workspace access is not currently possible (no 'tunneling').

Why it matters

Isolation is enforced at the platform boundary rather than in application code, which is a stronger security guarantee but also a hard partitioning constraint that forces duplication of shared resources across tenants.

Generalization

Where a platform offers only coarse-grained namespace isolation, multi-tenant designs get simpler to secure but lose the ability to share state or tooling across tenants.

Agents are bound to workspaces; API keys within a workspace control resource access.
Open source video
Creating one workspace per customer to enforce strict data segregation.
Open source video
we don't have uh kind of tunneling today
Open source video
Practicemedium noveltymoderate evidence

Evaluation is run as a continuous cloud harness that is updated in lockstep with new model releases: Anthropic accesses models early specifically to revise harness assumptions, and scales eval by spinning up ~10,000 parallel cloud sessions against diverse data. This implies the harness itself is a maintained artifact, not a fixed benchmark.

Why it matters

Treating the eval harness as a versioned, continuously-updated component (rather than a static test suite) is what prevents stale assumptions from silently invalidating results after model updates.

Generalization

Whenever the underlying model changes, evaluation scaffolding — assumptions, rubrics, test scaffolding — must be revised, so harness maintenance is an ongoing cost, not one-time setup.

Anthropic runs continuous evals in cloud environments, accessing models early to update harness assumptions.
Open source video
Spinning up 10,000 parallel cloud sessions to evaluate prompts and assumptions against diverse data sets.
Open source video
Architecturemedium noveltymoderate evidence

The managed sandbox has bounded compute, so GPU or high-memory workloads are handled by substituting a self-hosted sandbox (e.g., Modal) that runs the agent loop, with Anthropic providing cookbooks for third-party sandbox integrations. This implies the agent loop and the execution sandbox are separable layers.

Why it matters

Defines the abstraction boundary: orchestration/agent logic stays platform-managed while the execution environment becomes pluggable, giving teams a path to custom hardware without abandoning the managed control plane.

Generalization

Pluggable execution backends are the standard escape hatch for managed agent platforms whose default compute tier is insufficient.

For custom resource allocations (GPUs or high memory), developers should use self-hosted sandboxes like Modal.
Open source video
Anthropic provides cookbooks for first-party integrations with third-party sandbox providers.
Open source video
Mental Modelmedium noveltymoderate evidence

The coordinator–subagent pattern keeps the main agent active while sub-agents run parallel subtasks across separate threads/sessions, with a session event stream API as the coordination substrate. Combined with outcome-driven evaluation and explicit exit criteria, long-running (multi-hour) tasks are governed by completion conditions rather than step counts.

Why it matters

Provides a concrete concurrency and control-flow model for long-horizon agents: parallelism lives in sub-sessions, and termination is defined by verifiable exit criteria rather than iteration limits.

Generalization

Long-running agent tasks fare better under outcome-based termination (define success, let the agent iterate) than under fixed step budgets, and delegation to sub-sessions avoids blocking the coordinator.

Coordinator-subagent patterns allow main agents to remain active while sub-agents execute parallel subtasks.
Open source video
Outcome-driven evaluation and exit criteria are best for very long tasks.
Open source video
Setting exit criteria for a task like 'rebuild blank' and letting Claude iterate until criteria are met.
Open source video
Failure Modelow noveltystrong evidence

Eval signal quality is bounded by input data quality — the explicit 'garbage in, garbage out' framing means poor benchmark/rubric design produces uninformative metrics, so investment must go into refining historical sessions and scoring rubrics rather than just adding eval volume.

Why it matters

A high-volume eval pipeline can still produce misleading confidence; without curated historical data and rubrics, the metric is not a reliable gate for deploying prompt or model changes.

Generalization

Eval infrastructure throughput is cheap relative to the cost of curating representative, well-scored data — the bottleneck is data quality, not compute.

garbage in, garbage out
Open source video
High-quality data is critical for accurate evaluation; poor benchmark design leads to uninformative metrics.
Open source video

Deep dives

5

Agent infrastructure-as-code and CI/CD reconciliation

Research question

What is the minimal declarative resource model and reconciliation loop needed to make agent prompt YAML, long-term resources, and workspace bindings reproducible and reviewable through CI/CD?

Why

Agent lifecycle moves from manual console edits to version-controlled infrastructure, affecting rollback, audit, and multi-environment promotion.

Ant-CLI wraps Anthropic APIs to allow Terraform-like YAML configuration of agent resources.
Open source video
CI/CD pipelines can automatically push updates whenever code repo resources change.
Open source video
Source video

Workspace-level multi-tenancy and access boundaries

Research question

How do workspace-scoped API keys and the absence of cross-workspace tunneling constrain shared tooling and force duplication in multi-tenant agent deployments?

Why

Workspace isolation is the platform-enforced tenancy boundary, so design must account for hard partitioning and per-customer resource duplication.

Agents are bound to workspaces; API keys within a workspace control resource access.
Open source video
Creating one workspace per customer to enforce strict data segregation.
Open source video
we don't have uh kind of tunneling today
Open source video
Source video

Eval harness as a maintained, continuously updated artifact

Research question

How should eval harness assumptions, rubrics, and datasets be versioned and updated when a new model release invalidates prior harness assumptions?

Why

Static benchmarks silently lose validity after model changes; harness maintenance is an ongoing cost and a deployment gate.

Anthropic runs continuous evals in cloud environments, accessing models early to update harness assumptions.
Open source video
Spinning up 10,000 parallel cloud sessions to evaluate prompts and assumptions against diverse data sets.
Open source video
Source video

Pluggable execution sandboxes for GPU and high-memory agents

Research question

What guarantees and integration contracts are lost when the managed execution sandbox is replaced by a self-hosted sandbox such as Modal for GPU or high-memory workloads?

Why

Separates agent loop from execution runtime, providing an escape hatch but shifting operational responsibility to the team.

For custom resource allocations (GPUs or high memory), developers should use self-hosted sandboxes like Modal.
Open source video
Anthropic provides cookbooks for first-party integrations with third-party sandbox providers.
Open source video
Source video

Outcome-driven termination for long-running coordinator-subagent sessions

Research question

How should verifiable exit criteria be specified so that coordinator-subagent systems terminate on completion rather than step count, and what false-completion rate results?

Why

Defines control flow and reliability for multi-hour tasks where parallelism lives in sub-sessions.

Coordinator-subagent patterns allow main agents to remain active while sub-agents execute parallel subtasks.
Open source video
Outcome-driven evaluation and exit criteria are best for very long tasks.
Open source video
Setting exit criteria for a task like 'rebuild blank' and letting Claude iterate until criteria are met.
Open source video
Source video

Article ideas

4

The Workspace Is the Tenant: Strong Isolation's Hidden Tax

Workspace-level isolation is a stronger tenancy boundary than application filtering, but the lack of cross-workspace access turns shared tooling into duplicated resources, so multi-tenant agent design becomes a choice between compliance strength and operational reuse.

Angle

Multi-tenancy architecture tradeoff analysis.

Source video

Your Eval Harness Is Not a Benchmark: Maintain It or Mistake It

Because harness assumptions must be revised alongside model releases, evaluation scaffolding is a continuously maintained artifact; teams that treat evals as static benchmarks will deploy on stale signal.

Angle

Eval reliability and maintenance cost.

Source video

Agents Are Infrastructure: Why Prompt YAML Belongs in CI/CD

Treating agent definitions and long-term resources as declarative, version-controlled infrastructure forces review and rollback, but also imports the operational discipline and drift risks of IaC.

Angle

Infrastructure-as-code lens on agent lifecycle management.

Source video

Exit Criteria Beat Step Budgets for Long-Horizon Agents

Multi-hour agents should terminate on verifiable completion conditions, not iteration counts, and coordinator-subagent parallelism only helps if those conditions are machine-checkable.

Angle

Control-flow and reliability for long-running agent systems.

Source video

Project ideas

4

Tenant Boundary Probe

gatehouse

Per-customer workspaces enforce tenant isolation at the platform boundary, so cross-workspace resource access attempts will fail even when identical agent definitions and API keys are deployed across workspaces; the cost is duplicated shared resources.

Proof of concept

Deploy the same managed agent stack into two workspaces under one organization, use API keys from workspace A to attempt reads and writes on workspace B agents and resources, and inventory duplicated resources.

Measurement

Number of successful cross-workspace access attempts (expected 0), duplicated resource count, setup and management overhead per additional tenant.

Source video

Harness Drift Detector

beyond-evals

A versioned eval harness with explicit assumption checks will detect stale rubric and dataset assumptions after a model update before false-confidence metrics are used as a deployment gate.

Proof of concept

Freeze an old harness, run it against a new model release, compare scores and assumption-level failures against an updated harness, and flag metrics whose validity collapses.

Measurement

Stale-assumption detection recall, false-confidence delta between old and updated harness scores, metric variance across model versions.

Source video

Agent GitOps Reconciler

new

Declarative agent and resource YAML reconciled by CI/CD reduces configuration drift and manual interventions compared with console-managed agent updates.

Proof of concept

Build a minimal ant-cli-driven repo with agent prompt YAML and long-term resources, run a CI pipeline that plans and applies changes to a test workspace, inject drift, and measure reconciliation.

Measurement

Drift detection latency, manual interventions per release, rollback time, percentage of agent changes reviewed in PRs.

Source video

Outcome-Exit Benchmark

movement-lab

Outcome-driven exit criteria reduce false completion and wasted steps for multi-hour coordinator-subagent tasks compared with fixed step budgets.

Proof of concept

Run a long-running rebuild-style task with a coordinator and parallel sub-agents under two termination policies: step budget versus machine-checkable exit criteria.

Measurement

False completion rate, total agent steps and tokens, wall-clock time, human intervention count.

Source video

Architectural implications

3

Agent configuration is managed as declarative YAML in a repo and synced by CI/CD via ant-cli.

Before

Agent prompts and long-term resources configured manually, out-of-band, or per-environment in console/UI.

After

Agent resources live in version control and are applied via pipeline, with diffs, reviews, and rollbacks.

Consequence

Agent changes become reviewable in PRs and reproducible across environments, but require disciplined repo structure and sync tooling.

Source video

Isolation is enforced at the workspace/organization boundary, with API keys scoped to a workspace.

Before

Multi-tenant isolation handled in application logic (filtering, row-level checks).

After

Tenant isolation is a platform-level namespace construct; one workspace per customer where strict segregation is required.

Consequence

Stronger security guarantees but proliferation of workspaces and duplicated shared resources; no cross-workspace access ('tunneling').

Source video

Execution sandbox becomes pluggable (self-hosted providers like Modal) while the agent loop remains managed.

Before

All agent execution runs in the platform's default sandbox with its fixed resource profile.

After

The agent loop can be pointed at a self-hosted sandbox to obtain GPU or high-memory compute.

Consequence

Teams can scale compute-heavy agents beyond platform limits, at the cost of operating their own sandbox infrastructure.

Source video

Tradeoffs and failure modes

4

Strict workspace-level tenant isolation

Benefit

Customer-level data segregation enforced by the platform rather than application code, improving compliance posture.

Cost or risk

No cross-workspace access is supported today, so shared resources and cross-tenant workflows cannot be expressed; per-customer workspaces multiply management overhead.

we don't have uh kind of tunneling today
Open source video
Source video

Continuous evals tied to early model access

Benefit

Harness assumptions can be revised before/with model releases, keeping evaluation aligned with model behavior.

Cost or risk

Harness assumptions are a moving target; stale assumptions can silently degrade metric validity after model updates.

Anthropic runs continuous evals in cloud environments, accessing models early to update harness assumptions.
Open source video
Source video

Eval throughput via massive parallel cloud sessions

Benefit

Thousands of concurrent sessions allow prompt and assumption testing against diverse datasets at scale.

Cost or risk

Compute cost scales with concurrency, and throughput does not fix invalid data — bad inputs yield uninformative metrics.

Spinning up 10,000 parallel cloud sessions to evaluate prompts and assumptions against diverse data sets.
Open source video
Source video

Self-hosted sandbox for GPU/high memory

Benefit

Access to GPU and high-memory compute that the default managed sandbox cannot provide.

Cost or risk

Teams take on operating and integrating third-party sandbox infrastructure instead of relying on the managed environment.

For custom resource allocations (GPUs or high memory), developers should use self-hosted sandboxes like Modal.
Open source video
Source video

Open questions

4

How does fine-grained permission control evolve across larger enterprise workspaces?

Why unresolved

The summary describes workspace- and organization-level scoping of API keys but does not specify per-resource or per-role granularity within a workspace.

Research direction

Map the permission model (workspace vs organization vs key scope) and prototype least-privilege configurations for shared agents.

Source video

What are the latency tradeoffs when scaling parallel sub-agent sessions?

Why unresolved

The coordinator-subagent pattern is described structurally, but no latency or throughput numbers are given for parallel sessions.

Research direction

Benchmark coordinator-subagent fan-out at increasing concurrency and measure coordination/event-stream overhead.

Source video

How should exit criteria for long-running outcome-driven tasks be specified and validated?

Why unresolved

Exit criteria are recommended ('rebuild blank') but the summary gives no formalism for expressing or verifying them.

Research direction

Prototype declarative, machine-checkable exit criteria and measure false-positive completion rates over multi-hour runs.

Source video

Does the declarative YAML/ant-cli model cover all agent state, including long-term resources and memory?

Why unresolved

The summary claims 'agent long-term resources' are stored in YAML, but the boundary of what is declaratively managed is not defined.

Research direction

Enumerate which agent artifacts round-trip through ant-cli and which require out-of-band state management.

Source video

Key claims

7
factualVerification needed

ant-cli wraps Anthropic APIs to allow Terraform-like YAML configuration of agent resources, enabling CI/CD-driven updates.

Evidence

Ant-CLI wraps Anthropic APIs to allow Terraform-like YAML configuration of agent resources.

Question

Does ant-cli actually expose a Terraform-like declarative config surface, and what resources does it cover?

Source video
factualVerification needed

Agents are bound to workspaces, and API keys within a workspace control resource access, making workspaces the tenancy boundary.

Evidence

Agents are bound to workspaces; API keys within a workspace control resource access.

Question

What resources are scoped at the workspace versus organization level, and can agents be shared across workspaces?

Source video
factualVerification needed

Cross-workspace access ('tunneling') is not currently available.

Evidence

we don't have uh kind of tunneling today

Question

Is there any supported mechanism for cross-workspace resource sharing, and is one planned?

Source video
factualVerification needed

For GPU or high-memory needs, developers should use self-hosted sandboxes such as Modal, with first-party cookbooks provided.

Evidence

For custom resource allocations (GPUs or high memory), developers should use self-hosted sandboxes like Modal.

Question

What is the integration surface between Claude Managed Agents and self-hosted sandboxes, and what guarantees are lost?

Source video
factualVerification needed

Anthropic runs continuous cloud evals and accesses models early to update harness assumptions, at scales reaching 10,000 parallel sessions.

Evidence

Spinning up 10,000 parallel cloud sessions to evaluate prompts and assumptions against diverse data sets.

Question

What concurrency and cost envelope is realistic for third-party teams running comparable eval harnesses?

Source video
causalVerification needed

Evaluation quality is bounded by input data quality; poor benchmark design yields uninformative metrics.

Evidence

High-quality data is critical for accurate evaluation; poor benchmark design leads to uninformative metrics.

Question

How much does rubric/data curation improve eval signal relative to simply increasing eval volume?

Source video
causalVerification needed

Coordinator-subagent patterns let the main agent stay active while sub-agents execute parallel subtasks, and outcome-driven exit criteria suit very long tasks.

Evidence

Coordinator-subagent patterns allow main agents to remain active while sub-agents execute parallel subtasks.

Question

What latency and coordination overhead does the coordinator-subagent pattern introduce versus a single long-running session?

Source video

Connections

5