AI Engineer · Published 2026-10-14

Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

privateReview: unreviewedConfidence: high
MCPgatewaygovernanceagentstelemetryoptimizationagent-governancecontext-optimizationpii-blockingObservabilityhitlenterprise-agents
Open on YouTube ↗

Summary

Overview

  • Speaker: Harish Gaggar
  • Channel: AI Engineer
  • Main topic: Building and governing enterprise AI agents for production deployment
  • Purpose: To provide engineers and enterprise architects with a comprehensive framework and architectural blueprint for deploying secure, auditable, and cost-efficient AI agents in production. Harish Gaggar, Senior Staff Software Engineer at Credit Karma, explains how to transition AI agents from experiments to secure, governed enterprise production environments. He covers architectural patterns, MCP (Model Context Protocol) servers and gateways, monitoring, tracking and evaluation, PII blocking, and context optimization.

Topic Map

Challenges in Enterprise AI Agents

  • Explanation: Enterprise AI agents often fail in production because foundational controls such as gateways, proper tool execution, tracking, memory management, and security are missing.
  • Key claims:
    • AI agents fail in production when foundational controls are missing
    • Many lack secure gateways for authentication and authorization
    • Missing tracking, memory management, and PII data blocking
  • Examples:
    • Secrets leaking into LLMs or tracking systems
    • Lack of audit trails for agent tool executions
  • Terminology:
    • foundational controls
    • gateway
    • MCP gateway
  • Why it matters: Without proper enterprise controls, AI agents pose severe security, privacy, and cost risks in production.

Design Framework for Enterprise Agents

  • Explanation: A layered framework encompassing agent frameworks, OAuth authentication, customized MCP servers, continuous audit, PII stripping, tracking and eval, memory management, and context optimization.
  • Key claims:
    • Each layer must be scoped, sanitized, audited, and measured by design
    • An agent needs an enterprise-grade framework around it, not just a standalone agent
  • Examples:
    • LangGraph for task-specific nodes, skills, and prompts
    • FastMCP wrapper for exposing typed tools
  • Terminology:
    • agent framework
    • FastMCP
    • HITL
    • context optimizer
  • Why it matters: Structured layering ensures safety, compliance, and predictable behavior at enterprise scale.

MCP Servers and Gateways

  • Explanation: Model Context Protocol (MCP) servers turn REST APIs into typed, permission-scoped tools, while MCP gateways provide centralized authentication, routing, and JWE access token un-wrapping.
  • Key claims:
    • MCP servers enforce request filters, limits, and blocked fields
    • MCP gateway ensures every request runs as a real identity through an audited gateway
    • MCP servers never face the client directly
  • Examples:
    • Artifactory MCP and GitHub MCP
    • Redis session state storage for OAuth requests
  • Terminology:
    • MCP server
    • MCP gateway
    • FastMCP
    • JWE access token
  • Why it matters: Decoupling clients from servers via a secure gateway prevents unauthorized direct access to enterprise data and internal tools.

Monitoring, Tracking, and Evaluation

  • Explanation: Comprehensive telemetry using OpenTelemetry collectors, Tempo for traces, Prometheus for metrics, and Grafana for dashboards, alongside BigQuery tables for agent job tracking and evaluation.
  • Key claims:
    • AI agents cannot be trusted without continuous monitoring, evaluation, and feedback
    • Separate agent-side LLM telemetry from gateway operational telemetry
  • Examples:
    • BigQuery tables: agent_job_tracking, agent_quality_eval, agent_tool_invocations
    • OpenTelemetry collector routing to Tempo and Prometheus
  • Terminology:
    • OpenTelemetry
    • Tempo
    • Prometheus
    • Grafana
    • OTLP
  • Why it matters: Enables root cause analysis, latency tracking, cost monitoring, and quality evaluation across hundreds of deployed agents.

Agent Memory, PII Blocking, and Context Optimization

  • Explanation: Persistent checkpointing for workflows across runs, real-time PII stripping before LLM ingestion, and a context optimizer to prune, rerank, and compress context to save token costs.
  • Key claims:
    • Agent persistence ensures durability and seamless recovery across runs using thread IDs and checkpoints
    • PII trimming blocks high-severity secrets from ever reaching LLMs
    • Context optimization saves 70% to 90% in token costs
  • Examples:
    • SQLite for local development, Cloud Spanner for production
    • Blocking private keys, AWS access keys, SSNs, credit cards with 95%+ severity rules
  • Terminology:
    • checkpointing
    • thread ID
    • PII trimmer
    • context optimizer
    • token savings
  • Why it matters: Protects sensitive enterprise data from leaking while drastically reducing LLM operational costs.

Key Points

Build an enterprise framework, not just an AI agent

  • Explanation: Production agents require surrounding infrastructure for registration, CI/CD pipelines, secret management, memory, and tracking.
  • Evidence: Dev portal creates repository with workload identity, memory DB, tracking tables, and secret management automatically.
  • Practical implication: Standardizes agent development and deployment across the entire engineering organization.

Gateway-first security is the foundation

  • Explanation: All tool calls must be authenticated and governed through an MCP gateway before reaching backend servers.
  • Evidence: Architecture diagrams show MCP servers never facing clients directly; all requests pass through the gateway with JWE tokens.
  • Practical implication: Eliminates shadow IT and unauthenticated LLM tool invocations.

MCP turns REST APIs into safe, agent-friendly tools

  • Explanation: Using FastMCP server wrappers, developers expose typed, permission-scoped tools with strict input filters and limits.
  • Evidence: Internal security teams review all tools before deployment to production.
  • Practical implication: Prevents LLMs from constructing arbitrary or dangerous API calls.

Keep humans in control of high-risk actions

  • Explanation: Write and sensitive operations must pass through Human-In-The-Loop (HITL) approval workflows.
  • Evidence: Agent framework flow includes approval diamond checking execute vs reject states.
  • Practical implication: Mitigates catastrophic errors or unintended modifications by autonomous agents.

Optimize context before it becomes cost

  • Explanation: Context optimizers select relevant payloads, rerank information, remove duplicates, and compress prompts before sending to LLMs.
  • Evidence: Demo showed token usage dropping from 24,000 tokens down to 2,400 tokens (90% savings) for the same query.
  • Practical implication: Enormous reduction in API costs for high-volume enterprise AI agents.

Frameworks, Models & Processes

Agent Design & Governance Framework

  • How it works: A multi-layered architecture handling requests from agent framework through OAuth, MCP gateway, customized servers, PII stripping, tracking, memory, and context optimization.
  • Components:
    • Agent Framework (LangGraph)
    • OAuth Authentication
    • Customized MCP Server
    • Continuous Audit
    • PII/Secrets Stripping
    • Tracking and Evaluation
    • Memory Management
    • Context Optimizer
  • When to use: When deploying multi-agent workflows into enterprise production environments.

Context Optimizer Framework

  • How it works: Selects payload based on token limit, reranks relevant information, clusters to remove duplicates, and compresses the final context before LLM invocation.
  • Components:
    • Select
    • Rerank
    • Cluster
    • Compress
  • When to use: When handling large codebases, documentation, or extensive tool outputs to avoid token bloat and high costs.

Examples & Case Studies

Running a supply chain agent query searching for vulnerable Docker images across repositories in Artifactory.

  • Illustrates: How agent tracking, eval tables, and tool invocations record metrics and trajectories in BigQuery.
  • Lesson: Granular logging allows deep auditing of every LLM call, token usage, latency, and tool success rate.

Executing a complex query with and without the Context Optimizer enabled.

  • Illustrates: Token usage dropping from ~24,000 tokens to ~2,400 tokens (~90% cost savings).
  • Lesson: Context optimization preserves output quality while dramatically reducing token consumption.

Actionable Takeaways

  • Immediate:
    • Implement gateway-first authentication for all agent tool calls
    • Adopt FastMCP for building typed, secure MCP servers
    • Deploy PII and secret trimming before data reaches LLMs
  • Strategic:
    • Build a centralized developer portal for registering and deploying governed agents
    • Establish rigorous tracking, evaluation, and telemetry using OpenTelemetry and BigQuery
    • Always include Human-In-The-Loop approval gates for write and sensitive operations
  • Questions to investigate:
    • How do we scale MCP gateways across multi-cloud enterprise environments?
    • What are the latency tradeoffs of running multi-layer PII filters and context optimizers?

Claims Worth Verifying

  • Context optimizers can save 70% to 90% in token costs while maintaining agent accuracy (performance metric)
  • MCP servers combined with MCP gateways completely eliminate unauthenticated LLM tool execution (architectural claim)

Notable Quotes

"Enterprise AI agents often fail in production when foundational controls are missing" (at 2:09) "MCP server is a protocol layer that turns REST API into typed, permission-scoped tools an agent can safely call" (at 13:36) "Every request runs as a real identity through audited gateway" (at 26:49)

Compressed Summary

  • Enterprise AI agents require robust architectural controls, not just standalone code
  • MCP servers and gateways enforce strict authentication, routing, and tool scoping
  • Memory management via thread checkpointing ensures workflow durability and recovery
  • PII strippers block sensitive secrets from reaching LLMs and logging backends
  • Context optimizers cut token consumption by up to 90% through select, rerank, cluster, and compress
  • Keywords: mcp, gateway, governance, agents, telemetry, optimization
  • Core insight: Production-grade enterprise AI agents require gateway-first security, protocol-based tool servers (MCP), rigorous tracking, and context optimization to ensure safety, auditability, and cost efficiency.

Core insights

7
Architecturehigh noveltystrong evidence

The MCP gateway functions as a mandatory identity-bearing chokepoint: every agent tool call is authenticated as a real user identity and passes through an audited gateway with JWE access-token un-wrapping, and MCP servers are never exposed directly to clients.

Why it matters

Reframes MCP from a developer-convenience protocol into a security boundary; it eliminates shadow IT and unauthenticated LLM tool invocations, and makes the gateway — not the agent — the unit of access control and audit.

Generalization

For any tool-using agent system, insert a policy/identity enforcement layer between the agent runtime and backend APIs so authorization and audit live outside the LLM's control.

MCP gateway ensures every request runs as a real identity through an audited gateway
Open source video
MCP servers never face the client directly
Open source video
Mechanismmedium noveltystrong evidence

Wrapping REST APIs with FastMCP exposes typed, permission-scoped tools with request filters, limits, and blocked fields, reviewed by security before production, so the LLM cannot construct arbitrary or dangerous API calls.

Why it matters

The tool surface, not the prompt, becomes the enforcement point for API safety; constraints are declared at the schema/tool boundary rather than trusted to model behavior.

Generalization

Constrain agent capability by narrowing the callable API surface (typed args, hard limits, blocked fields) instead of relying on prompt instructions or output validation alone.

MCP servers enforce request filters, limits, and blocked fields
Open source video
MCP turns REST APIs into safe, agent-friendly tools
Open source video
Empirical Resulthigh noveltymoderate evidence

A context optimizer that selects relevant payloads, reranks, deduplicates, and compresses context before LLM ingestion cut token usage from 24,000 to 2,400 for the same query (90% reduction).

Why it matters

Demonstrates a concrete, measurable cost lever for high-volume agents, with the optimizer positioned as its own layer rather than a side effect of prompt engineering.

Generalization

Treat context construction as an optimizing pipeline (select/rerank/dedup/compress) with token-cost as a first-class metric to be benchmarked per query class.

Context optimization saves 70% to 90% in token costs
Open source video
Demo showed token usage dropping from 24,000 tokens down to 2,400 tokens (90% savings) for the same query.
Open source video
Practicemedium noveltymoderate evidence

Agent-side LLM telemetry is deliberately separated from gateway operational telemetry, with OpenTelemetry collectors routing traces to Tempo and metrics to Prometheus, plus BigQuery tables for job tracking and quality evaluation.

Why it matters

Splitting the two signal domains lets teams do root-cause analysis on agent reasoning/quality independently from infrastructure latency and reliability, which is hard when both are conflated in one pipeline.

Generalization

Instrument agent semantics (quality, tool invocations, eval scores) separately from infrastructure operations (gateway latency, errors) so each can be owned and queried independently.

Separate agent-side LLM telemetry from gateway operational telemetry
Open source video
BigQuery tables: agent_job_tracking, agent_quality_eval, agent_tool_invocations
Open source video
Mechanismmedium noveltystrong evidence

A real-time PII trimmer blocks high-severity secrets (private keys, AWS access keys, SSNs, credit cards) using 95%+ severity rules before anything reaches the LLM or tracking systems.

Why it matters

Moves data protection upstream of the model and of the observability stack, so leaked secrets cannot propagate into prompts, traces, or eval datasets.

Generalization

Place a deterministic, severity-thresholded redaction layer on the ingestion path ahead of any model or logging sink.

PII trimming blocks high-severity secrets from ever reaching LLMs
Open source video
Blocking private keys, AWS access keys, SSNs, credit cards with 95%+ severity rules
Open source video
Practicehigh noveltymoderate evidence

A dev portal auto-scaffolds each new agent with workload identity, a memory DB, tracking tables, and secret management, standardizing deployment across the whole engineering org.

Why it matters

Shifts governance from per-team discipline to platform-provided defaults, making production controls the path of least resistance rather than optional add-ons.

Generalization

Encode agent production requirements (identity, persistence, telemetry, secrets) into a golden-path scaffold so compliance is automatic and uniform.

Dev portal creates repository with workload identity, memory DB, tracking tables, and secret management automatically.
Open source video
Mechanismmedium noveltymoderate evidence

Agent persistence uses thread IDs and checkpoints for durability across runs, with SQLite for local development and Cloud Spanner for production.

Why it matters

Checkpointing gives seamless recovery and continuity for long-running workflows, and the local/prod backend split lets developers iterate without provisioning production state stores.

Generalization

Abstract memory behind a checkpointing interface so the same workflow code runs against a lightweight local store in dev and a durable distributed store in production.

Agent persistence ensures durability and seamless recovery across runs using thread IDs and checkpoints
Open source video
SQLite for local development, Cloud Spanner for production
Open source video

Deep dives

4

Quality-Quality Trade-off in Context Optimization

Research question

Does aggressive context optimization that yields 70-90% token savings preserve task accuracy on the same query set?

Why

Token cost reduction is meaningless if it degrades model performance; enterprises need to know the quality cost of optimization before adopting it.

Context optimization saves 70% to 90% in token costs
Open source video
Demo showed token usage dropping from 24,000 tokens down to 2,400 tokens (90% savings) for the same query.
Open source video
Source video

Latency and Reliability Overhead of Mandatory MCP Gateway

Research question

What is the added latency and failure mode when an MCP gateway (with JWE token unwrapping) becomes a mandatory hop for every tool call?

Why

The gateway is positioned as a security boundary, but its systemic risk and performance impact are unquantified, which could hinder adoption.

MCP gateway ensures every request runs as a real identity through an audited gateway
Open source video
MCP servers never face the client directly
Open source video
Source video

Precision and Recall of Severity-Thresholded PII Trimmer

Research question

What are the false positive and false negative rates of the 95%+ severity PII rules, and how are novel secret formats handled?

Why

Over-blocking can break legitimate workflows, while under-blocking leaks secrets; quantitative performance is needed to tune thresholds.

PII trimming blocks high-severity secrets from ever reaching LLMs
Open source video
Blocking private keys, AWS access keys, SSNs, credit cards with 95%+ severity rules
Open source video
Source video

Concurrency and Recovery Semantics of Checkpoint-Based Agent Persistence

Research question

How does checkpoint/thread-ID persistence behave under concurrent or partially failed long-running agent workflows?

Why

Reliable recovery is critical for production agents, but concurrency and failure scenarios are not addressed in the design.

Agent persistence ensures durability and seamless recovery across runs using thread IDs and checkpoints
Open source video
SQLite for local development, Cloud Spanner for production
Open source video
Source video

Article ideas

4

The MCP Gateway is the New Security Boundary: Why Your Agent Should Never Talk to a Server Directly

Centralizing identity, authentication, and audit in an MCP gateway is non-negotiable for enterprise agents; direct client-to-server access creates unmanageable shadow IT and security gaps.

Angle

Architectural argument with a call to adopt gateway-mediated tool access.

Source video

Context Optimization: The 90% Cost Savings That Might Be a Quality Trap

Treat context construction as an optimizing pipeline with token-cost as a first-class metric, but validate that pruning and compression do not degrade task quality.

Angle

Balanced analysis of cost vs. quality trade-offs with a call for paired evaluations.

Source video

Golden-Path Scaffolding: How to Make Agent Governance the Path of Least Resistance

Encoding production requirements into a dev portal scaffold shifts governance from per-team discipline to platform defaults, increasing compliance and standardizing deployment.

Angle

Platform engineering perspective on using automation to enforce governance.

Source video

Splitting Telemetry: Why Agent Quality Signals and Infrastructure Metrics Must Live Separately

Separating agent-side LLM telemetry from gateway operational telemetry enables independent root-cause analysis and quality evaluation, preventing conflation of model behavior with infrastructure events.

Angle

Observability architecture argument with practical implementation guidance.

Source video

Project ideas

4

ContextOptimizer Quality Benchmark

beyond-evals

The 70-90% token reduction achieved by a context optimizer (select/rerank/dedup/compress) does not significantly degrade task accuracy (within 2%) compared to full context on a representative query set.

Proof of concept

Implement a context optimizer pipeline and run paired evaluations on a QA dataset with and without optimization, measuring both token count and task accuracy (e.g., exact match, F1).

Measurement

Token count reduction and task accuracy difference (e.g., absolute percentage points).

Source video

Gateway Latency Overhead Profiler

gatehouse

The mandatory MCP gateway adds less than 50ms p99 latency to tool calls under load, and its failure mode is graceful with retries.

Proof of concept

Build a mock MCP gateway with JWE token unwrapping and benchmark end-to-end tool-call latency with and without the gateway hop under varying load.

Measurement

Latency percentiles (p50, p95, p99) and error rates.

Source video

PII Trimmer Precision/Recall Auditor

gatehouse

The 95%+ severity PII rules achieve >99% recall on known secret types but may have false positives on benign data containing patterns like credit card numbers in test data.

Proof of concept

Build a labeled corpus of secrets (private keys, AWS keys, SSNs, credit cards) and benign inputs, run the trimmer, and compute precision/recall.

Measurement

Precision, recall, F1, and false positive rate on benign inputs.

Source video

Checkpoint Concurrency Stress Tester

movement-lab

Checkpoint-based agent persistence with thread IDs can correctly recover from crashes and handle concurrent resume without data loss or duplication under production-like load.

Proof of concept

Simulate concurrent workflows with checkpoints in Cloud Spanner, inject failures, and measure recovery correctness.

Measurement

Percentage of workflows that recover correctly, duplication rate, and latency to resume.

Source video

Architectural implications

4

Tool access is mediated by a gateway that carries real identity, with MCP servers hidden behind it.

Before

Agents or clients connect directly to backend APIs and MCP servers.

After

All requests route through an audited gateway that unwraps JWE tokens and enforces identity.

Consequence

Access control and audit centralize at the gateway; direct server exposure and shadow IT become detectable and blockable.

Source video

A dedicated context optimizer layer sits between context assembly and the LLM.

Before

Full retrieved context is passed to the model, with cost managed by prompt tweaks.

After

Context is selected, reranked, deduplicated, and compressed before ingestion.

Consequence

Token cost becomes an engineered, measurable pipeline output rather than an emergent expense.

Source video

Telemetry is split into agent-side LLM signals and gateway operational signals.

Before

A single logging path conflates model behavior with infrastructure events.

After

OTel collectors route traces to Tempo, metrics to Prometheus, and quality/job data to BigQuery.

Consequence

Quality evaluation and infrastructure debugging get independent, queryable pipelines.

Source video

High-risk actions are gated by an explicit HITL approval node in the agent graph.

Before

Agents execute write/sensitive operations autonomously.

After

Write and sensitive actions branch through an execute-vs-reject approval diamond.

Consequence

Catastrophic autonomous modifications are bounded, at the cost of workflow latency and human load.

Source video

Tradeoffs and failure modes

4

Gateway as mandatory chokepoint

Benefit

Centralized identity, authentication, routing, and audit for every tool call.

Cost or risk

Introduces an extra hop and dependency in the critical path; its availability, latency, and JWE handling become systemic risks.

MCP gateway ensures every request runs as a real identity through an audited gateway
Open source video
Source video

Aggressive context optimization

Benefit

70% to 90% token-cost savings demonstrated on a real query.

Cost or risk

Pruning, deduplication, and compression can drop information the model needs, risking quality regressions that are not visible in the token-count metric.

Context optimization saves 70% to 90% in token costs
Open source video
Source video

High-severity PII blocking

Benefit

Prevents secrets and sensitive data from reaching LLMs, traces, or eval stores.

Cost or risk

Severity-thresholded rules may over-block legitimate inputs or miss novel secret formats, creating either false positives or coverage gaps.

Blocking private keys, AWS access keys, SSNs, credit cards with 95%+ severity rules
Open source video
Source video

Human-in-the-loop approval for write/sensitive actions

Benefit

Prevents catastrophic errors from autonomous agent actions.

Cost or risk

Adds human latency and operational burden, limiting throughput and autonomy for legitimate write operations.

Agent framework flow includes approval diamond checking execute vs reject states.
Open source video
Source video

Open questions

5

Do the demonstrated 70–90% token savings preserve task quality, or are they achieved by dropping context the model needed?

Why unresolved

The summary reports a single token-reduction demo but no accuracy or eval comparison on the optimized context.

Research direction

Run paired evaluations on the same query set with and without context optimization, tracking task accuracy against token count.

Source video

What latency and reliability cost does the mandatory gateway (including JWE token un-wrapping) impose on every tool call?

Why unresolved

The summary asserts the gateway pattern but does not quantify its overhead or failure behavior.

Research direction

Benchmark end-to-end tool-call latency and error rates with and without the gateway hop under load.

Source video

How are agent_quality_eval scores defined and kept comparable across hundreds of deployed agents?

Why unresolved

The summary lists the BigQuery eval table but not the scoring methodology or cross-agent normalization.

Research direction

Define a shared eval schema and metric contract so quality scores are comparable across heterogeneous agents.

Source video

What is the false-positive/false-negative rate of the 95%+ severity PII rules, and how are novel secret leaktokens handled?

Why unresolved

Only the rule types and threshold are given; no measured precision/recall is provided.

Research direction

Build a labeled corpus of secrets and benign inputs to measure trimmer precision and recall, including previously unseen formats.

Source video

How does checkpoint/thread-ID persistence behave under concurrent or partially failed long-running agent workflows?

Why unresolved

Persistence is described at a design level (SQLite vs Cloud Spanner) without concurrency or recovery semantics.

Research direction

Test crash-recovery and concurrent-resume scenarios against the checkpoint store in production-like conditions.

Source video

Key claims

7
factualVerification needed

MCP gateway ensures every request runs as a real identity through an audited gateway, and MCP servers never face clients directly.

Evidence

MCP gateway ensures every request runs as a real identity through an audited gateway

Question

What is the gateway's added latency and failure mode when it becomes a mandatory hop for every tool call?

Source video
causalVerification needed

Context optimization saves 70% to 90% in token costs, demonstrated by a drop from 24,000 to 2,400 tokens for the same query.

Evidence

Demo showed token usage dropping from 24,000 tokens down to 2,400 tokens (90% savings) for the same query.

Question

Does the same query produce equivalent answer quality after 90% context reduction?

Source video
causalVerification needed

MCP servers, built with FastMCP wrappers, enforce request filters, limits, and blocked fields so LLMs cannot construct arbitrary or dangerous API calls.

Evidence

MCP servers enforce request filters, limits, and blocked fields

Question

Can a constrained tool surface be bypassed via chained or indirect tool invocations?

Source video
opinionVerification not requested

AI agents cannot be trusted without continuous monitoring, evaluation, and feedback.

Evidence

AI agents cannot be trusted without continuous monitoring, evaluation, and feedback

Source video
factualVerification needed

Agent persistence through thread IDs and checkpoints provides durability and seamless recovery across runs, using SQLite locally and Cloud Spanner in production.

Evidence

Agent persistence ensures durability and seamless recovery across runs using thread IDs and checkpoints

Question

How does checkpoint recovery behave for concurrent or partially failed multi-step workflows?

Source video
causalVerification needed

Enterprise AI agents often fail in production because foundational controls such as gateways, tool execution, tracking, memory, and security are missing.

Evidence

AI agents fail in production when foundational controls are missing

Question

Which of these missing controls is most strongly associated with production failure, and in what proportion?

Source video
factualVerification needed

The dev portal auto-creates a repository with workload identity, memory DB, tracking tables, and secret management, standardizing agent development across the organization.

Evidence

Dev portal creates repository with workload identity, memory DB, tracking tables, and secret management automatically.

Question

Does golden-path scaffolding actually improve compliance rates versus team-managed setup?

Source video

Connections

4