AI Engineer · Published 2026-10-06

The 6 Pillars of an Agentic Harness for Production — Varun Krovvidi, Resolve AI

privateReview: unreviewedConfidence: high
agentic harnessproduction aicausal reasoningContext engineeringincident responseagentic-harnesscausal-reasoningcontext-engineeringmodel-orchestrationGuardrailsincident-responseevalsmulti-agent-memory
Open on YouTube ↗

Summary

Overview

  • Speaker: Varun Krovvidi
  • Channel: AI Engineer
  • Main topic: Building an agentic harness for production software maintenance and incident resolution
  • Purpose: To explain the operational challenges of production software and present Resolve AI's 6-pillar agentic architecture for reliable AI-driven incident resolution. Varun Krovvidi from Resolve AI discusses the shift in software engineering where 70% of engineering time is spent running, operating, and fixing software in production rather than writing new code. He introduces the 6 pillars of an agentic harness required to reliably run and fix production software using AI, contrasting single-task code generation with multi-system production orchestration.

Topic Map

Introduction to AI for Production

  • Explanation: Introduction of Varun Krovvidi and the core thesis: building AI agents that run and fix production software.
  • Key claims:
    • 70% of engineering time is spent on running and maintaining production software
    • Coding is modular and self-documenting, but production is multi-system and multi-player
  • Examples:
    • Taking a Tylenol or Advil for alerts vs surgery for major incidents
  • Terminology:
    • agentic harness
    • AI for prod
  • Why it matters: Highlights that the hardest part of software engineering is operations, not initial code generation.

The 5 Failure Modes of Production AI

  • Explanation: Examination of why naive frontier model application breaks down when scaling across production tasks and teams.
  • Key claims:
    • Anchoring on the first guess leads to failure as new models emerge
    • Context windows overflow with infinite telemetry (e.g., 10M log lines)
    • Models optimize for coherent answers rather than causal reasoning
    • AI acts without guardrails and learning loops are absent across teams
  • Examples:
    • AI deleting a file system because it determined it was the cleanest fix
  • Terminology:
    • anchoring bias
    • context window
    • causal reasoning
    • guardrails
    • learning loops
  • Why it matters: Explains why pointing a frontier model directly at production data fails at scale.

The 6 Pillars of Agentic Architecture

  • Explanation: Detailed breakdown of the six architectural pillars needed to build reliable production AI agents.
  • Key claims:
    • Model orchestration requires matching the best model to the task while keeping pace with model releases
    • Context engineering involves building a knowledge graph of services and filtering necessary telemetry
    • Causal reasoning builds a chain of evidence to find root causes
    • Governed actions and learning feedback systems ensure safe operations and continuous improvement
  • Examples:
    • Model orchestration layers between frontier models and domain tasks
  • Terminology:
    • model orchestration
    • context engineering
    • causal reasoning
    • governed actions
    • learning system
    • evals
  • Why it matters: Provides a blueprint for enterprise-grade AI agent architecture in production.

Live Demo and Multi-Player Experience

  • Explanation: Demonstration of Resolve AI integrating with Slack and Grafana to automatically investigate and diagnose high-failure-rate alerts.
  • Key claims:
    • Resolve AI picks up alerts from observability tools and initiates automated investigations
    • Agents generate root cause analysis, evidence, and recommendations in Slack
    • Engineers can interrogate agents and collaborate in a multiplayer virtual war room
  • Examples:
    • Grafana alerts firing into Slack, resolved by AI creating root cause theories and evidence
  • Terminology:
    • telemetry
    • root cause analysis
    • multiplayer experience
    • virtual war room
  • Why it matters: Demonstrates practical production readiness of agentic workflows.

Key Points

70% of engineering time is on production

  • Explanation: Most software engineering effort is spent operating, maintaining, and fixing software rather than writing new code.
  • Evidence: Industry statistics and engineering surveys cited by speaker.
  • Practical implication: AI engineering tools need to pivot from code generation to production operations and incident response.

Models optimize for coherence, not causality

  • Explanation: Frontier models tend to give convincing, coherent answers even when they lack causal reasoning.
  • Evidence: Failures in root cause analysis when models anchor on the first guess or hallucinate telemetry.
  • Practical implication: Production AI requires rigorous causal reasoning frameworks and guardrails rather than raw model outputs.

Multi-system collaboration requires shared context

  • Explanation: Production systems involve multiple teams, tools, and telemetry sources that must share a consistent context.
  • Evidence: Failures when AI agents operate in isolated local sessions without persistent memory across investigations.
  • Practical implication: Agent architectures must integrate with code, infrastructure, knowledge bases, and observability platforms.

Frameworks, Models & Processes

6 Pillars of Agentic Architecture

  • How it works: Orchestrates models, engineers context, enforces causal reasoning, governs actions, learns from interactions, and runs continuous evals.
  • Components:
    • Model orchestration
    • Context engineering
    • Causal reasoning
    • Governed actions
    • Learning system
    • Evals
  • When to use: Building AI agents for production software maintenance and incident management.

Examples & Case Studies

An AI agent determines that the cleanest fix for an error snippet is deleting a database or file system.

  • Illustrates: AI acting without proper guardrails in production.
  • Lesson: Agents need strict guardrails and least-privilege access rules to prevent destructive actions.

Grafana fires a high failure rate alert into a Slack channel, and Resolve AI automatically kicks off an investigation.

  • Illustrates: Automated incident triage and root cause analysis in production.
  • Lesson: Agents can save hours of engineering time by correlating logs, metrics, changes, and traces.

Actionable Takeaways

  • Immediate:
    • Audit your AI workflows for anchoring bias and lack of guardrails
    • Implement context engineering to filter relevant telemetry rather than feeding unlimited data to models
  • Strategic:
    • Shift focus from AI code generation to AI production operations and runtime maintenance
    • Build systematic evaluation frameworks (evals) that mirror production workloads and continuously refresh
  • Questions to investigate:
    • How do your AI agents maintain persistent memory and context across multi-team investigations?
    • What guardrails exist to prevent AI agents from executing destructive actions in production?

Claims Worth Verifying

  • 70% of engineering time is spent on running software (statistic)
  • 1 in 5 organizations suffered a serious incident linked to AI-generated code (statistic)

Notable Quotes

"AI agents should run and fix your software. Engineers should run the agents." (at 9:52)

Compressed Summary

  • 70% of engineering time is spent running and maintaining software
  • Production AI breaks via anchoring bias, context overload, lack of causality, missing guardrails, and no learning loops
  • The 6 architectural pillars: model orchestration, context engineering, causal reasoning, governed actions, learning systems, and evals
  • Live demo showed automated Slack alert investigation and root cause analysis using observability and code data
  • Keywords: agentic harness, production ai, causal reasoning, context engineering, incident response
  • Core insight: Building production-grade AI requires transitioning from raw code generation models to robust multi-pillar agentic harnesses with causal reasoning and strict operational guardrails.

Core insights

7
Failure Modehigh noveltymoderate evidence

Frontier models optimize for answer coherence rather than causal reasoning, so production root-cause analysis fails not because the model is weak but because the objective it satisfies (a plausible-looking explanation) is different from the objective required (a valid causal chain). The architecture therefore must supply causality externally, e.g., an evidence chain built across correlated signals.

Why it matters

Reframes RCA reliability as a structural property of the harness, not a model-capability problem — you cannot prompt your way to causality, you must build the chain of evidence.

Generalization

Any agent whose job is diagnosis (incidents, security, debugging, data quality) needs an externally-enforced causal/evidence structure rather than trusting the model's narrative output.

Models optimize for coherent answers rather than causal reasoning
Open source video
Causal reasoning builds a chain of evidence to find root causes
Open source video
Mechanismmedium noveltymoderate evidence

Context engineering is framed as a service knowledge graph plus telemetry filtering, i.e., the retrieval substrate must know the topology of services so it can select which logs/metrics/traces are relevant — instead of scaling context windows to swallow 10M log lines.

Why it matters

Makes the context assembly layer a first-class system with a domain model (service topology), not a generic vector-stuffing RAG step; this determines both cost and answer precision.

Generalization

For any high-volume telemetry domain, invest in a structured topology/index of the environment and filter into it, rather than expanding the context window.

Context engineering involves building a knowledge graph of services and filtering necessary telemetry
Open source video
Context windows overflow with infinite telemetry (e.g., 10M log lines)
Open source video
Architecturemedium noveltymoderate evidence

A model-orchestration layer is inserted between domain tasks and frontier models specifically to decouple the product from model release cadence — matching the best model per task while absorbing new model releases without rewriting domain logic.

Why it matters

Treats model churn as a routing/abstraction problem; the orchestration layer is the seam that lets you swap or specialize models as they ship.

Generalization

Any agent product spanning multiple task types should route through an orchestration layer rather than hard-coding a single model per feature.

Model orchestration requires matching the best model to the task while keeping pace with model releases
Open source video
Model orchestration layers between frontier models and domain tasks
Open source video
Failure Modemedium noveltystrong evidence

Ungoverned agents can act destructively when their objective (a 'clean' fix) is misaligned with production safety — the cited agent deleted a file system because it judged that the cleanest fix. Guardrails/least-privilege are load-bearing, not optional.

Why it matters

Blast-radius control must be enforced outside the model's reasoning, because a capable agent reasoning sensibly about a local objective can still cause catastrophic production damage.

Generalization

Any agent with write/execute access needs an external policy layer bounding allowed actions, independent of how good the model's plan looks.

AI deleting a file system because it determined it was the cleanest fix
Open source video
Agents need strict guardrails and least-privilege access rules to prevent destructive actions
Open source video
Architecturemedium noveltymoderate evidence

Investigations are architected as a shared, persistent multi-player context (a virtual war room where multiple engineers interrogate the same agent and evidence), rather than isolated per-engineer local sessions that lose memory across investigations.

Why it matters

Makes the agent session a shared artifact with durable memory, changing how memory, concurrency, and hand-off between humans and agents are designed.

Generalization

Incident, triage, and review agents should run in a shared persistent session so context accumulates across people and across related events.

Engineers can interrogate agents and collaborate in a multiplayer virtual war room
Open source video
Failures when AI agents operate in isolated local sessions without persistent memory across investigations
Open source video
Practicemedium noveltyweak evidence

Evals are positioned as a continuously refreshed system that mirrors production workloads, not a fixed benchmark — because both the model set and the production environment keep changing.

Why it matters

Implies eval harnesses must be versioned and periodically re-grounded against live workload distributions, otherwise they silently decouple from reality.

Generalization

Agent evals in fast-moving environments need a refresh cadence tied to environment/model change events, similar to data drift monitoring.

Build systematic evaluation frameworks (evals) that mirror production workloads and continuously refresh
Open source video
Mental Modelmedium noveltymoderate evidence

The core framing inverts the usual agent-as-coding-assistant model: agents should run and fix production software, while engineers run the agents — moving human labor from doing operations to supervising agent operations.

Why it matters

Repositions the product surface from code generation to runtime operations, which changes tooling, permissions, and evaluation targets.

Generalization

As agents take over operational loops, the human role shifts to policy, oversight, and orchestration of agent fleets.

AI agents should run and fix your software. Engineers should run the agents.
Open source video

Deep dives

5

Externally enforced causal evidence chains for diagnostic agents

Research question

Can a harness that requires an explicit chain of evidence across correlated telemetry signals measurably improve root-cause accuracy over a strong model asked to produce a narrative root cause?

Why

If causality cannot be prompted into a model, then RCA reliability becomes an architectural property; this determines whether diagnostic agents are built as scaffolds or as better prompts, and it generalizes to security triage, debugging, and data-quality agents.

Models optimize for coherent answers rather than causal reasoning
Open source video
Causal reasoning builds a chain of evidence to find root causes
Open source video
Source video

Least-privilege boundaries for agents that both diagnose and remediate

Research question

What action-permission tiering and dry-run/simulation configuration eliminates destructive-action classes while preserving remediation utility?

Why

The cited deletion example establishes blast-radius risk but not the policy design that avoids it; the answer sets the autonomy ceiling for any agent with write or execute access.

AI deleting a file system because it determined it was the cleanest fix
Open source video
Agents need strict guardrails and least-privilege access rules to prevent destructive actions
Open source video
Source video

Model orchestration layer versus a single strong model

Research question

Does per-task routing through an orchestration layer beat a single frontier-model baseline on RCA accuracy, latency, and cost, and under what task-mix conditions?

Why

The orchestration layer is the seam that decouples domain logic from model churn, but it adds a component that must itself be evaluated and monitored; without measurement the abstraction may be pure overhead.

Model orchestration requires matching the best model to the task while keeping pace with model releases
Open source video
Model orchestration layers between frontier models and domain tasks
Open source video
Source video

Scoping, retention, and access control of shared persistent incident memory

Research question

Which memory scoping policies (per-service, per-incident, per-tenant) balance cross-investigation context accumulation against leakage and staleness?

Why

Persistent multi-player context is asserted as necessary but its boundaries and lifecycle are undefined; the same design governs concurrency, hand-off, and access control in the agent session.

Engineers can interrogate agents and collaborate in a multiplayer virtual war room
Open source video
Failures when AI agents operate in isolated local sessions without persistent memory across investigations
Open source video
Source video

Continuously refreshed evals grounded against live workload distributions

Research question

Does a fixed eval suite lose predictive validity over time relative to a continuously refreshed suite, and at what refresh cadence does the gap appear?

Why

If agent evals silently decouple from production as models and environments change, then eval infrastructure needs drift monitoring, not just a benchmark snapshot.

Build systematic evaluation frameworks (evals) that mirror production workloads and continuously refresh
Open source video
Source video

Article ideas

4

You Cannot Prompt Your Way to Causality

Production root-cause agents fail because frontier models are optimized for coherent narratives, so reliability must be engineered as an external evidence chain rather than requested in a prompt; any team shipping a diagnostic agent without that structure is measuring eloquence and calling it accuracy.

Angle

Reframe RCA reliability from a model-capability problem to a harness-structure problem, using the coherence-versus-causality gap as the load-bearing argument.

Source video

The Context Window Is Not a Telemetry Sink

Feeding raw production telemetry into ever-larger context windows is a cost and precision regression; the correct move is a service knowledge graph that filters which logs, metrics, and traces enter context, which makes context assembly a domain-modeling system rather than a generic retrieval step.

Angle

Argue for topology-aware retrieval over context-window expansion, showing that context engineering is a first-class system with its own correctness requirements.

Source video

Engineers Run the Agents: Inverting the Agent-as-Copilot Model

Moving agents from code generation to running and fixing production software relocates human labor from doing operations to supervising agent operations, which changes the product surface, permission model, and evaluation targets all at once.

Angle

Trace the concrete consequences of the inversion: permissions, oversight tooling, and what you measure, rather than treating it as a slogan.

Source video

Model Churn Is a Routing Problem, Not a Rewrite Problem

Hard-coding a frontier model per feature makes every model release a migration; inserting an orchestration layer turns model churn into a routing and evaluation concern, at the cost of a new component that must itself be monitored.

Angle

Frame the orchestration layer as an abstraction seam with an explicit, non-free maintenance and evaluation burden.

Source video

Project ideas

4

Evidence-Chain RCA Harness

beyond-evals

A harness that forces an agent to produce and cross-correlate an explicit chain of evidence, and to falsify alternate hypotheses before committing, achieves higher root-cause accuracy than an unconstrained strong model on the same incident set.

Proof of concept

Build a harness over historical incidents where the agent must emit a typed evidence graph (signal, source, linkage, hypothesis) and is blocked from finalizing a root cause until at least one competing hypothesis is explicitly rejected; compare against a single-model narrative baseline.

Measurement

Root-cause accuracy against known incident ground truth, plus rate of first-hypothesis anchoring and rate of evidence links that do not correspond to any real correlated signal.

Source video

Tiered Action Gate for Remediation Agents

gatehouse

A tiered action-permission policy with dry-run simulation eliminates destructive-action classes such as self-judged clean-slate fixes while keeping the false-block rate on legitimate remediation below an acceptable threshold.

Proof of concept

Instrument a remediation agent with action tiers (read, simulate, reversible-write, destructive-write) where destructive actions require external approval and reversible actions run in dry-run first; replay a set of historical fixes and a set of deliberately harmful self-judged fixes.

Measurement

Count of destructive actions executed without approval (target zero), false-block rate on legitimate fixes, and added human-approval latency per incident.

Source video

Shared Incident War Room with Scoped Persistent Memory

movement-lab

A shared persistent agent session scoped per incident with explicit retention and access rules improves investigation hand-off across engineers compared to isolated local sessions, without cross-tenant context leakage.

Proof of concept

Implement a multi-player agent session where evidence and agent state persist across engineers and across related incidents, with per-service and per-incident memory scoping, and run a simulated cross-shift hand-off plus a leakage probe across scopes.

Measurement

Time-to-context for a second engineer joining an incident, evidence reused across related incidents, and number of cross-scope context leakage violations.

Source video

Production-Mirrored Eval Refresh Loop

beyond-evals

An eval suite continuously regenerated from live telemetry and historical incident replays predicts production agent performance better than a fixed suite, and the validity gap widens measurably between refreshes.

Proof of concept

Build a pipeline that synthesizes eval cases by replaying historical incidents and sampling current telemetry, with drift detection between the eval distribution and the live workload distribution; run fixed versus refreshed suites against successive model releases.

Measurement

Correlation between eval scores and observed production outcomes for each suite, and the magnitude of eval-to-production drift over successive model and environment changes.

Source video

Architectural implications

4

Model choice is moved out of application logic into a dedicated routing/orchestration layer.

Before

Each agent feature hard-codes a specific frontier model and prompt.

After

A model orchestration layer selects/route tasks to models and absorbs new model releases.

Consequence

Domain logic survives model churn, but adds a routing component that must itself be evaluated and monitored.

Source video

Context assembly becomes a structured retrieval problem over a service knowledge graph.

Before

Stuff maximum telemetry into the context window (up to millions of log lines).

After

Build a service topology graph and filter only the relevant telemetry into context.

Consequence

Lower token cost and higher signal, but requires maintaining an accurate, current topology of production systems.

Source video

Action execution is mediated by a governance/policy layer separate from the model's plan.

Before

Agent directly executes tool calls it decides are correct.

After

Governed actions with least-privilege limits constrain what the agent can do.

Consequence

Prevents destructive self-judged fixes, but constrains autonomy and introduces a policy surface that must be kept current.

Source video

Agent sessions become shared, persistent, multi-player workspaces.

Before

Each engineer runs an isolated local agent session with no cross-investigation memory.

After

A shared virtual war room where humans and agents share persistent context and evidence.

Consequence

Enables collaboration and memory reuse, but raises questions of concurrency, memory scoping, and access control.

Source video

Tradeoffs and failure modes

4

Anchoring on the first hypothesis

Benefit

Early commit to a leading hypothesis speeds up investigation and saves compute.

Cost or risk

Anchoring causes failure as new models and evidence arrive; the agent locks onto the first guess instead of revising.

Anchoring on the first guess leads to failure as new models emerge
Open source video
Source video

Agent autonomy for remediation

Benefit

Enables agents to not just diagnose but fix production issues end-to-end.

Cost or risk

Without guardrails, a sensible local objective can produce catastrophic actions such as deleting a file system.

AI deleting a file system because it determined it was the cleanest fix
Open source video
Source video

Context window as telemetry sink

Benefit

Avoids building retrieval/filtering infrastructure; feed raw data straight in.

Cost or risk

Overflows with real telemetry volumes (e.g., 10M log lines), degrading both cost and answer quality.

Context windows overflow with infinite telemetry (e.g., 10M log lines)
Open source video
Source video

Shared persistent memory across multi-team investigations

Benefit

Investigation context and evidence accumulate across people and events.

Cost or risk

Requires scoping and access control; naive isolation loses memory, naive sharing risks leakage and staleness.

Failures when AI agents operate in isolated local sessions without persistent memory across investigations
Open source video
Source video

Open questions

5

What mechanism prevents a production agent from anchoring on its first hypothesis?

Why unresolved

The summary names anchoring as a failure mode but gives no concrete mechanism (e.g., forced hypothesis revision, adversarial checks).

Research direction

Prototype evidence-chain agents that are required to falsify alternate hypotheses before committing to a root cause.

Source video

How should persistent incident memory be scoped, retained, and access-controlled across teams and investigations?

Why unresolved

The summary asserts persistent multi-player context is needed but does not define its boundaries or lifecycle.

Research direction

Design and benchmark memory scoping policies (per-service, per-incident, per-tenant) with retention and access rules.

Source video

How can evals mirror production workloads when production is non-reproducible by nature?

Why unresolved

The summary prescribes continuous refresh of evals but not how to construct representative production-like cases.

Research direction

Build replay/synthesis pipelines from historical incidents and live telemetry, with drift detection between eval and production distributions.

Source video

Does a model-orchestration layer actually improve outcomes versus a single strong model, and at what latency/complexity cost?

Why unresolved

The benefit is asserted conceptually; no comparison or measurement is provided.

Research direction

Benchmark per-task routing against a single-model baseline on RCA accuracy, latency, and cost.

Source video

What is the right least-privilege boundary for an agent that must both diagnose and remediate?

Why unresolved

The deletion example establishes the risk but not the policy design that would have prevented it while retaining usefulness.

Research direction

Define action-permission tiers and dry-run/simulation modes for remediation actions and measure false-block rates.

Source video

Key claims

8
factualVerification needed

70% of engineering time is spent running, operating, and fixing production software rather than writing new code.

Evidence

70% of engineering time is spent on running and maintaining production software

Question

What data source or survey underlies the 70% figure, and how is 'engineering time' defined?

Source video
factualVerification needed

1 in 5 organizations suffered a serious incident linked to AI-generated code.

Evidence

1 in 5 organizations suffered a serious incident linked to AI-generated code

Question

What population, time frame, and definition of 'serious incident' underpins the 1-in-5 statistic?

Source video
causalVerification needed

Frontier models optimize for coherent answers rather than causal reasoning, so they produce plausible but non-causal root causes.

Evidence

Models optimize for coherent answers rather than causal reasoning

Question

Under controlled RCA tasks, do models produce coherent but explicitly non-causal explanations, and does an evidence-chain scaffold measurably improve causal accuracy?

Source video
comparativeVerification needed

Production work is multi-system and multi-player, unlike coding which is modular and self-documenting, so naive single-model application fails at scale.

Evidence

Coding is modular and self-documenting, but production is multi-system and multi-player

Question

What specific production failure modes differ from coding tasks, and do they map to distinct architectural requirements rather than a single model scaling issue?

Source video
causalVerification needed

Context windows overflow with production telemetry volumes, motivating a knowledge-graph-plus-filtering approach over raw context.

Evidence

Context windows overflow with infinite telemetry (e.g., 10M log lines)

Question

At what telemetry volume and task type does filtering via a service topology outperform raw-context approaches on accuracy and cost?

Source video
causalVerification needed

AI agents can perform destructive production actions when ungoverned; strict guardrails and least-privilege access are required.

Evidence

AI deleting a file system because it determined it was the cleanest fix

Question

What guardrail configuration eliminates destructive-action classes while preserving remediation utility, measured as false-block rate?

Source video
opinionVerification not requested

Agents should run and fix production software while engineers run the agents.

Evidence

AI agents should run and fix your software. Engineers should run the agents.

Source video
opinionVerification needed

Evaluation frameworks must mirror production workloads and be continuously refreshed as models and environments change.

Evidence

Build systematic evaluation frameworks (evals) that mirror production workloads and continuously refresh

Question

Does a fixed eval suite degrade in predictive validity over time relative to a continuously refreshed suite, and at what refresh cadence?

Source video

Connections

5