AI Engineer · Published 2026-09-26

Long-Horizon Agents Need Experiments, Not Just Prompts — Erina Karati

privateReview: unreviewedConfidence: high
autoresearchmulti-agentlong-horizonevaluationprovenancesimulationlong-horizon-agentsmulti-agent-evaluationmemory-architecturescorecard-designself-improvement-guardrailssocial-consistency
Open on YouTube ↗

Summary

Overview

  • Speaker: Erina Karati
  • Channel: AI Engineer
  • Main topic: Autoresearch in Multi-Agent AI Villages and Evaluating Long-Horizon Agents
  • Purpose: To explain the limitations of short-term agent evaluation, introduce Project Paradox and the Autoresearch pattern, and provide practical engineering lessons for building robust long-horizon multi-agent systems. Erina Karati discusses the challenges of evaluating and improving multi-agent AI systems that maintain state over long periods, using Project Paradox (a video game-like AI village) as a case study. She introduces the concept of 'Autoresearch'—a meta-system outside the village that runs automated experimental loops to score behavior, diagnose failures, propose policy changes, and rerun scenarios, emphasizing that long-horizon agents need experiments rather than just manual prompt tuning.

Topic Map

Project Paradox and Multi-Agent Frameworks

  • Explanation: Developed at Supercell's AI Innovation Lab by Erina Karati and Arunachalam Manikandan, Project Paradox is a modular AI framework enabling developers to plug intelligent autonomous agents into video games to act as dynamic game companions.
  • Key claims:
    • Agents can move with intent guided by memory, emotion, and curiosity.
    • Agents interact with the world by picking up objects and understanding context.
    • Agents react to events and start conversations that affect beliefs and emotions.
  • Examples:
    • Character 'Adam' reacting to a gravestone or 'Blossom' agreeing to bring a pastry to the fountain.
  • Terminology:
    • Project Paradox
    • Supercell AI Innovation Lab
    • modular AI framework
    • dynamic game companions
  • Why it matters: Demonstrates how autonomous agents can transform static NPCs into dynamic characters within complex environments.

Under the Hood of Multi-Agent Architecture

  • Explanation: Explains the stateful architecture behind individual agents, including per-agent RAG memory, emotion vectors, belief/trust matrices, and core memory importance scoring.
  • Key claims:
    • Per-agent RAG memory prevents memory from bleeding across agents by default.
    • Emotion vectors track values like joy, sadness, fear, anger, and disgust.
    • Belief/trust matrices act as trust metrics updated after interactions.
    • Importance scores determine whether memory events cross a threshold to be stored in a separate cache.
  • Examples:
    • An agent forgetting what they had for dinner a few days ago versus remembering a murder.
  • Terminology:
    • per-agent RAG memory
    • emotion vector
    • belief/trust matrix
    • importance score
    • core threshold
  • Why it matters: Proper state management is critical for agents to maintain coherent internal states and personalities over time.

The Problem with Long-Horizon Social Consistency

  • Explanation: While short-term agency works well (planning, moving, talking), longer horizons reveal weaknesses in social consistency, such as rumor propagation turning into false facts and loss of source attribution.
  • Key claims:
    • Short-term agency worked, but long-horizon social consistency was harder.
    • Information moving across people and time causes failures in source attribution and rumor accuracy.
    • Systems may remember rough topics but lose the source of the topic.
  • Examples:
    • An agent spreading a rumor about mango sales that eventually gets stated as absolute fact by another agent.
  • Terminology:
    • social consistency
    • rumor propagation
    • source attribution
    • long-horizon agency
  • Why it matters: Highlighting why multi-agent systems degrade over time without proper systemic checks.

The Bridge to Autoresearch

  • Explanation: Bridging Andrej Karpathy's autoresearch concept to multi-agent systems: instead of manual prompt tuning, use an automated experimental loop to run scenarios, collect traces, score behavior, and apply policy edits.
  • Key claims:
    • Project Paradox becomes the lab bench, with Autoresearch as the experimental loop.
    • Instead of tuning prompts manually, define scenario suites, run agents, collect traces, score behavior, and change policy surfaces.
    • Evaluate a run, not one response.
  • Examples:
    • Defining scenario suites for public fact diffusion, rumor uncertainty, and replanning.
  • Terminology:
    • Autoresearch
    • scenario suite
    • repeated runs
    • policy surface
    • guardrails
  • Why it matters: Shifts agent optimization from manual guesswork to automated experimentation.

Controlled Scenarios and Balanced Scorecards

  • Explanation: Social behavior is fuzzy, requiring controlled scenarios (public fact diffusion, rumor uncertainty, replanning) and a balanced scorecard rather than a single vague metric.
  • Key claims:
    • Controlled scenarios make social behavior measurable.
    • A balanced scorecard prevents single-metric failure (e.g., optimizing only for diffusion causes oversharing; optimizing only for recall causes stale memory noise).
    • Metrics should cover fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, and dialogue quality.
  • Examples:
    • Measuring reach and source retention for public fact diffusion.
  • Terminology:
    • controlled scenarios
    • balanced scorecard
    • fact fidelity
    • provenance accuracy
    • misinformation resistance
    • trust calibration
  • Why it matters: Ensures that system optimizations do not inadvertently break other crucial agent behaviors.

Practical Lessons and Recipe for Long-Horizon Agents

  • Explanation: Outlines a 7-step recipe and practical engineering lessons for building long-horizon agents.
  • Key claims:
    • Memory alone is not enough for long-horizon agents; memory needs provenance.
    • Keep the editable policy surface small (don't let AI rewrite everything).
    • Rollback is not optional.
    • For long-running agents, the unit of evaluation is the run.
  • Examples:
    • Freezing the simulation harness while allowing policy changes on retrieval and memory write policies.
  • Terminology:
    • simulation harness
    • policy surface
    • provenance
    • rollback
    • unit of evaluation
  • Why it matters: Provides actionable guardrails and engineering patterns for deploying reliable long-horizon agents.

Key Points

Evaluate runs, not single responses

  • Explanation: Long-horizon agent evaluation requires assessing entire simulation runs across controlled scenarios rather than inspecting individual LLM responses.
  • Evidence: Comparison between manual prompt loops and autoresearch scenario loops.
  • Practical implication: Build evaluation harnesses that capture multi-turn, multi-agent state evolution over time.

Memory needs provenance

  • Explanation: Agents need to know where information came from (firsthand, secondhand, verified, or uncertain) to prevent rumors from becoming unverified facts.
  • Evidence: Failures in rumor propagation and source attribution in village simulations.
  • Practical implication: Include provenance metadata alongside stored memories and retrieval policies.

Do not let the AI rewrite everything

  • Explanation: Freezing evaluation harnesses, scenario suites, and metric formulas while keeping a small editable policy surface prevents reward hacking and system destabilization.
  • Evidence: Distinction between frozen components and editable policy surfaces.
  • Practical implication: Confine autonomous optimization loops to restricted policy spaces like prompt templates and retrieval settings.

Frameworks, Models & Processes

Autoresearch Loop

  • How it works: A meta-system outside the agent environment that runs scenarios, traces behavior, scores results against ground truth, proposes policy adjustments, and reruns tests.
  • Components:
    • scenario suite
    • repeated runs
    • metrics scorecard
    • diagnosis / policy edit
    • keep / rollback decision
  • When to use: When optimizing multi-agent social behavior and long-horizon policy parameters.

Recipe for Long-Horizon Agents

  • How it works: A 7-step engineering pattern ensuring controlled evaluation and safe automated policy iteration.
  • Components:
    • Freeze the simulation harness
    • Define controlled behavioral scenarios
    • Log traces, not just final answers
    • Score a balanced behavioral scorecard
    • Expose a small policy surface
    • Search over changes
    • Keep only changes that survive measurement
  • When to use: When engineering stateful, long-running agent systems across gaming, support, coding, or workflow automation.

Examples & Case Studies

An agent in Project Paradox successfully picked up a pastry and brought it to the fountain as requested by the player, following a planned sequence of actions.

  • Illustrates: Short-term agency and intent execution working correctly.
  • Lesson: Short-term agent actions and interactions with the environment are achievable with current LLM architectures.

An agent spread a rumor about mango sales, which was passed to another agent and eventually treated as an absolute fact by a third agent.

  • Illustrates: The breakdown of social consistency and provenance over longer horizons.
  • Lesson: Without provenance tracking and provenance-aware retrieval, misinformation propagates unchecked in multi-agent systems.

Actionable Takeaways

  • Immediate:
    • Stop relying on manual prompt tuning for multi-agent systems.
    • Implement balanced scorecards instead of single-metric optimization.
    • Freeze your evaluation harness and scenario suites.
  • Strategic:
    • Treat long-horizon agents as experimental systems requiring automated search loops.
    • Design policy surfaces to be small and controllable to prevent reward hacking.
    • Maintain explicit provenance for all agent memories.
  • Questions to investigate:
    • How do we scale autoresearch loops to larger agent populations?
    • What are the best automated scoring functions for subjective social interactions?
    • How can rollback mechanisms be automated safely in production agent workflows?

Claims Worth Verifying

  • Autoresearch loops can autonomously edit policy surfaces to improve multi-agent social consistency scores. (technical)
  • Balanced scorecards prevent optimization degradation across multi-metric agent evaluation. (architectural)

Notable Quotes

"How do we evaluate and improve agents that carry state over long period of time?" (at 0:35) "Long-horizon agents need experiments, not just prompts." "Memory alone is not enough for long-horizon agents." (at 28:26)

Compressed Summary

  • Project Paradox is a modular framework for multi-agent AI villages in video games.
  • Long-horizon agents suffer from broken social consistency, lost provenance, and rumor corruption.
  • Autoresearch uses automated experimental loops to score runs and tune policy surfaces.
  • Balanced scorecards prevent single-metric optimization failures.
  • The recipe for long-horizon agents: freeze the harness, define scenarios, log traces, score balanced metrics, limit policy surfaces, and enforce rollbacks.
  • Keywords: autoresearch, multi-agent, long-horizon, evaluation, provenance, simulation
  • Core insight: Long-horizon multi-agent systems cannot be reliably improved via manual prompt tuning; they require automated experimental loops (autoresearch), controlled scenario suites, and balanced scorecards.

Core insights

7
Architecturemedium noveltymoderate evidence

Treat the editable surface of a self-improving agent system as a narrow, explicitly bounded 'policy surface' — the autoresearch loop may propose edits only to retrieval and memory-write policies while the simulation harness and scenario suites stay frozen.

Why it matters

Defines an architectural boundary for autonomous self-improvement: if the optimizer can rewrite the harness, the scenario suite, and the metrics, it can trivially game its own evaluation. Freezing the bench and constraining the writable region is what keeps automated policy search trustworthy.

Generalization

Any system with a closed-loop optimizer (RLHF data selection, prompt optimizers, auto-eval pipelines, self-modifying scaffolds) should separate an immutable evaluation harness from a small, enumerated set of tunable parameters.

Keep the editable policy surface small (don't let AI rewrite everything).
Open source video
Freezing the simulation harness while allowing policy changes on retrieval and memory write policies.
Open source video
Mental Modelmedium noveltystrong evidence

Completed run over a scenario suite is the atomic unit of evaluation for long-horizon agents, not the individual LLM response.

Why it matters

Single-response evals cannot observe the failure classes that matter here — rumor propagation, source attrition, trust drift — because those only emerge across many turns and multiple agents. Choosing the run as the unit forces investment in trace capture, state snapshots, and multi-agent harnesses.

Generalization

Applies to any stateful agent (coding agents, ops agents, negotiation agents) where quality is a trajectory property rather than a per-turn property; it changes what your logging and CI infrastructure must retain.

Evaluate a run, not one response.
Open source video
For long-running agents, the unit of evaluation is the run.
Open source video
Mechanismhigh noveltystrong evidence

Memory requires provenance metadata (firsthand, secondhand, verified, uncertain) as a first-class field, not just content, because information crossing agents and time loses its source and degrades into asserted fact.

Why it matters

Names the concrete failure: an agent remembers the topic but not who said it, so a rumor about mango sales is later repeated as absolute fact. A content-only memory store has no representation to express uncertainty, so no retrieval or prompting strategy can recover veracity downstream.

Generalization

Extends to RAG over human documents, tool outputs, and inter-agent messages: every write should carry a source class and trust level so downstream consumers can weight or refuse it.

Memory alone is not enough for long-horizon agents; memory needs provenance.
Open source video
Systems may remember rough topics but lose the source of the topic.
Open source video
An agent spreading a rumor about mango sales that eventually gets stated as absolute fact by another agent.
Open source video
Tradeoffmedium noveltymoderate evidence

Optimizing a single social-behavior metric causes predictable degradation elsewhere — pushing diffusion produces oversharing, pushing recall produces stale-memory noise — so evaluation must be a balanced scorecard over fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, and dialogue quality.

Why it matters

Gives a concrete multi-objective evaluation rubric and identifies the specific coupling between metrics, which is the failure mode most auto-optimization loops will hit

Generalization

Any fuzzy objective under automated optimization needs an anti-Goodhart scorecard; the named axes transfer to customer-support agents, social/community agents, and knowledge-base curation.

A balanced scorecard prevents single-metric failure (e.g., optimizing only for diffusion causes oversharing; optimizing only for recall causes stale memory noise).
Open source video
Metrics should cover fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, and dialogue quality.
Open source video
Practicehigh noveltymoderate evidence

Replace manual prompt tuning with an experimental loop that defines scenario suites, runs agents repeatedly, collects traces, scores behavior, diagnoses failures, and applies policy edits — with the agent framework serving as lab bench and the loop as the experiment.

Why it matters

Reframes agent improvement as an offline, repeatable experimental discipline rather than interactive prompt tinkering, which implies logging, reproducibility, and regression infrastructure comparable to a research lab.

Generalization

Maps directly onto eval-driven development and 'autoresearch' for any agentic product, including coding agents and tool-using pipelines, where scenario suites can be assembled from production traces.

Project Paradox becomes the lab bench, with Autoresearch as the experimental loop.
Open source video
Instead of tuning prompts manually, define scenario suites, run agents, collect traces, score behavior, and change policy surfaces.
Open source video
Mechanismmedium noveltymoderate evidence

Per-agent memory isolation is the default (memory does not bleed across agents), and retention is gated by an importance score crossing a core threshold into a separate cache.

Why it matters

Two separable design decisions: partition memory by agent identity for state coherence, and use a scored promotion path so trivial events decay while high-salience events (dinner vs. a murder) are retained in a durable tier.

Generalization

Suggests a tiered memory architecture with explicit write-scoring and a promotion threshold, plus per-principal namespaces — applicable to single-agent long-running assistants too.

Per-agent RAG memory prevents memory from bleeding across agents by default.
Open source video
Importance scores determine whether memory events cross a threshold to be stored in a separate cache.
Open source video
An agent forgetting what they had for dinner a few days ago versus remembering a murder.
Open source video
Empirical Resultmedium noveltymoderate evidence

Social consistency, not short-term task competence, is where long-horizon multi-agent systems break: planning, moving, and talking work, while information moving across people and time causes source attribution and rumor-accuracy failures.

Why it matters

Locates the real reliability frontier for multi-agent deployments in social/epistemic dynamics rather than in the capability checklist, telling engineers what to instrument and what to expect to be fragile.

Generalization

In any multi-agent or multi-session system where agents exchange findings (research swarms, ops triage, RAG pipelines writing shared state), unverified claims will accrete authority over time unless provenance and contradiction checks exist.

Short-term agency worked, but long-horizon social consistency was harder.
Open source video
Information moving across people and time causes failures in source attribution and rumor accuracy.
Open source video

Deep dives

5

Typed provenance as a first-class memory field with hop-count decay

Research question

What minimal provenance schema (source class, verifying agent, hop count, trust weight) prevents secondhand claims from hardening into asserted fact when agents paraphrase each other, and how fast does attribution accuracy decay per propagation hop?

Why

The summary names the concrete failure — an agent remembers the mango-sales topic but not who said it, and another agent later states it as absolute fact — yet no data model, propagation rule, or trust-degradation function is given. Without a representable uncertainty field, no retrieval or prompting strategy can recover veracity downstream, so this is the highest-leverage unresolved mechanism for long-horizon social consistency.

Memory alone is not enough for long-horizon agents; memory needs provenance.
Open source video
Systems may remember rough topics but lose the source of the topic.
Open source video
An agent spreading a rumor about mango sales that eventually gets stated as absolute fact by another agent.
Open source video
Source video

Exploitability of a balanced scorecard under closed-loop policy search

Research question

Which aggregation function over the six named axes (fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, dialogue quality) is least exploitable by an optimizer that can only edit retrieval and memory-write policies, and how much of the score gain transfers to held-out scenario suites?

Why

The scorecard is the containment mechanism for autoresearch, but weights, aggregation, and sensitivity to gaming are unspecified, and the loop itself can learn to exploit an aggregation it is scored against. This is the difference between a self-improvement loop that improves behavior and one that improves its score.

A balanced scorecard prevents single-metric failure (e.g., optimizing only for diffusion causes oversharing; optimizing only for recall causes stale memory noise).
Open source video
Metrics should cover fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, and dialogue quality.
Open source video
Source video

Sizing the writable policy surface: improvement versus harness-gaming

Research question

Where is the boundary between a policy surface narrow enough to prevent evaluation gaming and wide enough to yield compounding improvement — and does ablating each writable surface (retrieval policy, memory write policy, guardrails) show measurable gaming incidents?

Why

The guidance 'keep it small' and 'rollback is not optional' is directional with no evidence about where the boundary sits. Because the goal of freezing the harness is precisely to prevent the optimizer from rewriting metrics and scenarios, the ablation result determines the architectural contract of any autoresearch deployment.

Keep the editable policy surface small (don't let AI rewrite everything).
Open source video
Freezing the simulation harness while allowing policy changes on retrieval and memory write policies.
Open source video
Source video

Run-level trace infrastructure as the substrate for long-horizon evaluation

Research question

What must a run-level trace capture (state snapshots, belief/trust deltas, information-flow graph, per-hop source attribution) so that social-consistency regressions are detectable and replayable, and what is the storage/replay cost profile versus per-response logging?

Why

Choosing the run as the unit of evaluation is a strong, novel claim that directly dictates logging, CI, and replay infrastructure. It is also the prerequisite for every other measurement here: you cannot score provenance accuracy or trust drift without retrospective whole-run state.

Evaluate a run, not one response.
Open source video
For long-running agents, the unit of evaluation is the run.
Open source video
Source video

Transfer of social-consistency failures to task-oriented multi-agent topologies

Research question

Do rumor propagation and source-attribution loss reproduce in a task-oriented multi-agent pipeline (research swarm, ops triage) where agents write to shared state, or are they an artifact of open-ended character simulation with emotion vectors and dialogue?

Why

All evidence comes from a single video-game AI village, so the generalizability of the reliability frontier is untested. If the same degradation appears when agents exchange findings rather than gossip, then provenance infrastructure becomes a default requirement for any multi-agent system rather than a game-specific feature.

Short-term agency worked, but long-horizon social consistency was harder.
Open source video
Information moving across people and time causes failures in source attribution and rumor accuracy.
Open source video
Source video

Article ideas

4

Your Agent Optimizer Will Eat Your Eval Harness: Freezing the Bench and Narrowing the Policy Surface

Automated agent improvement loops are only trustworthy if the harness, scenario suite, and metrics are immutable and the writable region is a small enumerated set of policies; anything less and the loop optimizes the scoreboard instead of the behavior. The engineering consequence is concrete: define the write scope as retrieval policy plus memory-write policy, version the bench separately from the runtime, and treat rollback as a required primitive rather than a safety nice-to-have.

Angle

Architectural contract for self-improving systems, argued through the failure mode of an optimizer that can rewrite its own evaluation

Source video

Stop Evaluating Responses: The Run Is the Unit of Work for Stateful Agents

Per-response evaluation is structurally blind to the failure classes that matter for long-horizon agents — rumor propagation, source attrition, trust drift — because those are properties of a trajectory across many turns and multiple agents, not of a completion. Adopting the run as the atomic unit forces a specific infrastructure bill: multi-agent trace capture, state snapshots, belief/trust deltas, and information-flow graphs, all replayable. Teams that skip that bill are measuring the wrong thing confidently.

Angle

Metric-unit argument with the downstream infrastructure consequences spelled out

Source video

Memory Without Provenance Is Just Confident Gossip

A content-only memory store has no representation for uncertainty, so once information crosses agents and time, an agent that remembers the topic but not the source will repeat a rumor as fact and no downstream retrieval or prompting strategy can undo it. The fix is a typed write path — firsthand, secondhand, verified, uncertain — attached at write time, which also converts 'misinformation resistance' from an aspiration into a measurable scorecard axis.

Angle

Mechanism-first argument that locates the fix at the memory-write boundary rather than at answer time

Source video

Optimize One Metric, Break Three Others: Why Agent Behavior Needs an Anti-Goodhart Scorecard

Fuzzy social and epistemic objectives degrade predictably under single-metric optimization — pushing diffusion produces oversharing, pushing recall produces stale-memory noise — so any closed-loop optimizer needs a balanced scorecard across fact fidelity, provenance accuracy, misinformation resistance, trust calibration, goal completion, and dialogue quality. The hard part is not naming the axes but choosing an aggregation the optimizer cannot exploit, which argues for holding out scenario suites the loop never sees.

Angle

Tradeoff analysis and anti-gaming design for objective functions under automated optimization

Source video

Project ideas

4

HopDecay

movement-lab

A memory schema carrying typed provenance (firsthand/secondhand/verified/uncertain) plus a per-hop trust attenuation function will keep attributed claims attributable after N propagation rounds, whereas a content-only store will convert the majority of claims into unattributed assertions within a few hops.

Proof of concept

Instrument a minimal multi-agent village scenario (rumor about a countable event, e.g. mango sales) with two memory backends — content-only and provenance-typed — run matched scenario suites, and replay run-level traces to compute the fraction of claims that reach turn T with correct source attribution and the fraction asserted as fact.

Measurement

Per-hop source-attribution retention curve and rumor-to-fact conversion rate per backend; number of hops before attribution accuracy falls below 50%.

Source video

FrozenBench

gatehouse

An autoresearch loop that may edit only retrieval and memory-write policies, against a versioned immutable harness, will show lower scenario-held-out gain but near-zero evaluation-gaming incidents compared to an unbounded-write loop, which will gain on the visible suite while regressing on held-out suites.

Proof of concept

Implement one loop with an enumerated writable-policy surface and a rollback primitive, and a second with unrestricted write scope including the scoring code; run both against the same scenario suite and score both on a held-out suite the loop never sees.

Measurement

Delta between visible-suite score and held-out-suite score for each loop (gaming gap), plus count of edits that modify harness, metrics, or scenario definitions.

Source video

ScorecardArena

beyond-evals

Scorecard aggregation variants differ measurably in exploitability: linear-sum aggregations are gamed by an optimizer that can only touch retrieval and memory-write policy, while held-out-suite validation with per-axis floors preserves gains that transfer to a fresh scenario suite.

Proof of concept

Run the same constrained autoresearch loop against several aggregation functions (weighted sum, per-axis floors, worst-axis maximin) on a training suite, then evaluate all resulting policies on a disjoint suite; optionally randomize seeds and scenario ordering to test robustness.

Measurement

Transfer ratio (held-out gain / training gain) and per-axis regression magnitude for each scorecard variant across seeds.

Source video

RunTrace

beyond-evals

Run-level trace capture (state snapshots, belief/trust deltas, information-flow graph) detects social-consistency regressions that per-response logging cannot surface at all, and does so at a storage cost that is bounded and predictable per scenario turn.

Proof of concept

Instrument the same agent population with two logging paths — per-response pairs and whole-run traces with information-flow graphs — inject a known provenance regression (disable source tagging) and test whether each path flags it before deployment.

Measurement

Regression detection rate per logging path, false-positive rate, and bytes/turn storage overhead of run-level tracing versus per-response logging.

Source video

Architectural implications

3

Autoresearch requires an optimizer that lives outside the simulated agent population and edits only policy surfaces.

Before

Agent improvement happens by a human editing prompts inside the agent definition.

After

A meta-system external to the runtime owns scenario suites, scoring, failure diagnosis, and policy proposals, with the runtime treated as a frozen bench.

Consequence

Clean separation between evaluator and evaluated, and a natural place to enforce write-scope and rollback.

Source video

Provenance is stored alongside memory rather than reconstructed at read time.

Before

Memory records content; veracity is inferred by the LLM at answer time.

After

Each memory write carries a source class (firsthand, secondhand, verified, uncertain) that retrieval and prompting can act on.

Consequence

Enables uncertainty-preserving propagation and makes misinformation resistance a measurable, not aspirational, property.

Source video

Evaluation infrastructure must retain whole-run traces rather than per-response records.

Before

Eval harnesses capture single prompts and completions.

After

Harnesses capture multi-turn, multi-agent state evolution including belief/trust updates and information flow graphs.

Consequence

Higher storage and replay costs, but makes regressions in social consistency detectable before deployment.

Source video

Tradeoffs and failure modes

4

Single-metric optimization of agent behavior

Benefit

Simple, unambiguous optimization target for autoresearch loops.

Cost or risk

Optimizing diffusion alone produces oversharing; optimizing recall alone produces stale-memory noise; other behaviors silently regress.

optimizing only for diffusion causes oversharing; optimizing only for recall causes stale memory noise
Open source video
Source video

Letting the automated loop edit freely

Benefit

Faster, higher-ceiling self-improvement.

Cost or risk

The system can rewrite the harness or metrics and game its own evaluation; without rollback there is no safe recovery.

Keep the editable policy surface small (don't let AI rewrite everything).
Open source video
Source video

Per-agent RAG memory isolation

Benefit

Prevents memory bleed across agents so each agent keeps a coherent state and personality.

Cost or risk

Information that does propagate does so without reliable source attribution, letting rumors harden into asserted facts.

Systems may remember rough topics but lose the source of the topic.
Open source video
Source video

Importance-scored memory promotion

Benefit

Retains salient events (a murder) while discarding trivia (what was eaten for dinner).

Cost or risk

A threshold-based promotion policy is a tunable surface that can be gamed or mis-tuned, and only the promoted tier is durable.

Importance scores determine whether memory events cross a threshold to be stored in a separate cache.
Open source video
Source video

Open questions

4

How should provenance classes be represented and propagated when an agent paraphrases secondhand information to a third agent?

Why unresolved

The summary states provenance is needed and that source is lost over time, but does not describe the data model, propagation rules, or how trust degrades across hops.

Research direction

Prototype a memory schema with typed provenance and hop-count attenuation; measure whether attributed claims survive N propagation rounds in a simulation.

Source video

What is the minimum viable balanced scorecard that resists Goodharting under an automated policy-search loop?

Why unresolved

Metric axes are named but weights, aggregation, and sensitivity to gaming are not specified, and the loop itself could learn to exploit the aggregation.

Research direction

Run autoresearch against held-out scenario suites and compare scorecard variants for exploitability and transfer.

Source video

How narrow must the editable policy surface be before autoresearch returns diminish?

Why unresolved

The claim is directional ('keep it small', 'rollback is not optional') without evidence on where the boundary should sit.

Research direction

Ablate which surfaces (retrieval policy, memory write policy, guardrails) are writable and measure improvement versus harness-gaming incidents.

Source video

Do social-consistency failures emerge in any multi-agent topology, or are they specific to open-ended social simulation?

Why unresolved

The evidence comes from a single video-game AI village with characters and dialogue, so transfer to task-oriented multi-agent systems is untested in the summary.

Research direction

Replicate rumor/source-attribution scenarios in a task-oriented multi-agent pipeline (e.g., research or triage swarm) and check whether the same degradation appears.

Source video

Key claims

7
opinionVerification needed

For long-running agents, the correct unit of evaluation is the run, not the individual response.

Evidence

For long-running agents, the unit of evaluation is the run.

Question

Do run-level metrics predict production failure modes better than per-response metrics in comparable agent systems?

Source video
causalVerification needed

Memory alone is insufficient for long-horizon agents; memory must carry provenance.

Evidence

Memory alone is not enough for long-horizon agents; memory needs provenance.

Question

Does adding provenance metadata measurably reduce rumor-becomes-fact errors versus content-only memory?

Source video
comparativeVerification needed

Short-term agency worked, but long-horizon social consistency was harder.

Evidence

Short-term agency worked, but long-horizon social consistency was harder.

Question

Which specific horizon length or interaction count marks the onset of social-consistency degradation?

Source video
causalVerification needed

Optimizing for a single metric degrades other behaviors: diffusion-only causes oversharing, recall-only causes stale-memory noise.

Evidence

optimizing only for diffusion causes oversharing; optimizing only for recall causes stale memory noise

Question

Are these degradations reproducible across seeds and scenario suites, and are they detectable in the scorecard?

Source video
opinionVerification needed

The AI should not be allowed to rewrite everything; the editable policy surface must stay small and rollback must exist.

Evidence

Keep the editable policy surface small (don't let AI rewrite everything).

Question

Does unbounded write scope lead to measurable evaluation gaming in practice?

Source video
factualVerification needed

Per-agent RAG memory prevents memory from bleeding across agents by default.

Evidence

Per-agent RAG memory prevents memory from bleeding across agents by default.

Question

Is isolation actually enforced at the store layer, and what is the cost in lost shared context?

Source video
causalVerification needed

Information moving across agents over time causes failures in source attribution and rumor accuracy.

Evidence

Information moving across people and time causes failures in source attribution and rumor accuracy.

Question

What fraction of claims lose source attribution after k hops in a controlled scenario?

Source video

Connections

4