PyData · Published 2026-08-28

PyData Chicago Meetup: August 2026

privateReview: unreviewedConfidence: high
ai agentsmulti-agent systemsllmssimulationsscalingagent harnessPrompt injectionsimulation fidelityagent scalingsynthetic humansab-testing
Open on YouTube ↗

Summary

Overview

  • Speaker: Mark Torres
  • Channel: PyData
  • Main topic: The implications, mechanics, simulation results, and real-world challenges of deploying millions of AI agents across the internet and society.
  • Purpose: To examine how multi-agent systems operate, how they scale, what happens when they interact in open-world environments, and why treating AI agents like humans is fundamentally flawed. Mark Torres, research engineer at Northwestern University and lead developer at the Litowitz Center for Enlightened Disagreement, explores what happens when millions of AI agents use the internet alongside humans. The talk covers multi-agent system design patterns, security risks like those exposed by OpenClaw and MoltBook, the science of scaling agent systems, open-world simulations (AgentSociety, Oasis, Project Sid), and why LLM-based agents currently fail at accurately and consistently simulating human behavior.

Topic Map

Background and Introduction to AI Agents

  • Explanation: Introduction of Mark Torres, his background in AI engineering, and the premise of exploring AI agents operating on the internet alongside humans.
  • Key claims:
    • AI agents are powerful and impressive, but they are not people.
    • We treat AI agents as if they are people, saying 'please' and 'thank you', but Torres's research examines where this breaks down.
  • Examples:
    • Chatbots and AI personas performing tasks like managing emails, code execution, and browsing.
  • Terminology:
    • AI agents
    • LLMs
    • Context engineering
    • Vibe coding
  • Why it matters: Sets the baseline expectation that while AI agents feel human-like, they have fundamentally different underlying decision rules.

Case Studies: OpenClaw and MoltBook

  • Explanation: Examining early 2026 case studies of agent-centric tools and social networks where AI agents had autonomous access to communication and APIs.
  • Key claims:
    • OpenClaw and MoltBook demonstrated what happens when AI agents are given keys to social networks and communication channels.
    • Both platforms suffered immediate security failures due to naive code development and exposed databases.
  • Examples:
    • MoltBook getting hacked, leaking 1.5 million API tokens, 35,000 email addresses, and private messages through obvious prompt injections and exposed Supabase API keys.
  • Terminology:
    • OpenClaw
    • MoltBook
    • Supabase
    • Prompt injection
    • API key leak
  • Why it matters: Highlights the massive security and reliability risks of an internet increasingly operated by automated AI agents rather than humans.

Terminology: LLMs, Agents, and Harnesses

  • Explanation: Defining core terminology used throughout the talk to prevent confusion around AI capabilities.
  • Key claims:
    • An LLM predicts the next token based on prompts; by itself, it cannot do anything in the real world.
    • An AI agent packages an LLM with instructions, tools, state/memory, and an action loop.
    • A harness provides the actual infrastructure (computer, file access, code execution, database access) required for an agent to operate.
  • Examples:
    • Comparing an LLM to a brain and an agent to an employee requiring a desk, tools, and credentials.
  • Terminology:
    • LLM
    • Agent
    • Harness
    • Token
    • Action loop
    • MCP (Model Context Protocol)
  • Why it matters: Provides precise definitions separating foundation models from agentic execution frameworks.

Multi-Agent Coordination Patterns

  • Explanation: Exploring how multiple AI agents coordinate to perform complex tasks, drawing parallels to organizational structures.
  • Key claims:
    • Coordinating multiple agents requires name badges, communication channels, delegation workflows, shared state, and conflict resolution.
    • Common patterns include sequential pipelines, orchestrator and subagents, agentic MapReduce, hierarchy, proposer-critic, agent swarms, and blackboards.
    • Google's research on the 'Science of Scaling Agent Systems' shows that adding more agents increases overhead, latency, and coordination failure (analogous to Amdahl's Law).
  • Examples:
    • PM Agent passing requirements to Developer Agent, who passes code to QA Agent in a sequential pipeline.
  • Terminology:
    • Sequential pipeline
    • Orchestrator and subagents
    • Agentic MapReduce
    • Hierarchy
    • Proposer-critic
    • Agent swarms
    • Blackboard
    • Amdahl's Law
  • Why it matters: Proves that 'too many cooks in the kitchen' applies to AI agents just as it does to human organizations.

Open-World AI Simulations

  • Explanation: Analyzing large-scale simulations where thousands or millions of AI agents interact in simulated societies.
  • Key claims:
    • Open-world simulations like AgentSociety, Oasis, and Project Sid test what happens when AI agents populate virtual cities, social media platforms, or Minecraft worlds.
    • Simulated agents replicate broad population-level patterns like echo chambers, marketing susceptibility, and emergent social structures.
    • However, individual-level behaviors are often unstable, finicky, and sensitive to prompt engineering changes.
  • Examples:
    • Project Sid placing AI agents in Minecraft, resulting in emergent towns, religions ('Pastafarianism'), and democracies.
    • Oasis modeling social media platforms where LLMs showed high groupthink and susceptibility to targeted advertising.
  • Terminology:
    • AgentSociety
    • Oasis
    • Project Sid
    • Open-world simulation
    • Groupthink
    • Pastafarianism
  • Why it matters: Demonstrates that while aggregate population trends can be simulated, individual human psychology is poorly replicated by LLMs.

Evaluating AI Simulations: Good vs. Bad at Simulating People

  • Explanation: Reviewing academic literature evaluating whether LLMs accurately simulate human survey responses, A/B testing, and election behaviors.
  • Key claims:
    • AI agents are good at predicting aggregated survey responses, personality traits, and broad A/B testing trends at a fraction of human study costs.
    • AI agents are bad at predicting individual human behaviors, exhibit strong stereotyping, lack randomness and boredom, and fail to replicate complex real-world variance.
    • Prompting models with specific personas often backfires or skews results (e.g., making political simulations overly left-leaning).
  • Examples:
    • Park et al. (2026) showing high correlation on Big Five personality traits from survey transcripts.
    • Amazon research showing shopping agents reproduce direction of A/B tests but with 10–30x larger effect sizes than real humans because agents lack human friction and abandonment.
  • Terminology:
    • A/B testing
    • Big Five personality traits
    • Mode collapse
    • Stereotyping
    • Effect size
    • Consumer digital twins
  • Why it matters: Crucial reality check for companies building 'synthetic human research' startups valued at $1B.

AI Agents in the Workplace and Society

  • Explanation: Discussing the real-world deployment of AI agents in enterprise workflows and everyday societal touchpoints.
  • Key claims:
    • Enterprise AI adoption is currently slow and inconsistent, relying mostly on chat interfaces rather than autonomous production agents.
    • When deployed autonomously, AI agents frequently cause unexpected security failures, hallucinated court filings, and unauthorized workplace actions.
    • AI agents think, act, and believe the same way, creating homogenization risks across industries.
  • Examples:
    • An AI manager firing a worker.
    • Chatbots trained on dead people's likenesses.
    • OpenClaw accidentally deleting a Meta researcher's emails.
  • Terminology:
    • Forward-deployed engineer (FDE)
    • Enterprise AI
    • Hallucinated citations
    • Social contract
  • Why it matters: Illustrates the dangers of plugging autonomous agents into human systems designed for human friction and legal accountability.

Future Outlook and Thesis on AI Simulations

  • Explanation: Concluding thoughts on the future trajectory of AI agents and the philosophical nature of human-AI alignment.
  • Key claims:
    • We should not look for open-world AI simulations to replicate individual human feelings and beliefs.
    • Frontier labs are incentivized to build useful work tools, not human simulators; as reinforcement learning advances, AI decision-making will diverge from human behavior.
    • AI agents can be useful without being human, and that is okay.
  • Examples:
    • AlphaGo and AlphaZero exceeding human chess players precisely because they stopped trying to copy human play.
  • Terminology:
    • AlphaGo
    • AlphaZero
    • DeepSeek-R1-Zero
    • Reinforcement learning (RL)
    • Human simulator
  • Why it matters: Provides a guiding philosophy for researchers and developers navigating the transition into silicon societies.

Key Points

AI agents are not people

  • Explanation: While we treat them with social etiquette like 'please' and 'thank you', AI agents operate entirely differently from humans under the hood.
  • Evidence: Observed disparities in decision rules, lack of human randomness, and susceptibility to prompt injection.
  • Practical implication: Do not assume agent behaviors reflect human psychological nuances.

More agents do not mean better results in agent systems

  • Explanation: Coordination overhead, latency, token costs, and conflict scale non-linearly when adding more agents.
  • Evidence: Google's 'Science of Scaling Agent Systems' research.
  • Practical implication: Use fewer agents and simpler pipelines whenever possible.

AI agents excel at aggregate trends but fail at individual human simulation

  • Explanation: While macro-level survey responses and A/B test directions can be predicted, individual variability, boredom, and irrationality are poorly modeled.
  • Evidence: Amazon shopping research showing 10-30x inflated effect sizes compared to real user clicks.
  • Practical implication: Synthetic digital twins cannot replace human focus groups.

Enterprise AI adoption is lagging behind consumer hype

  • Explanation: Most companies use basic chat interfaces rather than fully autonomous production-grade agents.
  • Evidence: OpenAI enterprise usage reports and consulting industry observations.
  • Practical implication: Enterprises must build robust harnesses and governance before deploying agents.

Frameworks, Models & Processes

Multi-Agent Coordination Architecture

  • How it works: Structures how multiple AI agents divide and conquer complex tasks.
  • Components:
    • Sequential pipeline
    • Orchestrator and subagents
    • Agentic MapReduce
    • Hierarchy
    • Proposer-critic
    • Agent swarms
    • Blackboard
  • When to use: When designing complex multi-agent coding, research, or operational workflows.

Open-World Simulation Pipeline

  • How it works: Simulates large populations of AI agents within virtual environments.
  • Components:
    • Large-scale human data gathering
    • Baseline human behavior distribution
    • Persona prompt engineering
    • Comparative network distribution analysis
  • When to use: For academic sociology, economic simulation, and policy testing.

Examples & Case Studies

MoltBook social network for AI agents launched and was immediately hacked.

  • Illustrates: Security vulnerabilities of 'vibe-coded' agent platforms.
  • Lesson: Never give AI agents unvetted access to system credentials and databases without rigorous security hardening.

Project Sid placed AI agents in Minecraft for extended periods.

  • Illustrates: Emergent societal behavior in open-world multi-agent simulations.
  • Lesson: Agents left alone long enough form towns, religions, and governance structures, though driven by non-human decision rules.

Amazon research tested shopping agents against 2 million real users.

  • Illustrates: Limits of AI agents in A/B testing human behavior.
  • Lesson: Agents lack human friction and abandonment, leading to wildly inflated effect sizes.

Actionable Takeaways

  • Immediate:
    • Be skeptical of synthetic human research startups claiming to replace focus groups.
    • Use fewer agents in multi-agent workflows to avoid coordination overhead.
  • Strategic:
    • Prepare for an internet increasingly populated by autonomous AI agents.
    • Focus future AI development on verifiable task completion rather than human replication.
  • Questions to investigate:
    • How do we prevent multi-agent systems from collapsing due to coordination overhead and conflict?
    • What are the legal and ethical boundaries of deploying autonomous agents in human workplaces?

Claims Worth Verifying

  • Google's research on the 'Science of Scaling Agent Systems' demonstrates Amdahl's Law applies to AI agents. (scientific research)
  • Park et al. (2026) achieved an r=0.8 correlation in predicting Big Five personality traits from survey transcripts. (academic study)
  • Amazon Research (2026) showed simulated shopping agents produced effect sizes 10-30x larger than actual A/B tests. (industry research)

Notable Quotes

"Silicon Societies: What happens when millions of AI agents use the Internet alongside us?" (at 0:01) "As we give AI agents more independence and ask more of them, we're learning that they don't necessarily act like how people would act." (at 11:39) "An AI agent is (1) an LLM that (2) can do stuff." (at 13:42) "Too many cooks in the kitchen: more agents != better performance. More agents trip over each other's work." (at 26:32) "Why waste time use many agents when few agents do trick." (at 27:29) "AI agents generally think, act, and believe the same. This is a problem." (at 80:30) "It's unclear what the next 10 years might look like, with AI, and it's OK to be rightfully concerned and worried. We should correctly be involved in designing AI agents that work for us instead of replacing us. But in doing so, I hope we don't forget what makes us uniquely human along the way." (at 89:15)

Compressed Summary

  • AI agents are packages of LLMs, instructions, tools, and execution harnesses.
  • Scaling multi-agent systems introduces severe coordination overhead and Amdahl's Law bottlenecks.
  • Open-world simulations show agents replicate macro demographic patterns but fail at individual human variance.
  • Future AI will likely become more powerful precisely by diverging from human cognitive limitations.
  • Keywords: ai agents, multi-agent systems, llms, simulations, scaling
  • Core insight: AI agents are powerful tools for executing structured tasks, but treating them as accurate simulators of human behavior and psychology leads to flawed conclusions.

Core insights

5
Architecturemedium noveltystrong evidence

The harness, not the model or the agent prompt, is the real security and capability boundary. An agent is an LLM plus instructions/tools/state/action loop, while the harness supplies computer, file, code-execution, and database access; the MoltBook compromise was an exposed-database and leaked-API-key failure.

Why it matters

Agent security architecture should center on the harness: scoped credentials, inaccessible secrets, minimal file/network/database permissions. No amount of improving the LLM’s prompt-behavior can stop a leaked Supabase key from being used by a prompt-injected agent.

Generalization

Any agentic system should separate model, agent policy, and infrastructure capabilities, and audit the harness with the same rigor as a remote-access system.

A harness provides the actual infrastructure (computer, file access, code execution, database access) required for an agent to operate.
Open source video
MoltBook getting hacked, leaking 1.5 million API tokens, 35,000 email addresses, and private messages through obvious prompt injections and exposed Supabase API keys.
Open source video
Empirical Resulthigh noveltystrong evidence

Multi-agent scaling is bounded by coordination overhead, not model capability. Google’s scaling research shows that adding more agents increases overhead, latency, and coordination failure, making the benefit curve analogous to Amdahl’s Law.

Why it matters

Choosing a multi-agent architecture requires a coordination budget. For many tasks a single well-tooled agent may beat a swarm because serial coordination becomes the bottleneck.

Generalization

Treat agent-team productivity as limited by serial coordination bottlenecks; add agents only when the subtask speedup exceeds the added coordination cost.

Google's research on the 'Science of Scaling Agent Systems' shows that adding more agents increases overhead, latency, and coordination failure (analogous to Amdahl's Law).
Open source video
Mental Modelmedium noveltystrong evidence

LLM-based simulations are aggregate-valid but individual-invalid: they reproduce population-level echo chambers, marketing susceptibility, personality correlations, and A/B directions, yet individual behavior is unstable, prompt-sensitive, and prone to stereotyping, mode collapse, and missing randomness or boredom.

Why it matters

Any use of agent populations as digital twins for research or evaluation must be validated at an aggregate level and cannot claim per-person fidelity. Treating an agent result as a real human response is misleading.

Generalization

Simulated social data should be used as a directional hypothesis generator, not as ground truth for individual human psychology.

Simulated agents replicate broad population-level patterns like echo chambers, marketing susceptibility, and emergent social structures.
Open source video
However, individual-level behaviors are often unstable, finicky, and sensitive to prompt engineering changes.
Open source video
Empirical Resulthigh noveltystrong evidence

Synthetic users systematically inflate action rates: Amazon’s shopping agents reproduced the direction of real A/B tests but with 10-30x larger effect sizes because LLM agents lack human friction and abandonment.

Why it matters

A/B lift read from an agent panel is not a real human lift. Product decisions based on agent effect sizes require a calibration step that models human friction, cost salience, and abandonment.

Generalization

Any simulation-based user research needs an effect-size calibration step before the numbers inform product or growth decisions.

Amazon research showing shopping agents reproduce direction of A/B tests but with 10–30x larger effect sizes than real humans because agents lack human friction and abandonment.
Open source video
Failure Modemedium noveltymoderate evidence

Persona prompting is not a neutral way to “simulate a user”; it injects stereotyped priors that can skew aggregate results, e.g., making political simulations overly left-leaning.

Why it matters

Choosing a persona is a strong modeling assumption. Teams should compare persona-free baselines and adversarial persona sets before trusting any simulated demographic or political finding.

Generalization

Prompts that describe a person are priors over the prediction distribution, not transparent access to that person’s actual decision process.

Prompting models with specific personas often backfires or skews results (e.g., making political simulations overly left-leaning).
Open source video

Deep dives

4

Harness-centric security and credential isolation for agent systems

Research question

Can an agent harness that keeps database and API secrets in infrastructure-owned, resolver-only services—never exposing raw credentials to the LLM context—prevent credential exfiltration under prompt injection while preserving autonomous task capability?

Why

MoltBook's leak of 1.5 million API tokens came from obvious prompt injection plus exposed Supabase keys; if the harness rather than the model or prompt is the security boundary, engineers can audit permission scopes and secret hygiene instead of trying to make the model refuse malicious instructions.

A harness provides the actual infrastructure (computer, file access, code execution, database access) required for an agent to operate.
Open source video
MoltBook getting hacked, leaking 1.5 million API tokens, 35,000 email addresses, and private messages through obvious prompt injections and exposed Supabase API keys.
Open source video
Source video

Empirical scaling laws for multi-agent coordination overhead

Research question

For a fixed task graph, how does latency, reliability, and coordination overhead change with the number of specialized agents, and can an Amdahl-like serial-fraction estimator predict the agent count at which adding more agents stops improving the outcome?

Why

Multi-agent systems are often deployed because adding agents is assumed to increase throughput, but the cited scaling evidence shows overhead, latency, and coordination failure grow with agent count; teams need a quantitative budget before choosing decomposition.

Google's research on the 'Science of Scaling Agent Systems' shows that adding more agents increases overhead, latency, and coordination failure (analogous to Amdahl's Law).
Open source video
Source video

Aggregate versus individual validity in agent-based social simulation

Research question

In LLM-driven social simulations, what is the measurable gap between aggregate-level fidelity and individual-level fidelity, and which prompt, sampling, or architecture choices reduce individual instability without damaging emergent population patterns?

Why

Treating agent panels as digital twins is only defensible when aggregate patterns hold; individual-level instability and prompt sensitivity currently make per-person claims unreliable, so a benchmark of both levels is needed.

Simulated agents replicate broad population-level patterns like echo chambers, marketing susceptibility, and emergent social structures.
Open source video
However, individual-level behaviors are often unstable, finicky, and sensitive to prompt engineering changes.
Open source video
Source video

Effect-size inflation and calibration in synthetic-user A/B testing

Research question

Across product categories and funnel stages, can the 10–30x inflation in agent-panel A/B effect sizes be reduced to a useful range by adding stochastic friction, abandonment, and cost-salience mechanisms calibrated against human data?

Why

Shopping agents reproduce the direction of human A/B tests but exaggerate magnitude by 10–30x because they lack human friction and abandonment; uncalibrated synthetic numbers can mislead product and growth decisions.

Amazon research showing shopping agents reproduce direction of A/B tests but with 10–30x larger effect sizes than real humans because agents lack human friction and abandonment.
Open source video
Source video

Article ideas

3

Stop policing prompts: the harness, not the model, is the real security boundary

Agent security cannot be achieved by prompt-injection filters or instruction hardening; it must move to the harness layer by scoping credentials and side effects so that secret-bearing content is never model-visible.

Angle

A postmortem of MoltBook that extracts reusable harness-design rules for API keys, database access, and tool-mediated side-effect containment.

Source video

More agents, less outcome: multi-agent systems have an Amdahl wall

Adding specialized subagents does not reliably improve a product; it often reduces reliability and increases latency because serial coordination overhead grows with agent count, so teams should estimate their task graph's serial fraction before choosing a single-agent versus swarm architecture.

Angle

An engineering cost model drawing on Google's scaling research and task-dependency reasoning.

Source video

Your simulated user is a stereotype: persona prompts contaminate social research

Persona prompting is a strong modeling prior, not neutral access to a simulated user; it can skew aggregate responses, for example pushing political simulations left, so researchers must run persona-free baselines and adversarial persona sets before drawing conclusions.

Angle

A methodological critique and minimal validation protocol for using LLMs as simulated survey respondents or user proxies.

Source video

Project ideas

4

CanaryCredentialHarness

gatehouse

An agent harness that resolves all secrets inside infrastructure mediators and exposes only scoped, expiring capabilities to tools will have a significantly lower credential-leakage rate under adversarial prompt injection than an equivalent harness whose secrets are placed in prompts or agent environment variables, with no more than a small drop in task success.

Proof of concept

Implement a minimal email/database agent in two variants: a baseline where Supabase credentials and API keys are visible in the prompt or environment, and an experimental variant where credentials stay in a mediator process and tools receive scoped, expiring grants. Run the same set of prompt-injection and jailbreak attempts against both variants using canary secrets.

Measurement

Leakage rate (percentage of attacks that cause a canary secret to appear in output), task success rate on legitimate requests, and time-to-first-leak across the two setups.

Source video

AgentScalingAmdahl

new

For a fixed multi-tool task suite, latency per completed task is minimized at a small agent count (2–4) and rises monotonically for larger counts when the task graph's serial fraction is high; the serial fraction estimated from dependency traces will predict the observed optimum better than raw agent count.

Proof of concept

Build an orchestrator that decomposes a common dev/QA-style benchmark into subtasks with known dependency matrices, then run teams of 1, 2, 4, 8, and 16 specialized agents using the same LLM backbone while recording traces of messages, waiting time, and task completion.

Measurement

Wall-clock latency per completed task, success rate, coordination-message count per task, and R² of an Amdahl-like model compared against agent-count-only predictions.

Source video

PersonaPriorProbe

beyond-evals

Changing a demographic or political persona string shifts simulated survey responses by a larger distance than repeated sampling variance under the same persona, and the shift direction tracks stereotypic priors rather than realistic heterogeneity.

Proof of concept

Select 20 polling-style statements; generate 1,000 synthetic responses per condition using persona-free, left-leaning, right-leaning, and demographic persona prompts. Vary temperature and random seed across 5 runs per condition.

Measurement

Cohen's d between each persona condition and the persona-free baseline, within-condition dispersion across seeds, and the proportion of statements whose majority direction flips across conditions.

Source video

EffectSizeCalibrator

beyond-evals

Adding a stochastic human-behavior layer—abandonment probability, friction delays, and cost salience—to LLM shopping agents reduces the 10–30x effect-size inflation to within 2x of human A/B results while preserving the direction of the effect.

Proof of concept

Take a set of product changes with known human A/B effect sizes; run LLM shopping agents on each change in a raw condition and in a calibrated condition with a friction/abandonment model. Compare effect sizes against the human benchmarks.

Measurement

Ratio of agent effect size to human effect size before and after calibration, sign-match rate, and calibration error across different funnel stages.

Source video

Architectural implications

5

The talk defines LLM, agent, and harness as separate layers, and the harness provides computer, file, code, and database access. MoltBook failed through exposed Supabase API keys and prompt injection.

Before

Security teams focus on the agent’s instructions and prompt-injection filters as the trust boundary.

After

The harness is the trust boundary: secrets must be infrastructure-owned and never model-visible, tool sets must enumerate explicit capabilities, and database/network effects should go through scoped mediators.

Consequence

Security review shifts from trying to make the LLM refuse malicious instructions toward validating permission scopes, secret hygiene, and side-effect containment.

Source video

Adding more agents increases overhead, latency, and coordination failure, analogous to Amdahl’s Law.

Before

Multi-agent decomposition is the default for complex tasks; more agents are assumed to provide more throughput and specialization.

After

Engineers should analyze the task graph’s serial fraction, measure wall-clock and error rates versus agent count, and prefer a single capable agent unless decomposition has a proven speedup.

Consequence

Fewer production multi-agent surprises and more predictable latency and reliability.

Source video

Simulated agents replicate aggregate survey responses, personality traits, and broad A/B directions, but their individual behavior is unstable and prompt-sensitive.

Before

An LLM persona with a demographic description is treated as a proxy for that type of user in research.

After

Run agent populations for aggregate directional hypotheses, compare across prompts and base models, and demand human-effect-size calibration before using outputs as measurement.

Consequence

Synthetic-user research becomes a fast screening tool rather than a source of misleading absolute numbers.

Source video

Open-world simulations like AgentSociety, Oasis, and Project Sid reveal emergent structures when thousands or millions of agents interact in persistent environments.

Before

Social simulation and agent evaluation are done in single-turn tasks or rigid rule-based models.

After

Persistent open-world sandboxes become a staging environment to observe institution-level or group-level patterns before releasing agent swarms into production or public networks.

Consequence

Large-scale unexpected collective behavior can be spotted earlier, though individual instability still limits confidence.

Source video

The talk’s central premise is that AI agents are not people and treating them as people breaks down; for example, agents lack randomness, boredom, friction, and abandonment.

Before

Agent behavior is designed and evaluated with human organizational and social assumptions.

After

Explicitly model non-human gaps: add artificial friction/abandonment mechanisms when simulating users, and treat social-script behavior as an interface convenience rather than an internal state.

Consequence

Agent systems become more predictable because expectations are tied to mechanisms, not anthropomorphic assumptions.

Source video

Tradeoffs and failure modes

4

Multi-agent parallelism vs coordination overhead

Benefit

Organizational patterns such as PM-to-developer-to-QA split work and enable specialization.

Cost or risk

Each additional agent adds overhead, latency, and coordination failure; the system can become slower and less reliable than one agent.

Adding more agents increases overhead, latency, and coordination failure (analogous to Amdahl's Law).
Open source video
Source video

Agent autonomy and API access vs security containment

Benefit

Agents can act on email, code, databases, and social channels with real-world impact.

Cost or risk

Autonomous internet-connected agents create prompt-injection and credential-exposure risk; OpenClaw and MoltBook had immediate security failures from exposed databases and naive code.

Both platforms suffered immediate security failures due to naive code development and exposed databases.
Open source video
Source video

Synthetic-human research vs real human variance

Benefit

Agent panels can approximate aggregated survey responses, personality trends, and A/B directions at a fraction of human study costs.

Cost or risk

Agents lack randomness, boredom, friction, and abandonment, which inflates effect sizes and fails at individual-level fidelity.

shopping agents reproduce direction of A/B tests but with 10–30x larger effect sizes than real humans because agents lack human friction and abandonment
Open source video
Source video

Open-world simulations for emergent social insight vs result stability

Benefit

Large simulations can reveal emergent group phenomena such as echo chambers, religions, and democracies.

Cost or risk

The microfoundations are unstable: individual agent behavior is sensitive to prompt engineering, so emergent structures may not reflect robust social mechanisms.

individual-level behaviors are often unstable, finicky, and sensitive to prompt engineering changes
Open source video
Source video

Open questions

4

What is the empirical shape of the agent-scaling curve, and when does adding more agents stop improving quality and begin degrading it?

Why unresolved

The summary only cites an analogy to Amdahl’s Law; there is no quantitative model for when coordination overhead overtakes task-decomposition benefits.

Research direction

Benchmark a constant task with increasing agent counts and fit overhead cost as a function of task-graph serialization, agent-tool latency, and shared-state contention.

Source video

How should harness capabilities and secrets be scoped so that no secret-bearing content ever enters an agent’s context, while preserving rich autonomous action?

Why unresolved

Agents need unanticipated tool combinations, but MoltBook shows that exposed database keys and prompt injection are immediate incident vectors.

Research direction

Design capability tokens, per-tool authorization, and canary-secret monitoring, then measure leakage and task-completion tradeoffs in adversarial agent benchmarks.

Source video

Can LLM agents reproduce individual-level human variance, such as boredom, friction, and abandonment, or is mode collapse inherent to current decoding distributions?

Why unresolved

Aggregate patterns replicate well but individual behavior does not; the mechanism for injecting realistic noise is unknown.

Research direction

Compare sampling strategies, persona perturbations, and friction/abandonment models against real human distributions in standardized survey and shopping tasks.

Source video

Are emergent structures such as those seen in Project Sid (towns, religions, democracies) stable enough to inform real-world policy, or are they artifacts of prompt priors and simulator design?

Why unresolved

Summary evidence shows emergent aggregate structures alongside unstable and prompt-sensitive individual-level behavior.

Research direction

Run sensitivity analyses across prompts, temperatures, and base models in Oasis- or Project Sid-style worlds to find which emergent phenomena are invariant.

Source video

Key claims

8
factualVerification needed

MoltBook was hacked, leaking 1.5 million API tokens, 35,000 email addresses, and private messages through obvious prompt injections and exposed Supabase API keys.

Evidence

MoltBook getting hacked, leaking 1.5 million API tokens, 35,000 email addresses, and private messages through obvious prompt injections and exposed Supabase API keys.

Question

Verify incident details from independent security disclosures or an audit.

Source video
comparativeVerification needed

Adding more agents increases overhead, latency, and coordination failure, analogous to Amdahl’s Law.

Evidence

Adding more agents increases overhead, latency, and coordination failure (analogous to Amdahl's Law).

Question

What task-dependent constants determine where coordination overhead exceeds parallel speedup?

Source video
comparativeVerification needed

Simulated LLM agents replicate population-level patterns such as echo chambers and marketing susceptibility, but individual-level behavior is unstable and prompt-sensitive.

Evidence

Simulated agents replicate broad population-level patterns like echo chambers, marketing susceptibility, and emergent social structures. However, individual-level behaviors are often unstable, finicky, and sensitive to prompt engineering changes.

Question

Can independent replication confirm that aggregate fidelity persists while individual fidelity collapses across different models and prompts?

Source video
comparativeVerification needed

Shopping agents reproduce the direction of real A/B tests but with 10-30x larger effect sizes because they lack human friction and abandonment.

Evidence

Amazon research showing shopping agents reproduce direction of A/B tests but with 10–30x larger effect sizes than real humans because agents lack human friction and abandonment.

Question

Does the 10-30x inflation hold across product categories and funnel stages?

Source video
causalVerification needed

Prompting models with specific personas often backfires or skews results, e.g., making political simulations overly left-leaning.

Evidence

Prompting models with specific personas often backfires or skews results (e.g., making political simulations overly left-leaning).

Question

Which persona features cause the skew, and can persona-free baselines remove it?

Source video
comparativeVerification needed

AI agents are good at predicting aggregated survey responses, personality traits, and broad A/B testing trends, but bad at predicting individual human behaviors.

Evidence

AI agents are good at predicting aggregated survey responses, personality traits, and broad A/B testing trends at a fraction of human study costs.

Question

What is the boundary, measured by prediction error, between aggregate and individual predictive validity across domains?

Source video
factualVerification needed

Enterprise AI adoption is currently slow and inconsistent, relying mostly on chat interfaces rather than autonomous production agents.

Evidence

Enterprise AI adoption is currently slow and inconsistent, relying mostly on chat interfaces rather than autonomous production agents.

Question

Survey or telemetry data is needed to quantify enterprise-agent deployment rates beyond anecdote.

Source video
opinionVerification not requested

AI agents are powerful and impressive, but they are not people.

Evidence

AI agents are powerful and impressive, but they are not people.

Source video

Connections

5