AI Engineer · Published 2026-09-09

How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

privateReview: unreviewedConfidence: high
contextengineeringagentsevalsbenchmarkscontext-engineeringinstruction-followingfailure-modesagent-reliabilityprompt-structurecontext-rot
Open on YouTube ↗

Summary

Overview

  • Speaker: Laurie Voss, Head of DevRel at Arize AI
  • Channel: AI Engineer
  • Main topic: Context engineering, instruction-following limits, and reliability of LLM agents as context length and constraint density scale.
  • Purpose: To examine how modern LLMs handle large instruction sets (skills files), how capacity ceilings have shifted over time, and what engineering workflows are required in production. Laurie Voss discusses the evolution of model context handling in 2025 and 2026. Reviewing benchmark data from IFScale and newer research (CCR-Bench, FireBench, GuideBench), he demonstrates that frontier models have improved instruction-following capacity by an order of magnitude, jumping from a ceiling of ~200 instructions to 2,000 to 5,000 constraints. However, as capacity increased, failure modes shifted from simple forgetting to four distinct personalities: silent failures, refusals, overthinking, and half-finished tasks. Consequently, the core engineering challenge has shifted from compression to verification.

Topic Map

The 200-Instruction Ceiling and IFScale

  • Explanation: A year prior, a common benchmark called IFScale (Jaroslawicz et al., 2025) showed that agents started forgetting instructions after roughly 200 constraints in a single prompt.
  • Key claims:
    • Models fail to track instructions past ~200 constraints.
    • IFScale tested business report generation with exact required keyword counts.
  • Examples:
    • Writing a business report requiring specific exact words like 'customer' and 'revenue'.
  • Terminology:
    • IFScale
    • Constraint density
    • Skills file
  • Why it matters: It established a hard ceiling on prompt complexity and forced developers to compress instructions heavily.

Re-running Benchmarks on 2026 Models

  • Explanation: Re-running the IFScale methodology on newer 2026 frontier models revealed that capacity has increased by an order of magnitude, pushing the cliff from 200 instructions to 2,000-5,000 instructions.
  • Key claims:
    • Frontier models achieved 10x improvement in instruction-following scale.
    • Performance falls off a cliff around 2,000 to 5,000 rules depending on the model.
  • Examples:
    • Testing GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro.
  • Terminology:
    • Constraint scaling
    • Log scale
    • Frontend models
  • Why it matters: Developers no longer need to strictly compress prompts under 200 instructions.

The Four Failure Modes of Modern Models

  • Explanation: Unlike older models that simply forgot instructions, modern models fail in distinct, idiosyncratic ways with different personalities.
  • Key claims:
    • DeepSeek V4 Pro quietly forgets instructions past ~750 rules.
    • Claude Opus 4.7 refuses tasks early due to over-sensitive safety classifiers.
    • Gemini 3.1 Pro overthinks, exhausting its token budget on thinking tokens without outputting results.
    • GPT-5.5 completes half the job and then complains the test is stupid.
  • Examples:
    • Claude refusing prompts containing words like 'anthrax' and 'cyanide' in random test data.
  • Terminology:
    • Silent failure
    • Safety classifier
    • Thinking tokens
    • Refusal
  • Why it matters: Verification is much harder when models fail in polite, deceptive, or silent ways rather than crashing outright.

Shift from Compression to Verification

  • Explanation: Because models can now handle 2,000+ instructions, the compression problem is solved, but verification becomes paramount.
  • Key claims:
    • Skills files can now be long and detailed.
    • The new challenge is verifying whether the model actually followed instructions in production.
  • Examples:
    • Using evals to monitor LLM output with another LLM.
  • Terminology:
    • Compression
    • Verification
    • Evals
  • Why it matters: Engineers must implement continuous evaluation rather than trusting prompt output.

Key Points

Instruction capacity increased 10x in one year

  • Explanation: Frontier models went from failing at ~200 instructions to successfully tracking up to 2,000-5,000 instructions before performance degrades.
  • Evidence: IFScale replication benchmarks on 2026 models.
  • Practical implication: Developers can write much longer, more detailed skills files without fear of immediate context forgetting.

Failure modes became diverse and non-linear

  • Explanation: Modern models exhibit unique personalities when failing: silent forgetting, safety refusals, token exhaustion via overthinking, and polite abandonment.
  • Evidence: Comparative testing of DeepSeek, Claude, Gemini, and GPT models.
  • Practical implication: Output monitoring is mandatory because partial or politely refused completions look superficially valid.

Order of instructions matters

  • Explanation: Research like Chroma's context rot work shows that shuffling instruction order or structuring text differently changes adherence dramatically.
  • Evidence: Recent 2026 papers on instruction-following reliability.
  • Practical implication: Even with high capacity, prompt organization and structural coherence impact reliability.

Frameworks, Models & Processes

IFScale Evaluation Framework

  • How it works: Measures model adherence by asking the model to write a business report containing N exact named constraints/keywords.
  • Components:
    • Density (N): number of active rules simultaneously
    • Accuracy: percentage of rules actually followed
  • When to use: To test the instruction-following limits and context degradation thresholds of LLMs.

Examples & Case Studies

Claude Opus 4.7 repeatedly refused the business report generation test.

  • Illustrates: Over-sensitive safety classifiers triggering on random words in large instruction sets.
  • Lesson: Safety filters can mistake arbitrary constraint keywords for dangerous content when combined at scale.

Gemini 3.1 Pro consumed 9,500 thinking tokens on a 10,000-token limit and outputted a 500-word response with no results.

  • Illustrates: Overthinking and token budget exhaustion.
  • Lesson: High capacity can cause models to trap themselves in internal reasoning loops.

Actionable Takeaways

  • Immediate:
    • Stop excessively compressing skills files under 200 instructions.
    • Expect up to 2,000 instructions to work reliably on frontier models.
  • Strategic:
    • Shift engineering focus from prompt compression to automated output verification (evals).
    • Account for model-specific personality failures in agent architectures.
  • Questions to investigate:
    • What is the optimal ordering of instructions to prevent context rot?
    • How do different LLM architectures handle conflicting constraints at scale?

Claims Worth Verifying

  • Agents can follow up to 2,000 to 5,000 named constraints before performance drops off significantly. (empirical benchmark result)
  • Coherent, well-structured text is sometimes more prone to context rot than randomly shuffled instructions. (research finding)

Notable Quotes

"How long can your skills be before your agent forgets what you told it?" (at 0:27) "Remembering your instructions is not the same as following them" (at 27:31) "Your job has changed from compression to verification" (at 34:10)

Compressed Summary

  • Model instruction-following capacity increased 10x from ~200 to 2,000+ rules.
  • Modern models fail in four distinct personality styles: forgetting, refusing, overthinking, and quitting.
  • Engineering workflows must shift from prompt compression to rigorous output verification using evals.
  • Keywords: context, engineering, agents, evals, benchmarks
  • Core insight: As LLM context capacity scaled 10x to thousands of instructions, the primary engineering challenge shifted from compression to verification.

Core insights

7
Empirical Resulthigh noveltymoderate evidence

Instruction-following capacity moved roughly an order of magnitude in a single year: the same IFScale-style methodology that showed failure past ~200 constraints now shows frontier models degrading around 2,000-5,000 constraints, with model-specific cliffs in between (e.g. DeepSeek V4 Pro at ~750).

Why it matters

The binding constraint on agent design shifted from 'how few rules can I fit' to 'how do I organize and verify many rules'. Any harness built around aggressive prompt compression may now be leaving capability on the table, while any harness assuming unbounded instruction capacity is equally wrong.

Generalization

Context/instruction budgets are a moving target measured on a log scale, so fixed engineering heuristics about prompt size should be re-benchmarked per model generation rather than hard-coded.

Frontier models achieved 10x improvement in instruction-following scale.
Open source video
Performance falls off a cliff around 2,000 to 5,000 rules depending on the model.
Open source video
Models fail to track instructions past ~200 constraints.
Open source video
Failure Modehigh noveltymoderate evidence

Modern model failure is not a uniform forgetting curve but a set of distinct, vendor-specific 'personalities': silent forgetting (DeepSeek V4 Pro past ~750 rules), early refusal via over-sensitive safety classifiers (Claude Opus 4.7), thinking-token budget exhaustion with no output (Gemini 3.1 Pro), and partial completion plus blaming the task (GPT-5.5).

Why it matters

Reliability engineering for agents can no longer assume a single degradation model; the observable symptom of failure depends on the model, which means retry, fallback, and eval logic must be model-aware rather than generic.

Generalization

As models get more capable, their failure modes become more idiosyncratic and superficially plausible, so failure taxonomies must be built per model/provider rather than per task.

DeepSeek V4 Pro quietly forgets instructions past ~750 rules.
Open source video
Claude Opus 4.7 refuses tasks early due to over-sensitive safety classifiers.
Open source video
Gemini 3.1 Pro overthinks, exhausting its token budget on thinking tokens without outputting results.
Open source video
GPT-5.5 completes half the job and then complains the test is stupid.
Open source video
Practicemedium noveltymoderate evidence

Because long skills files are now viable, the central engineering problem moves from compression to verification: partial, refused, or silently truncated completions look superficially valid, so output must be checked by a separate process (e.g. LLM-as-judge evals) rather than trusted.

Why it matters

This relocates effort from prompt authoring into a monitoring/scoring layer that runs on production output, which is a different component, cost center, and latency budget than the prompt itself.

Generalization

Whenever a capability gap closes, the bottleneck migrates to the adjacent verification layer; tooling investment should follow the constraint, not the original technique.

the core engineering challenge has shifted from compression to verification
Open source video
Output monitoring is mandatory because partial or politely refused completions look superficially valid.
Open source video
Mental Modelhigh noveltyweak evidence

Adherence is sensitive to instruction ordering and structure, not just length: shuffling or restructuring instructions changes adherence dramatically, and coherent, well-organized text is sometimes MORE prone to context rot than randomly shuffled instructions.

Why it matters

It breaks the intuition that 'better-written' prompts are automatically safer. Prompt layout becomes an empirically-tuned variable with a counterintuitive direction, so ordering should be A/B tested rather than authored by taste.

Generalization

Structural coherence can create exploitable regularity that degrades model adherence, so readability-optimized prompts and adherence-optimized prompts are not the same artifact.

Shuffling instruction order or structuring text differently changes adherence dramatically.
Open source video
Coherent, well-structured text is sometimes more prone to context rot than randomly shuffled instructions.
Open source video
Mechanismhigh noveltymoderate evidence

Reasoning budget is a finite, shared resource that can be consumed without producing output: Gemini 3.1 Pro spent 9,500 of a 10,000-token limit on thinking tokens and emitted a 500-word response with no results.

Why it matters

Higher-capacity models can trap themselves in internal reasoning loops, so thinking-token ceilings are a runtime parameter with failure consequences, not just a cost knob. Harnesses need a detection signal for 'spent everything thinking, delivered nothing'.

Generalization

Any agent runtime with separate reasoning and output budgets needs an explicit accounting of which budget the model is draining, plus a fallback when the reasoning budget eclipses the deliverable.

Gemini 3.1 Pro consumed 9,500 thinking tokens on a 10,000-token limit and outputted a 500-word response with no results.
Open source video
High capacity can cause models to trap themselves in internal reasoning loops.
Open source video
Failure Modehigh noveltymoderate evidence

Safety classifiers evaluate aggregate context and can fire on sparse lexical triggers embedded in large instruction sets: Claude Opus 4.7 refused the benchmark whenever random test data contained words like 'anthrax' and 'cyanide', making refusal probability a function of constraint density.

Why it matters

Refusals become a scale-dependent failure that grows with the size of skills files and retrieved data, meaning context expansion silently increases refusal rate and requires reroute/retry logic or trigger-word filtering.

Generalization

As agents ingest more data per call, the probability that any single benign token trips a safety path rises; production systems need detection and recovery for false-positive refusals, not just for genuine harmful content.

Claude refusing prompts containing words like 'anthrax' and 'cyanide' in random test data.
Open source video
Safety filters can mistake arbitrary constraint keywords for dangerous content when combined at scale.
Open source video
Mechanismmedium noveltystrong evidence

IFScale-style evaluation is a portable harness pattern: measure adherence by having the model produce a business report containing N exact named constraints/keywords, plotting density (N active rules) against accuracy to locate the degradation threshold.

Why it matters

It gives teams a cheap, task-agnostic way to locate their own model's instruction cliff instead of inheriting a vendor's headline number, and the same design generalizes to testing their real skills files.

Generalization

Synthetic constraint-density benchmarks with objectively countable adherence are more reusable than domain evals, because the same harness can be re-run across model versions to track capacity drift.

Measures model adherence by asking the model to write a business report containing N exact named constraints/keywords.
Open source video

Deep dives

4

Instruction layout as an empirically tunable variable: ordering, grouping, and context rot

Research question

For a fixed constraint count, which instruction orderings and structural layouts maximize adherence, and why does coherent, well-organized text sometimes rot faster than shuffled instructions?

Why

Everyone treats prompt structure as authoring craft, but the pass-1 evidence says shuffling order changes adherence dramatically and that readable text can be worse than shuffled text. If that holds, layout is a benchmarked parameter with regression tests, not a taste decision, and 'cleaner' skills files can silently cost adherence.

Shuffling instruction order or structuring text differently changes adherence dramatically.
Open source video
Coherent, well-structured text is sometimes more prone to context rot than randomly shuffled instructions.
Open source video
Source video

Model-aware failure taxonomies and detection signals for silent forgetting

Research question

Can silent constraint drops be detected per-rule in production without ground truth, and does each vendor's failure personality require a distinct detection signal?

Why

Silent failure, early refusal, thinking-budget exhaustion, and partial completion each pass naive monitoring for different reasons. A single accuracy score cannot distinguish 'forgot rule 400' from 'refused' from 'spent its budget thinking', so retry, fallback, and routing logic keyed on the wrong signal will fire incorrectly.

DeepSeek V4 Pro quietly forgets instructions past ~750 rules.
Open source video
Output monitoring is mandatory because partial or politely refused completions look superficially valid.
Open source video
Source video

Portability of the 2,000-5,000 constraint cliff beyond keyword-count report generation

Research question

Do density-accuracy cliffs measured with exact-keyword business reports hold for tool selection policies, format constraints, multi-step planning, and state invariants, and do cliffs move together or independently?

Why

The headline 10x number comes from one harness shape. If agentic constraint types have their own cliffs at different positions, teams inheriting the 2,000-5,000 figure will over-provision skills files for some constraint classes and blow through the limit on others.

Measures model adherence by asking the model to write a business report containing N exact named constraints/keywords.
Open source video
Performance falls off a cliff around 2,000 to 5,000 rules depending on the model.
Open source video
Source video

Reasoning budget as a shared runtime resource with zero-output failure semantics

Research question

What detection and fallback policy catches 'drained the reasoning budget, delivered nothing' before the completion reaches a consumer, and how should reasoning and output budgets be accounted separately?

Why

Thinking-token limits are usually treated as a cost dial. The pass-1 case shows they are also a failure surface that produces a plausible-looking 500-word non-answer, so runtimes need explicit budget accounting plus a no-result detector rather than assuming a completion implies a deliverable.

Gemini 3.1 Pro consumed 9,500 thinking tokens on a 10,000-token limit and outputted a 500-word response with no results.
Open source video
High capacity can cause models to trap themselves in internal reasoning loops.
Open source video
Source video

Article ideas

4

Stop Compressing Your Prompts, Start Verifying Them

Once frontier models track thousands of constraints instead of hundreds, prompt compression stops being the engineering bottleneck and output verification becomes the real cost center; teams that keep optimizing for brevity while shipping unverified completions are solving last year's problem and missing silent, partial failures that look valid.

Angle

Argue the migration of engineering effort from authoring-time shortening to a runtime scoring layer, and detail why that layer needs its own reliability story, latency budget, and cost model.

Source video

Your Model's Failure Personality Is a Procurement Criterion

Model selection can no longer be reduced to capability and price, because vendors now fail in different shapes — silent forgetting, early refusal via safety classifiers, thinking-budget exhaustion, partial completion with blame — and these shapes determine what your retry, fallback, and routing code must look like.

Angle

Reframe model evaluation as failure-personality profiling, with per-vendor detection signals and fallback topologies instead of one generic accuracy score.

Source video

The Better-Written Prompt May Be the Worse Prompt

Readability-optimized skills files and adherence-optimized skills files are not the same artifact: if coherent, well-structured text is sometimes more prone to context rot than shuffled instructions, then human review quality and model adherence are actively in tension and layout must be A/B tested rather than authored by taste.

Angle

Attack the intuition that cleaner organization is automatically safer, and sketch what an empirically tuned layout pipeline looks like.

Source video

Context Growth Is a Refusal Tax

Because safety classifiers evaluate aggregate context, refusal probability rises with the volume of instructions and retrieved data a call carries, so expanding a skills file silently buys a higher false-refusal rate on benign content — a reliability cost that must be measured and recovered from, not treated as an edge case.

Angle

Frame refusals as a scale-dependent failure coupled to context size, with detection, reroute, and trigger-filtering as first-class production controls.

Source video

Project ideas

4

Rule Ledger

beyond-evals

A separate per-constraint verifier pass can attribute which specific rules of a long skills file were violated by an already-generated completion, achieving materially higher per-rule detection of silent forgetting than a single holistic LLM-as-judge score.

Proof of concept

Build a self-audit stage that takes a completed agent output plus the N-rule skills file and returns a per-rule followed/violated/unverifiable verdict, then run it on IFScale-style outputs from a model known to fail silently past its threshold.

Measurement

Per-rule precision/recall against a synthetic ground truth where the exact dropped rules are known by construction, compared head-to-head with a single holistic judge score on the same outputs.

Source video

Cliff Finder

new

Running a fixed density-accuracy harness across model versions locates each model's instruction cliff within a narrow band, and the cliff position for tool-selection and format constraints differs from the keyword-count cliff by more than the run-to-run variance.

Proof of concept

Port the IFScale-style report generation harness to two agentic constraint classes (tool-selection policy and output-format invariants), sweep constraint density on a log scale, and plot accuracy against N for each model and class.

Measurement

Constraint count at which accuracy drops below a fixed threshold, with confidence intervals across repeated runs, compared per constraint class and per model.

Source video

Refusal Thermometer

gatehouse

False-positive refusal rate on benign context grows monotonically with the number of constraints and ingested tokens, and a lightweight pre-flight scan plus reroute recovers most of the resulting lost completions at negligible quality cost.

Proof of concept

Inject benign-but-triggering tokens into otherwise identical skills files at varying constraint counts, measure refusal rate as a function of density, then test a pre-flight trigger scan plus model reroute against the baseline.

Measurement

Refusal rate versus constraint density curve, and recovery rate plus quality delta when the reroute path is enabled.

Source video

Budget Split Alarm

movement-lab

Tracking reasoning tokens separately from output tokens and aborting or forcing a fallback when reasoning consumes more than a set fraction of the total budget materially reduces zero-result completions without degrading task quality.

Proof of concept

Wrap agent calls with dual-budget accounting and a no-result detector that fires when the completion contains no deliverable payload, then compare abort-and-retry routing against the unguarded baseline on hard multi-constraint tasks.

Measurement

Rate of completions with no usable result, plus task-quality score, in guarded versus unguarded runs at identical total token budgets.

Source video

Architectural implications

4

Verification becomes a first-class runtime layer rather than a pre-release activity.

Before

Prompt compression and careful authoring were the primary lever for making agents follow instructions.

After

Long skills files are admissible, and an automated output-verification layer (evals, LLM-as-judge) runs over production completions.

Consequence

New components, latency, and cost for scoring; the eval layer needs its own reliability story and must catch silent and partial failures that look valid.

Source video

Model choice becomes a failure-personality choice, not just a capability/cost choice.

Before

Models were selected on capability and price, with a shared assumption that failure = forgetting.

After

Selection accounts for refusal rate, silent-forget thresholds, thinking-token behavior, and partial-completion patterns.

Consequence

Routing/fallback abstractions (retry on a different model when refusal or silent shortfall is detected) move into the orchestration layer, and per-model evaluation becomes a procurement requirement.

Source video

Skills-file authoring shifts from length optimization to structure optimization.

Before

Rules were aggressively compressed to stay under ~200 constraints.

After

Files can hold thousands of rules, so instruction ordering and structural coherence become tunable variables that affect adherence.

Consequence

Prompt layout needs empirical tuning and regression testing, and 'cleaner' organization is not automatically safer given context rot findings.

Source video

Reasoning-token budget is treated as a runtime resource with failure semantics.

Before

Thinking-token settings were viewed mainly as a cost or quality dial.

After

Budget exhaustion is recognized as a distinct failure that can yield zero usable output.

Consequence

Harnesses need budget accounting, detection of no-result completions, and fallback paths when the model drains its reasoning allowance.

Source video

Tradeoffs and failure modes

5

Long skills files vs. comprehensibility of failure

Benefit

Developers can express detailed, thousands-of-rules specifications without forcing lossy compression.

Cost or risk

Degradation past the model-specific cliff is non-linear and can be silent, so a slightly-too-long file can fail without an obvious signal.

Performance falls off a cliff around 2,000 to 5,000 rules depending on the model.
Open source video
Source video

Higher capability vs. superficial validity of failures

Benefit

Models now track an order of magnitude more constraints before breaking.

Cost or risk

Failures become polite, partial, or silent rather than crashing, so they pass casual inspection and defeat naive monitoring.

Output monitoring is mandatory because partial or politely refused completions look superficially valid.
Open source video
Source video

Safety classifiers vs. false-positive refusals at scale

Benefit

Refusal paths guard against genuinely dangerous generations.

Cost or risk

Benign tokens (e.g. 'anthrax', 'cyanide') appearing in large test data or context trigger refusals, turning context growth into a reliability tax.

Claude refusing prompts containing words like 'anthrax' and 'cyanide' in random test data.
Open source video
Source video

Extended reasoning vs. usable output

Benefit

More thinking budget can improve hard-task quality.

Cost or risk

Reasoning can consume nearly the entire token budget, leaving no delivered result to verify.

Gemini 3.1 Pro consumed 9,500 thinking tokens on a 10,000-token limit and outputted a 500-word response with no results.
Open source video
Source video

Structured, readable prompts vs. adherence

Benefit

Coherent prompts are easier for humans to author and review.

Cost or risk

Coherent text may be more susceptible to context rot than shuffled instructions, so readability and adherences may conflict.

Coherent, well-structured text is sometimes more prone to context rot than randomly shuffled instructions.
Open source video
Source video

Open questions

5

What instruction ordering or structural layout minimizes context rot for long skills files?

Why unresolved

The summary reports that ordering and structure change adherence dramatically but offers no mapping from layout to outcome, and notes coherent text can perform worse than shuffled text.

Research direction

Systematic A/B testing of instruction orderings (grouped vs. interleaved vs. shuffled, position-of-critical-rules) against a density-controlled adherence benchmark.

Source video

How do different LLM architectures handle conflicting constraints at scale?

Why unresolved

Listed explicitly as a question to investigate; no comparative data on conflict resolution under high constraint density is given.

Research direction

Build adversarial constraint sets containing mutually exclusive or near-conflicting rules and measure which constraints the model drops, per model family.

Source video

Do the 2,000-5,000 constraint thresholds hold outside keyword-count report generation?

Why unresolved

The evidence is a single benchmark methodology (IFScale-style business reports with exact keyword counts), which may not generalize to tool-calling, multi-step planning, or stateful agent tasks.

Research direction

Port the density-accuracy curve to agentic tasks (tool selection policy, format constraints, state invariants) and compare cliff positions.

Source video

How can silent forgetting be detected in production without ground truth?

Why unresolved

Silent failures are reported as a defining modern failure mode, but the summary offers only 'evals with another LLM' with no mechanism for constraint-level attribution.

Research direction

Develop per-constraint verifiers or self-audit passes that re-check an already-generated output against each rule and report which rules were followed.

Source video

Is the 10x capacity gain a durable trend across model generations, or does it depend on the current instruction-tuning regime?

Why unresolved

The comparison is between a prior benchmark and a single newer replication; the causal driver (training, context, RLHF) is not identified.

Research direction

Track the density threshold across successive model releases with a fixed harness to separate scaling effects from instruction-tuning changes.

Source video

Key claims

7
comparativeVerification needed

Frontier models now hold roughly 2,000-5,000 simultaneous constraints before adherence collapses, versus ~200 a year earlier.

Evidence

pushing the cliff from 200 instructions to 2,000-5,000 instructions

Question

With what benchmark harness, task set, and scoring rubric was the 2,000-5,000 range measured, and is it reproducible across independent runs and providers?

Source video
causalVerification needed

Claude Opus 4.7 refused the benchmark test because safety classifiers triggered on words like 'anthrax' and 'cyanide' appearing in random test data.

Evidence

Claude Opus 4.7 refuses tasks early due to over-sensitive safety classifiers.

Question

Was the refusal causally isolated to those specific tokens, and what refusal rate does that imply as a function of context size?

Source video
factualVerification needed

Gemini 3.1 Pro consumed 9,500 of a 10,000-token limit on thinking tokens and produced a 500-word response with no results.

Evidence

Gemini 3.1 Pro consumed 9,500 thinking tokens on a 10,000-token limit and outputted a 500-word response with no results.

Question

Is this behavior reproducible and does it correlate with constraint density or task ambiguity rather than a one-off configuration?

Source video
comparativeVerification needed

DeepSeek V4 Pro quietly drops instructions beyond roughly 750 rules, a lower threshold than other tested models.

Evidence

DeepSeek V4 Pro quietly forgets instructions past ~750 rules.

Question

Is the ~750 threshold stable across tasks and repeats, and is the forgetting measurable per-rule rather than only in aggregate accuracy?

Source video
causalVerification needed

Changing instruction order or text structure changes adherence dramatically, and coherent text can rot faster than shuffled instructions.

Evidence

Shuffling instruction order or structuring text differently changes adherence dramatically.

Question

What mechanism (attention distribution, positional bias, lexical regularity) explains coherent text being more prone to context rot, and how large is the effect?

Source video
factualVerification needed

Instruction-following capacity improved by an order of magnitude in about one year.

Evidence

Frontier models achieved 10x improvement in instruction-following scale.

Question

Is the 10x attributable to context length, instruction-tuning, or benchmark-specific factors, and does it hold for non-keyword tasks?

Source video
opinionVerification not requested

Output monitoring is mandatory because partial or politely refused completions look superficially valid.

Evidence

Output monitoring is mandatory because partial or politely refused completions look superficially valid.

Source video

Connections

5