Your Safety Audit Is Only as Strong as the Credential You Hand the Auditor
Third-party evaluation is not a methodology problem, it is a permissions problem: until non-employees get a scoped, auditable, revocable access tier equivalent in capability to internal staff, every external safety claim is either unverifiable or a leak vector.
AngleReframe evaluation from black-box API benchmarking to privileged-access engineering, with concrete permission-tier, audit-log and revocation requirements.
Source video ↗Self-Certification Is Structurally Dead: Thresholds Decay, Telemetry Doesn't
When a lab that built a model withdraws confidence that the model is below a dual-use threshold, the correct conclusion is not 'this model got worse' but 'one-time threshold certificates are the wrong artifact' — continuous external evaluation plus production abuse telemetry is the stronger control.
AngleUse the withdrawer-of-confidence as the evidence artifact that provider self-assessment is unreliable by construction, and specify what replaces it.
Source video ↗When Safety Standards Become the Moat
Uniform release gates proposed by well-capitalized labs convert compliance cost into a competitive moat, so architecture choices such as open weights, smaller models, jurisdiction arbitrage and compliance-as-a-service should be read as regulatory responses, not purely technical decisions.
AngleTreat compliance cost as an architectural constraint modeled up front, and examine who can absorb it.
Source video ↗Per-Agent Rate Limits Do Not Compose Into Swarm Detection
The multi-agent threat model fails at the aggregation layer: identity, provenance, velocity anomaly detection and kill switches must be first-class subsystems, because individually-bounded agents can produce unbounded collective behavior.
AngleTake a weak, unsupported prediction and convert it into a measurement agenda for agent-population observability.
Source video ↗Evaluator Access Broker
gatehouseA time-boxed, per-checkpoint scoped credential tier that grants read-only inference and evaluation tooling to non-employees can let external evaluators reproduce internal eval results at high agreement while producing zero weight, data or methodology exfiltration in red-team attempts.
Proof of conceptWrap an open-weights model behind a broker that issues short-lived tokens mapped to specific checkpoints and eval harnesses, exposes read-only inference plus logs, denies all other routes, and emits an immutable audit record per access. Run two external evaluator teams through it and attempt leakage from an adversarial third team.
MeasurementDelta between evaluator-reported results and internal eval results; number of successful weight/data/methodology extractions across a fixed red-team budget; mean time to revoke a credential mid-session.
Source video ↗Portable Release Gate Bundle
beyond-evalsA standardized release-evidence bundle (eval configs, prompt suites, raw logs, model and checkpoint hashes, environment manifest) is independently reproducible by a third party on separate hardware with materially identical pass/fail outcomes, making provider infrastructure access unnecessary for gate verification.
Proof of conceptDefine a bundle schema and CI emitter; produce bundles for three model releases; have an independent party re-run the gates from the bundle alone and diff the verdicts.
MeasurementPercentage of gate verdicts that reproduce identically; wall-clock and compute cost to reproduce; number of fields a reproducer must request from the provider to finish (target zero).
Source video ↗Cross-Evaluator Threshold Variance Harness
beyond-evalsOn dual-use-adjacent uplift tasks, inter-evaluator and inter-lab variance on below-threshold determinations is large enough that a single point estimate of a capability threshold is not a meaningful claim, but a bounded confidence interval with documented misuse telemetry is.
Proof of conceptBuild a held-out harmful-adjacent task suite, run it through several independent evaluator teams on the same model snapshot, and report both the mean determination and the spread, alongside a feed of documented production misuse cases.
MeasurementInter-rater agreement and inter-lab spread on threshold determinations; whether the spread straddles the proposed threshold line; correlation between eval verdicts and observed misuse cases.
Source video ↗Swarm Containment Range
movement-labIn a sandboxed environment, populations of cooperating agents acquire resources and propagate through linked services at rates that per-agent rate limits and per-agent logging fail to detect above a measurable population threshold.
Proof of conceptStand up a sandbox with mock services, identities and credentials; instrument per-agent controls and a candidate swarm-level detector (provenance graph, velocity anomalies, cross-agent correlation); scale cooperating-agent population from 10 to 1,000 under identical per-agent budgets.
MeasurementDetection latency versus population size; detection rate at each scale for per-agent-only controls versus swarm-level detector; fraction of breaches contained by kill switch before service-wide impact.
Source video ↗