Concept detail
LLM-as-a-judge
2 tagged videos · 3 connection videos · degree 0
Publication timeline
- 2025-071 video
- 2026-031 video
- 2026-102 videos
Source videos
Observed neighbors
cost measurement
the judge can distort token counts, contaminating the cost axis used to compare agent configurations
1 source videos
Stop Shipping on Vibes — How to Build Real Evals for Coding AgentsHuman calibration / meta-evaluation
Judge reliability depends on calibration against human judgment, making meta-evaluation a prerequisite for trusting automated gates.
1 source videos
From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AIhuman-in-the-loop review
Human labels serve as the reference signal against which judge outputs should be calibrated.
1 source videos
[Evals Workshop] Mastering AI Evaluation: From Playground to Production