[Evals Workshop] Mastering AI Evaluation: From Playground to Production

Overview

• Speaker: Doug Guthrie and Carlos Esteban
• Channel: AI Engineer
• Main topic: AI System Evaluation and Observability with Braintrust
• Purpose: To educate AI engineers on how to build robust testing, evaluation, and observability frameworks for LLM applications using Braintrust. This hands-on workshop led by solutions engineers from Braintrust covers the comprehensive lifecycle of evaluating AI and LLM systems. It introduces core concepts like prompt engineering, offline and online evaluations, datasets, scorers (code-based and LLM-as-a-judge), and human-in-the-loop workflows. Through practical activities, attendees learn how to use Braintrust's UI and TypeScript SDK to run evaluations, track performance regressions, set up online logging and scoring, and close the feedback loop for continuous improvement.

Topic Map

Introduction to Evals

• Explanation: Explains why evaluations are critical for production AI systems, citing high hallucination rates, performance degradation over time, and prompt drift.
• Key claims:
  • LLMs cannot guarantee consistent performance without observability and testing.
  • Writing evals is becoming a core skill for product managers and AI engineers.
  • Evals help answer questions around user experience, cost efficiency, brand consistency, and regression detection.
• Examples:
  • In 2023 testing, major LLMs had a 91.4% hallucination rate in certain domains.
  • Top LLM code-success rates dropped by 80% over 3 months due to provider updates.
• Terminology:
  • evals
  • hallucination rate
  • performance degradation
  • prompt drift
• Why it matters: Without evals, teams push changes to production blindly, risking broken workflows and poor user experiences.

Core Concepts of Braintrust

• Explanation: Outlines the three pillars of Braintrust: prompt engineering (IDEs and playgrounds), evals (measuring improvement/regression), and AI observability (production logging).
• Key claims:
  • Effective AI development requires cycling between prompt engineering, structured evals, and production observability.
  • Evals provide a performance score from 0-100% across test scenarios.
• Examples:
  • Using playgrounds for rapid prototyping before locking in prompt versions.
• Terminology:
  • prompt engineering
  • evals
  • AI observability
  • ground truth
• Why it matters: Establishes a mental model for how developers should build, test, and monitor AI applications.

Ingredients of an Eval

• Explanation: Breaks down an evaluation into three core components: Task, Dataset, and Scorer.
• Key claims:
  • A task is the code or prompt being evaluated, requiring an input and output.
  • A dataset is a collection of real-world or synthetic test cases.
  • A scorer is the logic (LLM-as-a-judge or code function) that grades output on a 0-1 scale.
• Examples:
  • Using code-based exact match functions alongside GPT-4 as an LLM judge.
• Terminology:
  • task
  • dataset
  • scorer
  • LLM-as-a-judge
  • code function
• Why it matters: Provides a clear modular framework for structuring any AI evaluation.

Offline vs Online Evals

• Explanation: Contrasts offline evals (done in development via playgrounds and SDKs to catch issues early) with online evals (monitoring real production traffic and user feedback).
• Key claims:
  • Offline evals proactively identify issues before deployment using predefined datasets.
  • Online evals provide real-time monitoring, tracing, and user feedback capture in production.
• Examples:
  • Running experiments in the Braintrust UI versus streaming production logs via proxy or SDK.
• Terminology:
  • offline evals
  • online evals
  • tracing
  • proxy SDK
• Why it matters: Development needs structured test suites, while production needs continuous monitoring and regression alerts.

Moving into Production and Human-in-the-Loop

• Explanation: Discusses why logging production traces, running online scoring rules, and incorporating human review are essential for closing the feedback loop.
• Key claims:
  • Production logging allows debugging issues in minutes rather than hours.
  • Human review establishes ground truth and aligns AI models with business compliance and user expectations.
• Examples:
  • Adding thumbs-up/thumbs-down feedback scores in web applications to filter low-performing traces.
• Terminology:
  • logging
  • online scoring
  • human-in-the-loop
  • HITL
  • ground truth
• Why it matters: Closes the gap between lab testing and real-world usage through continuous learning and refinement.

Key Points

Evals cut dev time and reduce costs

• Explanation: Automating test reviews replaces manual QA, allowing faster iteration and safer releases.
• Evidence: Customer examples like Zapier and Coda saw 5x to 20x increases in feature delivery and productivity.
• Practical implication: Teams can iterate on prompts and models with high confidence without manual bottlenecking.

Two mental models for evals: Offline and Online

• Explanation: Offline evals test changes during development; online evals monitor production traffic and capture real user feedback.
• Evidence: Braintrust supports both playground experiments and SDK/proxy-based production tracing.
• Practical implication: Engineers must test rigorously before shipping and continuously monitor after deployment.

Combine code-based and LLM-as-a-judge scorers

• Explanation: Code scorers handle deterministic binary checks, while LLM judges evaluate subjective criteria like tone, style, and completeness.
• Evidence: Workshops demonstrated setting up both accuracy scorers (LLM) and formatting scorers (TypeScript).
• Practical implication: Using both types provides a balanced evaluation of both strict constraints and nuanced quality.

Frameworks, Models & Processes

Braintrust Eval Framework

• How it works: Combines a task function, a test dataset, and one or more scorers to evaluate LLM outputs programmatically or via UI.
• Components:
  • Task
  • Dataset
  • Scorer
• When to use: Whenever testing new prompts, model swaps, or agentic workflows.

Examples & Case Studies

Building a changelog generation app and evaluating it against historical git commits.

• Illustrates: How to define inputs, expected outputs, prompts, and custom scorers for text summarization tasks.
• Lesson: Structured datasets and multi-metric scoring (accuracy, formatting, completeness) give granular insight into prompt performance.

Actionable Takeaways

• Immediate:
  • Start small with synthetic datasets and iterate.
  • Use Braintrust Playground for rapid prompt prototyping and A/B testing models.
  • Wrap LLM API clients with Braintrust SDK loggers to capture production traces effortlessly.
• Strategic:
  • Treat prompt engineering and evaluation as a core engineering workflow, not an afterthought.
  • Incorporate human-in-the-loop review to establish ground truth and catch nuanced failures.
  • Feed production logs back into evaluation datasets to continuously expand test coverage.
• Questions to investigate:
  • How can we balance deterministic code-based assertions with stochastic LLM-as-a-judge scorers?
  • What is the optimal sampling rate for online scoring in high-throughput production environments?

Claims Worth Verifying

• In 2023 testing, major LLMs contained factual errors or fabrications up to 91.4% of the time in certain domains. (statistic)
• Over 3 months, top LLM's code-success rate dropped from 52% to 10% an 80% decline after a routine update. (statistic)

Notable Quotes

"Writing evals is going to become a core skill for product managers. It is such a critical part of making a good product with AI." "An Eval (short for evaluation) is a structured test that checks how well your AI system performs."

Compressed Summary

• Evals ensure LLM reliability, prevent regressions, and catch hallucinations.
• Every eval requires three ingredients: Task, Dataset, and Scorer.
• Offline evals test in development; online evals monitor production logs and user feedback.
• Braintrust provides playgrounds, experiments, SDK tracing, and online scoring to streamline the LLM lifecycle.
• Keywords: evals, llm, observability, prompt-engineering, braintrust
• Core insight: Systematic evaluation using structured tasks, datasets, and scorers is essential for moving LLM applications reliably from playground experimentation to production.