LangChain · Published 2026-09-22

How To Build A Harness With Jev | A LangChain x TypeSafe Conversation

privateReview: unreviewedConfidence: high
agent harnessjevtypesafelangchainmiddlewareagent-harnessmodel-routingtool-safetyObservabilitytyped-decisionssystem-one-model
Open on YouTube ↗

Summary

Overview

  • Speaker: Sydney Runkle, Allie Laabs, Hunter Lovell
  • Channel: LangChain
  • Main topic: Building an agent harness using Jev and LangChain with TypeSafe
  • Purpose: To educate developers on building agent harnesses and leveraging Jev models for efficient agent engineering. Sydney Runkle, Allie Laabs, and Hunter Lovell discuss building agent harnesses using Jev, a System One model from TypeSafe AI. They explore how Jev differs from traditional LLMs by providing fast, typed-safe, non-text-generating decision-making capabilities (Choice, Score, Noul) that make agent development faster and cheaper. Hunter Lovell demonstrates practical integration using LangChain, LangGraph, and LangSmith middleware for tool validation and model routing.

Topic Map

Introduction to Agent Engineering and LangSmith

  • Explanation: Sydney Runkle introduces the speakers and sets the context of the agent development lifecycle (build, test, deploy, monitor, govern).
  • Key claims:
    • Agents are a new kind of software with endless inputs and non-deterministic outputs.
    • LangSmith Fleet enables building agents without code.
  • Examples:
    • Preparing for tomorrow's meeting using integrated Slack, Gmail, and Google Docs apps.
  • Terminology:
    • LangSmith
    • Agent Development Lifecycle
    • LangGraph
    • DeepAgents
  • Why it matters: Establishes the platform context for engineering robust AI agents.

What is Jev and System One Models

  • Explanation: Allie Laabs explains Jev as a System One model from TypeSafe AI that does not generate text, but instead makes fast, type-safe decisions.
  • Key claims:
    • Jev does not generate text; it is a System One model focused on fast, type-safe decisions.
    • 200-200x faster and 40-400x cheaper than traditional LLMs for structured choices.
  • Examples:
    • Routing a support ticket to billing, technical, or sales.
  • Terminology:
    • System One
    • System Two
    • TypeSafe AI
    • LLM Gateway
  • Why it matters: Introduces a new class of models designed specifically for fast machine-to-machine decisions rather than heavy generative text.

The Three Types of Questions: Choice, Score, Noul

  • Explanation: Detailed breakdown of the interface types offered by Jev: Choice (classification), Score (scaling along an axis), and Noul (Bernoulli probability / boolean continuum).
  • Key claims:
    • Choice picks one option from a list.
    • Score rates the state on a scale from 0 to n.
    • Noul (Bernoulli probability) evaluates if a statement is true or false on a continuous 0 to 1 scale.
  • Examples:
    • Evaluating customer frustration levels on a scale.
    • Checking if a message conveys urgency or time-sensitivity.
  • Terminology:
    • Choice
    • Score
    • Noul
    • Bernoulli Probability
  • Why it matters: Provides developers with explicit structural primitives for decision-making.

Live Walkthrough: Implementing Jev in LangChain Harness

  • Explanation: Hunter Lovell demonstrates Python code using LangChain and Jev middleware for auto-mode tool validation and model routing.
  • Key claims:
    • Jev middleware can intercept tool calls and block destructive actions based on defined safety criteria.
    • Model router middleware uses Jev to select between fast and deep models based on prompt complexity.
  • Examples:
    • Blocking a permanent customer account deletion tool call when unauthorized.
    • Routing a prompt to a fast model vs deep model depending on task requirements.
  • Terminology:
    • AutoModeMiddleware
    • ModelRouterMiddleware
    • LangSmith Tracing
    • SupportState
  • Why it matters: Shows practical, production-ready code integration of Jev into existing LangChain agent architectures.

Key Points

Jev is API-first software

  • Explanation: Jev is not an end-user product; it is an API product meant to be integrated into software by programmers to build smart software.
  • Evidence: Hunter Lovell explains that Jev requires a programmer to put it into software.
  • Practical implication: Developers must write harness code around Jev to utilize its decision-making speed.

Bridging human interfaces and machine code

  • Explanation: Traditional tool calling uses text generation as a bridge between machine execution and human interfaces, which is inefficient.
  • Evidence: Hunter Lovell contrasts CLI tool calling with direct machine-to-machine decision-making.
  • Practical implication: Using Jev avoids unnecessary text generation overhead in backend logic.

Observability and Auditing

  • Explanation: LangSmith tracing provides full visibility into Jev decisions, probabilities, confidence scores, and middleware executions.
  • Evidence: Hunter Lovell demonstrates viewing traces in LangSmith for TypeSafeClassifier and ModelRouterMiddleware.
  • Practical implication: Developers can audit exactly why an agent took a specific path or why a tool call was blocked.

Frameworks, Models & Processes

Agent Development Lifecycle

  • How it works: Iterative process of building, testing, deploying, and monitoring AI agents.
  • Components:
    • Build
    • Test
    • Deploy
    • Monitor
    • Govern
  • When to use: When engineering enterprise-grade AI agents systematically.

Jev Question Types

  • How it works: Categorizes input state evaluation into three distinct structured output types.
  • Components:
    • Choice
    • Score
    • Noul
  • When to use: When designing agent decision points, safety checks, or model routing.

Examples & Case Studies

A customer support agent attempts to delete a customer account without administrator approval.

  • Illustrates: AutoModeMiddleware blocking risky tool execution using Jev.
  • Lesson: Jev prevents unauthorized destructive actions by evaluating risk probability before tool execution.

A prompt is evaluated by ModelRouterMiddleware to choose between a fast model and a deep model.

  • Illustrates: Optimizing cost and latency by routing tasks appropriately.
  • Lesson: Simple tasks use fast models, while complex tasks use reasoning-heavy models.

Actionable Takeaways

  • Immediate:
    • Install LangChain TypeSafe integration for Python.
    • Implement AutoModeMiddleware for tool safety checks.
    • Use ModelRouterMiddleware to optimize model costs.
  • Strategic:
    • Adopt System One models for high-frequency decision points in agent loops.
    • Prioritize auditability and observability in agent harnesses using LangSmith.
  • Questions to investigate:
    • How do context length limits affect complex state evaluations?
    • What are the best practices for setting confidence thresholds in domain-specific tasks?

Claims Worth Verifying

  • Jev is 200-200x faster and 40-400x cheaper than traditional LLMs. (Performance benchmark)

Notable Quotes

"Agents are a new kind of software. Endless inputs and non-deterministic outputs." (at 0:02) "Jev does not generate text. It is a System One model from TypeSafe AI."

Compressed Summary

  • Jev is a fast, type-safe System One model for structured agent decisions.
  • Supports three question types: Choice, Score, and Noul.
  • Integrates smoothly into LangChain via AutoMode and ModelRouter middleware.
  • LangSmith provides comprehensive tracing and observability for agent decisions.
  • Keywords: agent harness, jev, typesafe, langchain, middleware
  • Core insight: Building robust agent harnesses requires specialized System One models like Jev for fast, type-safe decision-making rather than relying solely on heavy generative LLMs.

Core insights

6
Architecturehigh noveltystrong evidence

A distinct class of 'System One' decision models can replace text generation for machine-to-machine decisions: Jev returns typed answers (Choice, Score, Noul) rather than tokens, removing the text-generation step from agent control flow.

Why it matters

If decision points in an agent loop no longer need to round-trip through token generation, then routing, classification, and gating can be handled by a cheaper, faster, type-safe layer while a generative model is reserved for actual text work.

Generalization

Separate 'what should I decide' from 'what should I say' into different models behind different interfaces; decision layers want types and latency, generation layers want expressiveness.

Jev does not generate text; it is a System One model focused on fast, type-safe decisions.
Open source video
Jev does not generate text. It is a System One model from TypeSafe AI.
Open source video
Mechanismmedium noveltystrong evidence

Tool-call safety is enforced in harness middleware that intercepts calls before execution and blocks them against defined criteria, rather than relying on instructions embedded in prompts or in the tool itself.

Why it matters

Moves safety from a probabilistic, prompt-level suggestion to an explicit pre-execution gate in the runtime, which is a materially different reliability posture for destructive actions.

Generalization

Put authorization and risk checks in the call path between the model and the side effect, not in the model's context; any agent runtime can adopt an interception point for irreversible tools.

Jev middleware can intercept tool calls and block destructive actions based on defined safety criteria.
Open source video
Blocking a permanent customer account deletion tool call when unauthorized.
Open source video
Practicemedium noveltystrong evidence

Model routing is itself expressed as a structured decision made by the fast typed model, selecting between fast and deep models based on prompt complexity.

Why it matters

Cost and latency become a runtime optimization controlled by the harness rather than a static configuration chosen at development time.

Generalization

Any heterogeneous model fleet (cheap/expensive, small/large, local/remote) needs an explicit, auditable routing decision; the router should be the cheapest component in the stack.

Model router middleware uses Jev to select between fast and deep models based on prompt complexity.
Open source video
Simple tasks use fast models, while complex tasks use reasoning-heavy models.
Open source video
Mechanismhigh noveltymoderate evidence

The decision interface is decomposed into three typed primitives — Choice (pick one from a list), Score (0-to-n rating), and Noul (Bernoulli probability on a continuous 0-to-1 scale) — giving each decision point an explicit output shape.

Why it matters

Naming the shape of a decision (classification vs ordinal score vs boolean-with-confidence) lets engineers set thresholds and calibrate gating logic instead of parsing free text.

Generalization

Design agent control flow as a set of typed question primitives rather than natural-language prompts; each primitive implies its own thresholding and calibration strategy.

Choice picks one option from a list.
Open source video
Score rates the state on a scale from 0 to n.
Open source video
Noul (Bernoulli probability) evaluates if a statement is true or false on a continuous 0 to 1 scale.
Open source video
Practicemedium noveltymoderate evidence

Tracing exposes the internals of the decision model — probabilities, confidence scores, and middleware executions — making non-deterministic control flow auditable.

Why it matters

You can reconstruct why an agent took a path or why a tool call was blocked, which is the prerequisite for debugging and governing an agent in production.

Generalization

Any component that makes probabilistic decisions inside an agent must emit its score, not just its outcome; observability is only as good as the decision metadata emitted.

LangSmith tracing provides full visibility into Jev decisions, probabilities, confidence scores, and middleware executions.
Open source video
Developers can audit exactly why an agent took a specific path or why a tool call was blocked.
Open source video
Mental Modelhigh noveltymoderate evidence

Using text generation as the bridge between machine execution and a human-facing interface is framed as an unnecessary inefficiency in backend logic.

Why it matters

It reframes prompt/tool-call round-trips as accidental coupling to a human interface format, suggesting backend agent logic should use direct machine-to-machine decisions wherever the consumer is another program.

Generalization

Keep human-readable interfaces at the boundary where humans actually read them; internal agent-to-agent or agent-to-tool boundaries should carry typed data, not prose.

Traditional tool calling uses text generation as a bridge between machine execution and human interfaces, which is inefficient.
Open source video
Using Jev avoids unnecessary text generation overhead in backend logic.
Open source video

Deep dives

5

Threshold calibration for probabilistic tool-call gating (Noul)

Research question

For a given domain, how does the choice of Noul threshold trade off false-blocking of legitimate tool calls against false-negatives that let destructive calls through, and can a stable per-domain calibration heuristic be derived from a labeled evaluation set?

Why

The harness owner now owns the correctness of the pre-execution gate; without a calibration method, threshold selection is a guess that either blocks legitimate irreversible-adjacent work or fails its only job.

Noul (Bernoulli probability) evaluates if a statement is true or false on a continuous 0 to 1 scale.
Open source video
Jev middleware can intercept tool calls and block destructive actions based on defined safety criteria.
Open source video
What are the best practices for setting confidence thresholds in domain-specific tasks?
Open source video
Source video

Decision-model context limits vs. generative baselines at scale

Research question

At what state size does a typed decision model's Choice/Score/Noul accuracy degrade relative to a prompted LLM baseline, and does the typed model degrade earlier or later than the generative one?

Why

If typed decisions degrade faster with state size, the harness must route large-state decisions back to a generative model, which changes the whole cost model behind the 'decision layer is cheap' premise.

How do context length limits affect complex state evaluations?
Open source video
Score rates the state on a scale from 0 to n.
Open source video
Choice picks one option from a list.
Open source video
Source video

End-to-end economics of a typed router in the call path

Research question

On a mixed prompt workload, does inserting a typed routing decision before every call produce net latency and cost savings versus a single deep model once router overhead and misroute cost are included?

Why

The router adds a hop and a failure surface to every request; the claimed savings are only real if misroutes and middleware overhead do not consume them, and no measurement is reported.

Model router middleware uses Jev to select between fast and deep models based on prompt complexity.
Open source video
Simple tasks use fast models, while complex tasks use reasoning-heavy models.
Open source video
Source video

Bypass and trace-completeness of pre-execution gates

Research question

Can a blocked tool call be re-attempted or routed around by the agent after a middleware block, and are the probability and confidence values of each block actually persisted and queryable in traces?

Why

A gate is only as strong as its unbypassability, and aggregate auditing requires stored decision metadata rather than a per-run visual trace.

LangSmith tracing provides full visibility into Jev decisions, probabilities, confidence scores, and middleware executions.
Open source video
Developers can audit exactly why an agent took a specific path or why a tool call was blocked.
Open source video
Blocking a permanent customer account deletion tool call when unauthorized.
Open source video
Source video

Head-to-head equivalence of typed decisions vs. prompted structured output

Research question

For classification, ordinal scoring, and boolean decisions on the same task suite, how does a dedicated non-text decision model compare in accuracy to prompting a general LLM for the same structured output?

Why

The speed and cost claims are comparative but the summary offers no accuracy comparison; if the typed model is meaningfully less accurate, the savings are paid for in decision quality, which is worse than latency.

Jev is 200-200x faster and 40-400x cheaper than traditional LLMs for structured choices.
Open source video
Jev does not generate text; it is a System One model focused on fast, type-safe decisions.
Open source video
Choice picks one option from a list.
Open source video
Source video

Article ideas

4

Your Agent's Control Flow Should Not Speak English

Using text generation as the bridge between machine execution and human interfaces couples backend agent logic to a human-readable format it never needed, and the fix is to make every internal decision boundary carry typed data, reserving prose for the boundaries where a human actually reads the output.

Angle

Trace a single tool-call round trip and separate the bytes that exist for the machine from the bytes that exist for a human reader; show what disappears when the machine-facing layer returns typed decisions instead of prose.

Source video

The Harness Owns the Gate, Not the Prompt

Tool-call safety expressed as system-prompt instructions is a probabilistic suggestion, while a pre-execution interception layer is a deterministic choke point; moving authorization into the call path changes who is accountable for irreversible actions and makes false-block behavior an explicit engineering concern rather than an emergent one.

Angle

Argue that any irreversible tool warrants an interception point between model intent and side effect, then enumerate the new obligations the harness owner inherits once that gate exists.

Source video

The Cheapest Model in the Stack Should Be the Router

Model routing is itself a decision, so the component that arbitrates between cheap and expensive models should be the cheapest and most auditable part of the runtime; a router that costs as much as the thing it routes is a net negative and should be replaced by static configuration.

Angle

Treat routing as a heterogeneous-fleet problem: every mixed cheap/expensive fleet needs an explicit, logged routing decision, and the router's misroute rate is the metric that determines whether the fleet is worth having.

Source video

What Would Falsify a 200x Speed and 40x Cost Claim?

Comparative performance claims about decision models are only actionable when stated with a task set, a baseline, and a volume assumption; until then they should be treated as a hypothesis to test rather than a budget line, because the cost profile at production volumes can invert the ranking.

Angle

Take the headline multipliers seriously as a claim and list the measurement conditions that would need to hold for them to survive contact with a production workload.

Source video

Project ideas

4

Gatehouse Noul Gate

gatehouse

A pre-execution interception gate that blocks irreversible tool calls when a typed boolean-probability decision exceeds a calibrated threshold can drive destructive tool executions to zero on a labeled adversarial suite while keeping false-blocks on legitimate calls below an agreed rate.

Proof of concept

Instrument a harness middleware that evaluates every irreversible tool call (account deletion, payment, config mutation) with a boolean-probability decision and blocks above threshold; run a labeled suite of authorized and unauthorized calls, then sweep the threshold to trace the false-block / false-negative curve.

Measurement

False-negative rate on unauthorized calls and false-block rate on authorized calls across at least five thresholds; report the threshold where the tradeoff curve bends.

Source video

Router Economics Instrumentation

movement-lab

Adding a typed routing decision in front of a two-tier model fleet reduces net cost and p95 latency on mixed workloads only when the misroute rate stays below a workload-specific break-even point.

Proof of concept

Build a duplicate-instrumented harness that runs a mixed prompt workload twice, once through a typed router selecting fast vs. deep models and once against a single deep model, logging per-request route decision, latency, and cost.

Measurement

Net cost per request, p50/p95 latency, and misroute rate (requests where the fast model produced an output the deep model would have changed); compute the break-even misroute rate.

Source video

Threshold Calibration Bench

beyond-evals

Per-domain evaluation sets yield stable confidence-threshold heuristics for probabilistic decisions that transfer across decision points within the same domain but not across domains.

Proof of concept

Assemble labeled decision sets for two or three domains, sweep the continuous 0-to-1 decision threshold for each, and test whether a threshold calibrated on one domain's evaluation set holds on a held-out set from a different domain.

Measurement

Precision and recall of the gating decision per threshold per domain, plus cross-domain threshold transfer error.

Source video

State-Size Degradation Sweep

new

Typed decision accuracy degrades measurably beyond a context-size knee point, and that knee arrives at a smaller state size than the equivalent prompted-LLM baseline's degradation point.

Proof of concept

Take a single decision task (classification with Choice and ordinal scoring with Score) and vary the state payload size across a sweep; run the same tasks through a typed decision model and a prompted LLM baseline on identical inputs.

Measurement

Decision accuracy versus state token count for both systems; identify each system's knee point and the size at which the typed model falls below the baseline.

Source video

Architectural implications

3

Decision-making is factored out of the generative model into a separate, typed-answer service callable from middleware.

Before

A single LLM prompt handles classification, routing, and tool selection alongside text generation, with all outputs parsed from prose.

After

A fast typed model returns Choice/Score/Noul answers at the decision points, while a generative model is invoked only when text is actually required.

Consequence

Harness code becomes the orchestrator of heterogeneous models; interface definitions and thresholds, not prompts, become the primary artifacts to maintain.

Source video

Safety checks move into a pre-execution interception layer between agent intent and side effect.

Before

Guardrails are expressed as system-prompt instructions and hope the model complies before calling a destructive tool.

After

Middleware evaluates risk with a typed decision and can block the call outright before the tool runs.

Consequence

Irreversible operations gain a deterministic choke point, but the harness owner now owns that gate's correctness and false-block behavior.

Source video

Routing between cheap and expensive models becomes a runtime function of the prompt rather than a static choice.

Before

A single model (or hand-tuned per-feature selection) serves all traffic, so simple and complex tasks pay the same cost and latency.

After

A router decision assigns each request to a fast or deep model based on task requirements.

Consequence

Cost and latency profile becomes tunable at runtime, with new failure modes around misrouting and router overhead.

Source video

Tradeoffs and failure modes

3

Routed model selection

Benefit

Simple tasks use fast models, so cost and latency drop for the common case.

Cost or risk

Complex tasks sent to a non-reasoning model may silently underperform; the routing decision itself can be wrong and is only as reliable as the classifier behind it.

Simple tasks use fast models, while complex tasks use reasoning-heavy models.
Open source video
Source video

Probabilistic gating of tool calls

Benefit

Destructive actions can be blocked by evaluating risk probability before tool execution.

Cost or risk

A probability output (Noul on a 0-to-1 continuum) requires a threshold choice; the summary states best practices for setting confidence thresholds are an open question, so mis-set thresholds can block legitimate calls or let unsafe ones through.

What are the best practices for setting confidence thresholds in domain-specific tasks?
Open source video
Source video

Typed decision models and state size

Benefit

Decisions are fast, type-safe, and free of text-generation overhead.

Cost or risk

Complex state evaluation may be constrained by context length limits, which the summary lists as unresolved.

How do context length limits affect complex state evaluations?
Open source video
Source video

Open questions

4

How do context length limits affect complex state evaluations in a typed decision model?

Why unresolved

The summary records it as a question to investigate during the walkthrough rather than a demonstrated constraint.

Research direction

Benchmark decision accuracy of Choice/Score/Noul against increasing state size to find the point where typed decisions degrade relative to a generative model.

Source video

What are best practices for setting confidence thresholds in domain-specific tasks?

Why unresolved

Threshold calibration for the continuous 0-to-1 Noul output is named as an open question, not answered.

Research direction

Build evaluation sets per domain and measure precision/recall of tool-call blocking across thresholds to derive calibration heuristics.

Source video

Does inserting a router/middleware decision before every call add enough overhead or failure surface to offset the model savings?

Why unresolved

The summary provides no measurement of middleware overhead or misrouting cost, only the routing mechanism.

Research direction

Instrument end-to-end latency and misroute rate of ModelRouterMiddleware versus a single deep model on a mixed prompt workload.

Source video

How reliably does a non-text decision model generalize across decision points compared to prompting a general LLM for the same structured output?

Why unresolved

The summary asserts speed and cost advantages but offers no head-to-head accuracy comparison against prompted LLM classification.

Research direction

Run an equivalence study on classification, scoring, and boolean decisions between Jev and a prompted LLM baseline on the same task suite.

Source video

Key claims

7
comparativeVerification needed

Jev is 200-200x faster and 40-400x cheaper than traditional LLMs for structured choices.

Evidence

200-200x faster and 40-400x cheaper than traditional LLMs for structured choices.

Question

Under what task set, model baseline, and measurement conditions were the 200x speed and 40-400x cost figures obtained, and do they hold at production request volumes?

Source video
factualVerification needed

Jev does not generate text; it produces typed decisions.

Evidence

Jev does not generate text; it is a System One model focused on fast, type-safe decisions.

Question

What exactly does the API return (labels, scores, probabilities), and are there any tasks where it still emits text?

Source video
causalVerification needed

Jev middleware can intercept tool calls and block destructive actions based on defined safety criteria.

Evidence

Jev middleware can intercept tool calls and block destructive actions based on defined safety criteria.

Question

What are the false-negative and false-positive rates of blocking decisions, and can a blocked tool call be bypassed by the agent?

Source video
causalVerification needed

Model router middleware uses Jev to select between fast and deep models based on prompt complexity.

Evidence

Model router middleware uses Jev to select between fast and deep models based on prompt complexity.

Question

How accurately does the router identify prompts that genuinely require the deep model, and what is the cost of misrouting?

Source video
factualVerification needed

LangSmith tracing provides full visibility into Jev decisions, probabilities, confidence scores, and middleware executions.

Evidence

LangSmith tracing provides full visibility into Jev decisions, probabilities, confidence scores, and middleware executions.

Question

Are probability and confidence values persisted per decision in traces, and can traces be queried in aggregate for auditing?

Source video
opinionVerification not requested

Jev is API-first and requires a programmer to embed it in software.

Evidence

Jev is not an end-user product; it is an API product meant to be integrated into software by programmers to build smart software.

Source video
opinionVerification needed

Traditional tool calling uses text generation as a bridge between machine execution and human interfaces, which is inefficient.

Evidence

Traditional tool calling uses text generation as a bridge between machine execution and human interfaces, which is inefficient.

Question

How much measured latency and cost in a real agent loop is attributable specifically to text-generation-based tool calling?

Source video

Connections

4