Latent Space · Published 2026-09-21

Why I couldn't build Jev at OpenAI — Diogo Almeida, TypeSafe Co-founder & CEO

privateReview: unreviewedConfidence: high
composable aitypesafeprimitivesllm architecturesystem onestructured-outputllm-primitivestype-safetycomposable-aiautoregressive-limitscost-efficiencyevaluationsystem-one
Open on YouTube ↗

Summary

Overview

  • Speaker: Diogo Almeida
  • Channel: Latent Space
  • Main topic: Building machine-native composable AI with TypeSafe
  • Purpose: Provide a deep technical and philosophical dive into TypeSafe's approach to AI systems, reasoning models, and why the current paradigm of instruction-following LLMs falls short for production software engineering. Diogo Almeida, co-founder and CEO of TypeSafe, discusses the launch of Jev, the reasoning challenges in AI, why auto-regressive LLMs are structurally limited for software, and the philosophy behind machine-native composable AI.

Topic Map

The State of AI and Jev Launch

  • Explanation: Diogo discusses the emotional and technical reality of launching Jev, navigating chaotic industry expectations, and prioritizing developers over enterprise buyers.
  • Key claims:
    • The AI field is a carnival of mirrors with people saying weird things.
    • Jev is optimized for intelligence-per-dollar.
    • Targeting developers and engineers is key before moving to enterprise.
  • Examples:
    • Jev trending on Twitter and being categorized as a personal feed.
  • Terminology:
    • Jev
    • System One models
    • intelligence-per-dollar
  • Why it matters: Establishes the positioning of TypeSafe in the rapidly shifting AI landscape.

Critique of Auto-Regressive LLMs

  • Explanation: Analyzing Yann LeCun's claims about auto-regressive LLMs being doomed and why probability error diverges exponentially with sequence length.
  • Key claims:
    • Auto-regressive LLMs cannot be made factual, non-toxic, or fully controllable.
    • As sequence length increases, the probability of error diverges exponentially.
    • LLMs struggle with decision-making because string models are bad at reasoning over extended token outputs.
  • Examples:
    • Yann LeCun's slide on Unpopular Opinions about AR-LLMs.
  • Terminology:
    • Auto-regressive LLMs
    • mode collapse
    • mode dropping
    • sequence length
  • Why it matters: Frames the fundamental architectural critique that motivated the creation of TypeSafe's system.

TypeSafe Primitives and Architecture

  • Explanation: Deep dive into TypeSafe's primitives: Choice, Score, and Noul, designed for system-one reasoning and structured JSON output.
  • Key claims:
    • TypeSafe primitives map directly to programming primitives like switches, if-statements, and sorting.
    • Structured JSON output eliminates template-based prompting fragility.
    • System One models operate beyond simple instruction-following.
  • Examples:
    • Using Noul for yes/no validation.
    • Using Score for thresholding bug severity.
  • Terminology:
    • Choice
    • Score
    • Noul
    • System One
    • Structured JSON
  • Why it matters: Provides the technical solution to bridging AI models with robust software engineering.

Reliability vs. Capabilities Alignment

  • Explanation: Discussing the tension between capability alignment and safety alignment, arguing that safety alignment often misaligns with developer needs.
  • Key claims:
    • Capability alignment means doing what the user wants; safety alignment often restricts developer utility.
    • Reliability is a catch-all for type safety, determinism, and robustness.
    • Public benchmarks are easily gameable and lead to overfitted models.
  • Examples:
    • APIs rejecting requests due to misaligned safety guardrails.
  • Terminology:
    • Capability alignment
    • Safety alignment
    • Public benchmarks
    • Robustness
  • Why it matters: Defines TypeSafe's stance on developer freedom and system reliability.

Key Points

Intelligence-per-dollar is the ultimate metric

  • Explanation: Optimizing models for cost and speed while maintaining high reasoning capability determines practical economic value.
  • Evidence: Jev's design philosophy focusing on cost efficiency over raw parameter scale.
  • Practical implication: Enables scalable AI integration into production software without prohibitive costs.

Decomposition beats monolithic prompt engineering

  • Explanation: Breaking complex tasks into smaller, structured, typed queries yields more reliable software than relying on single massive prompt passes.
  • Evidence: TypeSafe SDK usage patterns mapping questions into structured JSON primitives.
  • Practical implication: Reduces failure rates and hallucinations in agentic workflows.

Public benchmarks are anti-thetical to real reliability

  • Explanation: Publicly available benchmarks can be gamed and do not reflect real-world workflow performance.
  • Evidence: Labs overfitting models to leaderboards rather than building robust composable primitives.
  • Practical implication: Developers should rely on internal evaluations and workflow-specific metrics.

Frameworks, Models & Processes

TypeSafe Primitives Framework

  • How it works: Combines question types into structured JSON schemas that enforce type safety and deterministic parsing on LLM outputs.
  • Components:
    • Choice (select from options with probabilities and confidence)
    • Score (rate against criteria with confidence and level definitions)
    • Noul (yes/no evaluation with probability)
  • When to use: When building robust, production-grade AI agents and software pipelines requiring deterministic outputs.

Examples & Case Studies

Andy Gao built a voice-controlled computer use application for Mac using TypeSafe's Jev.

  • Illustrates: The speed and responsiveness of machine-native composable AI in real-world automation.
  • Lesson: Composability and low latency enable rich, interactive agentic experiences.

Actionable Takeaways

  • Immediate:
    • Explore TypeSafe documentation and primitives (Choice, Score, Noul).
    • Decompose complex prompts into structured JSON outputs.
  • Strategic:
    • Prioritize reliability and intelligence-per-dollar over raw model size.
    • Build software abstractions that decouple business logic from underlying LLM calls.
  • Questions to investigate:
    • How do scaling laws apply to structured reasoning models?
    • What is the optimal trade-off between KV caching and multi-agent latency?

Claims Worth Verifying

  • Probability error diverges exponentially with sequence length in auto-regressive LLMs. (theoretical)
  • TypeSafe primitives allow 10x cheaper and 10x faster batching via structured parallel queries. (architectural)

Notable Quotes

"How can AI be so unbelievably smart? How can we like, solve Millennium Prize problems in math, but still not automate even the most basics of works." (at 0:00) "Composable AI: Build Prod, Not God." (at 34:09)

Compressed Summary

  • Auto-regressive LLMs face exponential error divergence over long sequences.
  • TypeSafe introduces primitives like Choice, Score, and Noul for structured AI integration.
  • Intelligence-per-dollar and reliability matter more than gaming public benchmarks.
  • Keywords: composable ai, typesafe, primitives, llm architecture, system one
  • Core insight: Building reliable production software with AI requires moving beyond monolithic prompting toward machine-native composable primitives and structured reasoning.

Core insights

6
Architecturehigh noveltymoderate evidence

Reliability in LLM-backed software can be moved out of the prompt and into the type system by decomposing work into machine-native primitives (Choice, Score, Noul) that emit structured JSON with probabilities/confidence, where each primitive maps to a programming construct (switch, if-statement, sorting, thresholding).

Why it matters

It relocates the responsibility for correctness from fragile template text and post-hoc parsing into a typed contract, making LLM calls behave like ordinary typed function calls that business logic can be written against.

Generalization

Any agent runtime can treat model invocation as a set of narrow typed query operators rather than a free-form generation step; this applies wherever outputs must be parsed deterministically or fed into downstream code.

TypeSafe primitives map directly to programming primitives like switches, if-statements, and sorting.
Open source video
Combines question types into structured JSON schemas that enforce type safety and deterministic parsing on LLM outputs.
Open source video
Structured JSON output eliminates template-based prompting fragility.
Open source video
Mental Modelhigh noveltyweak evidence

Sequence length is framed as the structural failure axis of auto-regressive LLMs: the probability of error is claimed to diverge exponentially as sequence length grows, and string models are claimed to be bad at reasoning over extended token outputs.

Why it matters

If the failure is a function of generated span rather than prompt quality, then the highest-leverage engineering move is reducing tokens generated per decision, not improving prompt wording or adding retries.

Generalization

Motivates an agent design rule of short, bounded outputs per step and explicit composition across steps instead of long chain-of-thought passes for anything that must be dependable.

As sequence length increases, the probability of error diverges exponentially.
Open source video
LLMs struggle with decision-making because string models are bad at reasoning over extended token outputs.
Open source video
Probability error diverges exponentially with sequence length in auto-regressive LLMs. (theoretical)
Open source video
Tradeoffmedium noveltymoderate evidence

Intelligence-per-dollar (reasoning capability per unit cost and latency) is proposed as the primary optimization target, ahead of raw parameter scale, with the model line explicitly positioned for developers before enterprise.

Why it matters

Reframes model selection as a cost/latency engineering decision for production workloads rather than a capability-maximization decision, which changes what you benchmark and what you deploy.

Generalization

Applies to any production agent: the viable abstraction is the cheapest model that hits the required reliability on your workflow-specific eval, not the strongest model available.

Jev is optimized for intelligence-per-dollar.
Open source video
Optimizing models for cost and speed while maintaining high reasoning capability determines practical economic value.
Open source video
Tradeoffmedium noveltymoderate evidence

Safety alignment is presented as frequently opposed to capability alignment: guardrails tuned for safety can reject legitimate developer requests, and 'reliability' is redefined to mean type safety, determinism, and robustness rather than refusal behavior.

Why it matters

For production systems, misaligned safety guardrails become an availability failure mode — nondeterministic refusals in code paths the developer already authorized — so refusal behavior must be treated as part of the reliability contract.

Generalization

Any agent platform calling third-party hosted models inherits a policy layer it does not control; reliability engineering must account for policy-induced nondeterminism, not just model error.

Capability alignment means doing what the user wants; safety alignment often restricts developer utility.
Open source video
APIs rejecting requests due to misaligned safety guardrails.
Open source video
Reliability is a catch-all for type safety, determinism, and robustness.
Open source video
Practicemedium noveltymoderate evidence

Public leaderboard benchmarks are claimed to be gameable and to induce overfitting away from composable primitives, so evaluation should be internal and workflow-specific instead of benchmark-driven.

Why it matters

Changes how models are selected and how progress is measured inside a team: workflow-level evals replace public scores as the decision signal, which also implies owning eval infrastructure.

Generalization

Directly applicable to agent engineering: score models on your own task distribution and failure taxonomy, because public rankings do not predict performance on decomposable production pipelines.

Public benchmarks are easily gameable and lead to overfitted models.
Open source video
Labs overfitting models to leaderboards rather than building robust composable primitives.
Open source video
Developers should rely on internal evaluations and workflow-specific metrics.
Open source video
Empirical Resulthigh noveltyweak evidence

The claimed payoff of structured parallel queries is roughly 10x cheaper and 10x faster batching relative to the alternative, and system-one style models are positioned as operating beyond simple instruction following.

Why it matters

If batching narrow typed queries is materially cheaper and faster than monolithic generation, it changes the economics and latency profile of agent loops and enables interactive applications.

Generalization

Suggests agent throughput is bounded by query granularity and batching strategy, not only by model choice — a testable claim about runtime design.

TypeSafe primitives allow 10x cheaper and 10x faster batching via structured parallel queries. (architectural)
Open source video
System One models operate beyond simple instruction-following.
Open source video

Deep dives

4

Structured typed primitives vs monolithic prompting: reliability and cost

Research question

Under controlled conditions, how much of the reported 10x cost and latency improvement from typed primitives is attributable to structured output constraints versus parallel batching and hardware effects, and does the primitive approach reduce end-to-end error rates?

Why

If the gain is mostly from batching rather than type safety, the architectural prescription changes; if it comes from constraining the output space, it validates decomposition as a reliability strategy.

TypeSafe primitives allow 10x cheaper and 10x faster batching via structured parallel queries.
Open source video
TypeSafe primitives map directly to programming primitives like switches, if-statements, and sorting.
Open source video
Structured JSON output eliminates template-based prompting fragility.
Open source video
Source video

Exponential error divergence with sequence length

Research question

Does per-step error probability grow exponentially with generated sequence length on decision tasks, and does constraining output to short typed responses flatten the curve?

Why

This is the core motivation for decomposition; if the curve is real, production agents must cap generated spans and compose across steps rather than rely on long chain-of-thought.

As sequence length increases, the probability of error diverges exponentially.
Open source video
LLMs struggle with decision-making because string models are bad at reasoning over extended token outputs.
Open source video
Source video

Safety alignment as a production availability risk

Research question

How often do safety-aligned model APIs reject legitimate developer requests, and what fallback architectures (provider routing, self-hosting) minimize availability loss?

Why

Refusal nondeterminism in authorized code paths is a reliability failure; treating safety alignment as an availability concern forces engineering for fallbacks and policy-aware telemetry.

Capability alignment means doing what the user wants; safety alignment often restricts developer utility.
Open source video
Reliability is a catch-all for type safety, determinism, and robustness.
Open source video
Source video

Decomposing open-ended synthesis into typed primitives

Research question

To what extent can open-ended generation tasks be decomposed into Choice/Score/Noul primitives, and where does the abstraction break down?

Why

The primitive framework is only demonstrated on classification and thresholding; the boundary of applicability determines whether it can cover general agent workloads.

TypeSafe primitives map directly to programming primitives like switches, if-statements, and sorting.
Open source video
Using Noul for yes/no validation.
Open source video
Using Score for thresholding bug severity.
Open source video
Source video

Article ideas

4

The Prompt is Not a Type System: Moving Reliability into the Interface

Reliability in LLM-backed software is an interface problem, not a prompt-engineering problem; typed primitives and structured outputs make model calls behave like ordinary typed functions.

Angle

Argue that prompt templates push correctness into fragile text that must be parsed and repaired, while typed primitives (Choice, Score, Noul) relocate correctness to a schema boundary where failures become type errors.

Source video

Long Generation is a Reliability Bug

Auto-regressive sequence length is a structural risk: error probability diverges exponentially, so production agents should cap generated spans and compose short typed queries.

Angle

Connect the theoretical claim about exponential error to practical agent design rules: short bounded outputs per step, explicit composition across steps, and batching narrow queries instead of monolithic prompts.

Source video

Safety Guardrails Are an Availability Risk

Safety alignment often opposes capability alignment, creating nondeterministic refusals that break legitimate production code paths; reliability engineering must treat refusals as availability failures.

Angle

Recast guardrail rejections as a reliability and availability concern, not just a content concern, and outline fallback strategies like provider routing and self-hosting.

Source video

Benchmarks Are Not Evals

Public leaderboards are gameable and drive overfitting away from composable primitives, so teams must own workflow-specific evaluation harnesses to select models and measure progress.

Angle

Show why public scores fail to predict performance on decomposable production workflows and what internal eval infrastructure should measure instead.

Source video

Project ideas

4

Typed Primitive Runtime Benchmark

new

Structured typed queries (Choice/Score/Noul) reduce end-to-end error rate and cost compared to monolithic prompting on a production-like workflow, after controlling for total tokens and hardware.

Proof of concept

Implement a small runtime with typed primitives; run a suite of decision tasks (bug triage, content classification) as monolithic prompts versus decomposed typed queries; measure error rate, cost per task, p95 latency, and parse failure rate.

Measurement

Relative error reduction, cost per task, p95 latency, and parse failure rate compared to the monolithic baseline.

Source video

Sequence Length Error Curve

beyond-evals

Per-step error probability grows exponentially with generated sequence length on decision tasks, and constraining output to short typed responses flattens the curve.

Proof of concept

Generate synthetic decision tasks with controllable sequence length; measure error versus length for free-form outputs versus typed short outputs; fit an exponential growth model to each condition.

Measurement

Fitted exponential growth parameter and comparison of error curves between free-form and typed outputs.

Source video

Refusal Reliability Monitor

gatehouse

Safety-aligned API refusals on benign developer queries occur at a measurable rate that degrades production availability; fallback routing reduces availability loss.

Proof of concept

Build a proxy that logs refusals across providers for a fixed set of benign developer queries; implement fallback to an alternate provider or self-hosted model; measure availability improvement and latency cost.

Measurement

Refusal rate per provider, availability improvement with fallback, and added p95 latency.

Source video

Workflow-Specific Eval Harness

beyond-evals

Public benchmark scores do not predict performance on decomposable production workflows; internal evals with a failure taxonomy better select models.

Proof of concept

Build a small eval harness for a real workflow (e.g., code review triage) and compare model rankings from public leaderboards vs internal evals; measure rank correlation and cost savings.

Measurement

Rank correlation between public scores and internal performance; cost savings from selecting the best internal model.

Source video

Architectural implications

4

Correctness responsibilities shift from prompt text and output parsing to the interface between model and program.

Before

Business logic is entangled with prompt templates, and correctness depends on free-form text that must be parsed and repaired after the fact.

After

Model calls are expressed as typed primitives (Choice, Score, Noul) returning structured JSON with explicit confidence, and business logic is written against those typed results.

Consequence

Parsing/retry scaffolding shrinks and is replaced by schema validation; failures become type errors at a defined boundary rather than silent textual drift.

Source video

Agent step granularity becomes a deliberate design parameter rather than an artifact of prompting.

Before

Complex tasks are attempted as a single large prompt pass whose error probability grows with generated sequence length.

After

Complex tasks are decomposed into many small, typed, parallel queries with short outputs that are composed by ordinary code.

Consequence

Per-step error is bounded at the cost of orchestration overhead, driving new questions about batching, KV caching, and multi-agent latency.

Source video

Model selection is driven by a workflow-local cost/reliability frontier instead of public capability rankings.

Before

Models are chosen by leaderboard position, with evaluation delegated to public benchmarks.

After

Models are chosen by intelligence-per-dollar on internal, workflow-specific evaluations.

Consequence

Teams must own eval harnesses and cost/latency telemetry; public benchmark movement loses decision authority.

Source video

Third-party policy layers become part of the runtime reliability surface.

Before

Model APIs are assumed to execute any authorized developer request.

After

Safety alignment is treated as an independent, sometimes hostile, layer that can reject valid requests.

Consequence

Reliability engineering must include refusal handling, fallback providers, or self-hosted models as an availability strategy.

Source video

Tradeoffs and failure modes

5

Decomposition into structured primitives vs monolithic prompting

Benefit

Lower failure and hallucination rates, deterministic parsing, type safety, and cheap parallel batching.

Cost or risk

More orchestration, many round trips, and new latency/state management problems (the summary explicitly leaves KV caching vs multi-agent latency unresolved).

Decomposition beats monolithic prompt engineering
Open source video
Source video

Safety alignment vs capability alignment

Benefit

Guardrails constrain harmful outputs for general users.

Cost or risk

Guardrails reject legitimate developer requests, creating nondeterministic refusals that break production code paths the user already intended.

APIs rejecting requests due to misaligned safety guardrails.
Open source video
Source video

Cost-optimized models vs raw capability

Benefit

Cheaper, faster inference that makes production integration economically viable at scale.

Cost or risk

Optimizing for intelligence-per-dollar risks capability regressions on tasks that need deep reasoning, unless constrained by workflow-specific evals.

Jev is optimized for intelligence-per-dollar.
Open source video
Source video

Public benchmarks vs real reliability

Benefit

Public benchmarks are cheap, comparable signals of general capability.

Cost or risk

They are gameable, drive overfitting to leaderboards, and do not measure performance on decomposable production workflows.

Public benchmarks are easily gameable and lead to overfitted models.
Open source video
Source video

Long auto-regressive generation vs short bounded outputs

Benefit

Long generation allows rich, open-ended responses in a single pass.

Cost or risk

Error probability is claimed to diverge exponentially with sequence length, making long outputs structurally unreliable for decision-making.

As sequence length increases, the probability of error diverges exponentially.
Open source video
Source video

Open questions

5

Do scaling laws for structured reasoning models behave like those for free-form generation models, or does constraining the output space change the curve?

Why unresolved

The summary raises this as an unanswered question and provides no measurements; the primitives are asserted rather than characterized across scales.

Research direction

Train or evaluate matched models under structured-primitive output constraints versus free-form output and compare accuracy, cost, and latency scaling.

Source video

What is the optimal trade-off between KV caching and multi-agent latency in a runtime built from many small typed queries?

Why unresolved

Explicitly listed as an open question; decomposition creates many short calls whose caching behavior competes with parallel fan-out.

Research direction

Benchmark orchestration topologies (shared prefix KV cache vs independent parallel agent calls) on end-to-end latency and cost under realistic query mixes.

Source video

How much of the claimed 10x cheaper / 10x faster batching is attributable to structured primitives versus batching and hardware effects?

Why unresolved

The claim is architectural and unverified in the summary; no baseline, workload, or measurement conditions are given.

Research direction

Reproduce with a controlled workload: identical task sets executed as monolithic prompts versus batched typed queries, holding model and hardware constant.

Source video

Does exponential error divergence over sequence length hold for modern reasoning models, or is it an artifact of a specific modeling assumption?

Why unresolved

The summary marks this as theoretical and cites only a slide of unpopular opinions, not an experiment.

Research direction

Empirically measure per-step error probability versus output length on decision tasks and fit the growth curve.

Source video

How do typed primitives handle tasks that are inherently open-ended rather than reducible to selection, scoring, or yes/no?

Why unresolved

The framework enumerates only Choice, Score, and Noul, leaving open-ended synthesis unaddressed in the summary.

Research direction

Test decomposition of open-ended generation into primitive sub-steps and measure where the abstraction breaks down.

Source video

Key claims

8
causalVerification needed

Probability of error in auto-regressive LLMs diverges exponentially as sequence length increases.

Evidence

As sequence length increases, the probability of error diverges exponentially.

Question

Can per-step error probability versus generated length be measured and shown to grow exponentially on decision tasks, and under what modeling assumptions?

Source video
comparativeVerification needed

TypeSafe primitives allow 10x cheaper and 10x faster batching via structured parallel queries.

Evidence

TypeSafe primitives allow 10x cheaper and 10x faster batching via structured parallel queries. (architectural)

Question

Against what baseline, on what workload and hardware, does structured parallel querying achieve ~10x cost and latency improvements?

Source video
predictionVerification needed

Auto-regressive LLMs cannot be made factual, non-toxic, or fully controllable.

Evidence

Auto-regressive LLMs cannot be made factual, non-toxic, or fully controllable.

Question

Is this an absolute architectural impossibility, or a statement about current training and alignment techniques?

Source video
causalVerification needed

Public benchmarks are gameable and lead labs to overfit models away from robust composable primitives.

Evidence

Public benchmarks are easily gameable and lead to overfitted models.

Question

Is there measurable evidence that leaderboard-optimized models underperform on workflow-specific composable tasks?

Source video
causalVerification needed

Structured JSON output eliminates template-based prompting fragility.

Evidence

Structured JSON output eliminates template-based prompting fragility.

Question

Compared to schema-constrained decoding or plain JSON-mode prompting, does the primitive approach measurably reduce parse failures and downstream errors?

Source video
opinionVerification not requested

Safety alignment often restricts developer utility while capability alignment means doing what the user wants.

Evidence

Capability alignment means doing what the user wants; safety alignment often restricts developer utility.

Source video
comparativeVerification needed

Decomposing complex tasks into smaller structured typed queries yields more reliable software than single massive prompt passes.

Evidence

Breaking complex tasks into smaller, structured, typed queries yields more reliable software than relying on single massive prompt passes.

Question

On which task families does decomposition outperform monolithic prompting after controlling for total tokens and number of calls?

Source video
opinionVerification needed

System One models operate beyond simple instruction-following, and primitives are designed for system-one reasoning.

Evidence

System One models operate beyond simple instruction-following.

Question

What concretely distinguishes system-one primitive behavior from instruction-following with constrained decoding?

Source video

Connections

6