AI Engineer · Published 2026-08-19

Trading Desks to Clinical Trials: Parallels in Applied Vertical AI — Ayush Bhardwaj, Allos AI

privateReview: unreviewedConfidence: high
vertical aiagentsmoatdomain expertisehealthcarefinancevertical-aiverifiable-rewardsLLM-as-judgeexpert-in-the-loophitl-vs-aitlagent-observabilityproprietary-data-moateval-harness
Open on YouTube ↗

Summary

Overview

  • Speaker: Ayush Bhardwaj
  • Channel: AI Engineer
  • Main topic: Building and scaling applied vertical AI in high-stakes industries like finance and pharma
  • Purpose: To educate AI engineers on the realities of building and deploying vertical AI agents in high-stakes, regulated industries, highlighting why domain expertise and proprietary data are critical moats. Ayush Bhardwaj shares lessons from building applied vertical AI for hedge funds and pharma-tech startups (Allos AI). He explains that while the easy part of building AI agents fits on a single screen (tools, prompts, run loops), the real moat in vertical AI is not the base model, infrastructure, or ecosystem, but domain expertise and proprietary data. The talk outlines a seven-step recipe for applied vertical AI, emphasizing that the traditional human-in-the-loop paradigm must shift to AI-in-the-loop because models cannot yet reason from cause rather than correlation.

Topic Map

Defining Applied Vertical AI

  • Explanation: Explaining what applied vertical AI is, contrasting general-purpose AI (like Google Translate) with industry-specific AI built on a field's own data to simulate human jobs.
  • Key claims:
    • Applied vertical AI is built for one industry and runs on that field's own data.
    • Its aim is to simulate the job of a person in that particular industry.
  • Examples:
    • Google Translate as general-purpose applied AI
    • Allos AI (building drugs with AI on a lab's own data)
    • Legal-tech and prop-tech firms building vertical AI
  • Terminology:
    • Applied Vertical AI
    • Allos AI
    • Pharma-tech
  • Why it matters: Clarifies the distinction between building generic AI wrappers versus high-value domain-specific agent systems.

The Seven-Step Recipe for Vertical AI

  • Explanation: A structured approach to building vertical AI agents, covering problem formulation, data identification, prompting, observability, hiring the user, iteration, and shipping.
  • Key claims:
    • You must pick a very narrow task rather than asking an agent to do everything.
    • Proprietary data is the moat because base models have never seen it.
    • Steps 1-4 are the easy part that fits on one screen; iteration and evaluation are where projects fail.
  • Examples:
    • Finance: flag when a bond looks mispriced
    • Pharma: rank which molecules to test next
    • Trader's checklist for a thesis vs. scientist's protocol for a screen
  • Terminology:
    • Formulate the problem
    • Identify the data
    • Write the prompts
    • Observability
    • Hire the user
    • Iterate
    • Ship
  • Why it matters: Provides a practical blueprint for engineering teams attempting to deploy vertical AI products.

Why You Can't Iterate (The Moat Problem)

  • Explanation: Discussing why standard reinforcement learning and iteration loops fail in finance and pharma: models cannot verify themselves, and the necessary proprietary data was never on the internet.
  • Key claims:
    • The model cannot verify itself in niche domains because reinforcement learning via verifiable rewards is only great at math and code.
    • The trade theses in finance and failed experiments in pharma are proprietary and never published.
    • An LLM judge cannot tell alpha from jargon because it predicts the next probable word rather than understanding real value creation.
  • Examples:
    • Hedge funds publishing long-position holdings quarterly (resulting in lower returns)
    • Pharma companies hiding nearly a third of clinical trial results where trials failed
  • Terminology:
    • Reinforcement learning
    • Proprietary data
    • Alpha
    • Echo chamber
  • Why it matters: Explains why 89% of enterprise AI agents never reach production: they lack the proprietary training signals needed for vertical accuracy.

Hire the User and Build the Learning Loop

  • Explanation: The solution to the iteration problem: hiring domain experts (traders, scientists) to curate data, write prompts, and serve as the expert judge in the learning loop.
  • Key claims:
    • A domain expert who can curate data is the ultimate moat.
    • The expert helps decompose problems, refine prompts, and judge outputs.
    • Moving from Human-in-the-Loop (HITL) to AI-in-the-loop (AITL) where the expert does the work and AI assists.
  • Examples:
    • Hiring a 20-year-old scientist or senior trader to evaluate AI outputs
    • Supervised fine-tuning, reinforcement learning from human feedback, rubrics as rewards, and error analysis
  • Terminology:
    • Domain expert
    • Human-in-the-loop (HITL)
    • AI in the loop (AITL)
    • Supervised fine-tuning
    • Rubrics as rewards
  • Why it matters: Shifts the perspective on enterprise AI development from pure software engineering to expert-in-the-loop data generation systems.

Key Points

Steps 1 through 4 of building AI agents are the easy part.

  • Explanation: Tools, prompts, run loops, and basic observability can be built by a single 10x engineer in minutes.
  • Evidence: A standard agent script using an LLM, tools, and prompt loader fits on a single slide.
  • Practical implication: Engineering teams should not mistake writing basic agent code for having a sustainable product moat.

Model, infrastructure, and ecosystem are commodities; domain expertise is the moat.

  • Explanation: Everyone has access to base models and cloud providers, but proprietary data and expert judgment cannot be bought off the shelf.
  • Evidence: Major foundation model providers do not have access to private hedge fund trade theses or failed pharma clinical trial data.
  • Practical implication: Startups and enterprises must invest heavily in acquiring or generating proprietary domain data through expert workflows.

Finance and pharma require AI-in-the-loop (AITL) rather than Human-in-the-Loop (HITL).

  • Explanation: In high-stakes domains, experts do the primary work and AI assists by generating candidate hypotheses or narrowing options, because models reason from correlation, not causation.
  • Evidence: LLMs cannot distinguish alpha from jargon without domain expert supervision.
  • Practical implication: Do not attempt to fully automate expert judgment in specialized verticals too early.

Frameworks, Models & Processes

The Seven-Step Vertical AI Recipe

  • How it works: A sequential methodology for designing, building, evaluating, and scaling vertical AI agents.
  • Components:
      1. Formulate the problem (pick one narrow job)
      1. Identify the data (curate proprietary data sources)
      1. Write the prompts (model the expert's workflow)
      1. Observability (log tool calls and traces)
      1. Hire the user (bring in domain experts)
      1. Iterate (build continuous learning loops)
      1. Ship (deploy and scale with continuous data generation)
  • When to use: When developing AI applications for specialized industries like finance, healthcare, legal, or pharma.

Examples & Case Studies

Transitioning from hedge fund applied AI to pharma-tech AI startup.

  • Illustrates: That the underlying DNA of building vertical AI is identical regardless of whether time-to-market is weeks or years.
  • Lesson: The core engineering challenges and data requirements remain consistent across distinct regulated industries.

Attempting to use an LLM as the judge for financial alpha generation.

  • Illustrates: An LLM cannot evaluate alpha because it only predicts the next word and lacks real-world trading experience.
  • Lesson: You cannot automate the judge in domains requiring true causal understanding without human domain experts.

Actionable Takeaways

  • Immediate:
    • Narrow down your agent's task scope to a single specific job.
    • Set up robust observability for tool calls and prompt traces from day one.
    • Evaluate whether your proprietary data source is truly defensible.
  • Strategic:
    • Hire domain experts early to guide data curation and agent evaluation.
    • Shift from human-in-the-loop to AI-in-the-loop workflows in complex domains.
    • Recognize that model infrastructure is a commodity; your data loop is your moat.
  • Questions to investigate:
    • How can we capture proprietary feedback loops from our domain experts efficiently?
    • Are our AI agent outputs evaluated on correlation or causation?

Claims Worth Verifying

  • 89% of enterprise AI agents never reach production. (statistic)
  • Nearly a third of clinical trials that must report results never do. (statistic)

Notable Quotes

"People ship agents everything. The only question is whether they work." (at 5:11) "The DNA of building it is similar, whether your time to market is weeks or years." (at 3:51) "The work that encodes real judgment was never on the internet, so it was never in the model." (at 17:04) "Model, infra, ecosystem equals commodity. Domain expertise equals moat." (at 31:44)

Compressed Summary

  • Applied vertical AI requires narrowing agent tasks to specific industry jobs.
  • Steps 1-4 of building agents are easy; the challenge lies in evaluation and proprietary data.
  • Models cannot verify themselves in niche domains because reinforcement learning requires verifiable rewards.
  • Finance and pharma hide their winning trades and failed experiments, making public data insufficient.
  • Hiring domain experts to curate data and act as evaluators forms the ultimate defensible moat.
  • Keywords: vertical ai, agents, moat, domain expertise, healthcare, finance
  • Core insight: In vertical AI, model infrastructure is a commodity, and true defensibility comes from proprietary data and domain expert learning loops.

Core insights

6
Mechanismmedium noveltymoderate evidence

Reinforcement learning with verifiable rewards is only effective where a ground-truth verifier exists (math, code); in vertical domains like finance and pharma the model cannot verify itself because the outcome signal is never published, so the bottleneck is verifier/data availability rather than model capability.

Why it matters

It relocates the hard part of vertical agent engineering away from modeling and toward constructing a reward/verification signal, which changes what teams should budget and hire for.

Generalization

Any agent domain without cheap programmatic verification (law, medicine, investing, ops triage) needs an externally supplied judge, not a self-improving loop.

The model cannot verify itself in niche domains because reinforcement learning via verifiable rewards is only great at math and code.
Open source video
The trade theses in finance and failed experiments in pharma are proprietary and never published.
Open source video
Mental Modelmedium noveltymoderate evidence

A domain expert who can curate data is the moat; models, infrastructure, and ecosystem are commodities, and the expert's role is to decompose problems, refine prompts, and act as the judge in the learning loop.

Why it matters

It reframes hiring and team structure as the core technical investment: the expert is not a customer or a reviewer but a component in the data-generation pipeline.

Generalization

Value in applied AI accrues to whoever can convert expert judgment into structured training/eval signal, not to whoever assembles the most capable agent stack.

A domain expert who can curate data is the ultimate moat.
Open source video
The expert helps decompose problems, refine prompts, and judge outputs.
Open source video
Major foundation model providers do not have access to private hedge fund trade theses or failed pharma clinical trial data.
Open source video
Failure Modehigh noveltymoderate evidence

LLM-as-judge breaks specifically in jargon-heavy verticals: the judge predicts the next probable word rather than modeling real value creation, so it cannot distinguish genuine alpha from plausible-sounding jargon and collapses into an echo chamber.

Why it matters

It invalidates the default cheap-eval strategy for exactly the domains where evaluation is most expensive, and predicts reward hacking / score inflation in automated eval loops.

Generalization

Judge validity is a function of whether the target quality dimension correlates with text likelihood; if not, judge scores will reward fluency and penalize correct-but-unconventional outputs.

An LLM judge cannot tell alpha from jargon because it predicts the next probable word rather than understanding real value creation.
Open source video
Architecturemedium noveltymoderate evidence

In high-stakes verticals the automation boundary inverts from Human-in-the-Loop to AI-in-the-loop: the expert performs the primary work while the AI generates candidate hypotheses or narrows options, because models reason from correlation rather than causation.

Why it matters

It specifies an agent output contract (candidate generation / ranking) instead of a decision contract (final answer), which determines UI, latency budget, and where human review must be a hard gate.

Generalization

When causality is required and unverifiable, position the model as a recall-oriented candidate generator with a human ranker/decider downstream.

Moving from Human-in-the-Loop (HITL) to AI-in-the-loop (AITL) where the expert does the work and AI assists.
Open source video
Do not attempt to fully automate expert judgment in specialized verticals too early.
Open source video
Practicemedium noveltymoderate evidence

Steps 1-4 of an agent build (formulate problem, identify data, write prompts, observability) are commoditized and fit on one screen; the project's failure or success is determined by the iteration and evaluation loop built on expert-generated rubrics, SFT/RLHF, and error analysis.

Why it matters

It tells teams where to stop investing (agent scaffolding) and where the durable engineering work is (the feedback/curation loop), which affects roadmap and staffing decisions.

Generalization

Agent scaffolding is converging to commodity; the defensible engineering artifact is the domain-specific eval-and-data-flywheel harness.

Steps 1-4 are the easy part that fits on one screen; iteration and evaluation are where projects fail.
Open source video
Tools, prompts, run loops, and basic observability can be built by a single 10x engineer in minutes.
Open source video
Supervised fine-tuning, reinforcement learning from human feedback, rubrics as rewards, and error analysis
Open source video
Tradeofflow noveltymoderate evidence

Narrow task scoping is a prerequisite for iteration: 'flag when a bond looks mispriced' or 'rank which molecules to test next' rather than a general agent, because a scoped task is the only thing an expert can reliably judge and refine.

Why it matters

Scope is chosen for evaluability rather than for product ambition, which constrains what the first shipped version can claim to do.

Generalization

In expert-judged domains, task granularity should be derived from the smallest unit a domain expert can grade consistently, not from user-perceived value.

You must pick a very narrow task rather than asking an agent to do everything.
Open source video
Finance: flag when a bond looks mispriced
Open source video
Pharma: rank which molecules to test next
Open source video

Deep dives

5

Verifier scarcity and delayed credit assignment in vertical agent loops

Research question

When outcome labels arrive months or years later and are confounded (market regime, co-interventions), what pipeline shapes can attribute an agent's proposal back to the realized outcome without collapsing into noise?

Why

It decides whether a vertical agent can have a learning loop at all. If proposals cannot be linked to outcomes, the eval harness degenerates into static rubric scoring and the product never improves past its first version — the exact condition behind the 89% never-reach-production claim.

The model cannot verify itself in niche domains because reinforcement learning via verifiable rewards is only great at math and code.
Open source video
The trade theses in finance and failed experiments in pharma are proprietary and never published.
Open source video
Source video

Detecting LLM-judge drift from expert judgment to stylistic plausibility

Research question

Can adversarial probe sets of fluent-but-wrong and awkward-but-correct domain outputs reliably detect when an LLM judge has stopped tracking expert judgment and started tracking jargon fluency?

Why

It is a concrete, cheap diagnostic for the failure that silently invalidates automated eval in exactly the domains where eval is most expensive. Without it, a judge that scores higher with every prompt refinement may be inflating scores rather than tracking quality.

An LLM judge cannot tell alpha from jargon because it predicts the next probable word rather than understanding real value creation.
Open source video
The expert helps decompose problems, refine prompts, and judge outputs.
Open source video
Source video

Calibrating expert-authored rubrics: inter-expert agreement as an eval-prioritization signal

Research question

Do multi-expert disagreement rates on agent outputs predict which cases are worth turning into eval items, and does rubric validation across experts reduce the echo chamber the LLM judge was rejected for?

Why

Replacing an LLM judge with one expert simply relocates the echo chamber. Treating disagreement as a triage signal converts an unavoidable cost into the eval-set construction mechanism, and sets the throughput ceiling of the release cadence.

A domain expert who can curate data is the ultimate moat.
Open source video
The expert helps decompose problems, refine prompts, and judge outputs.
Open source video
Source video

Observability schemas as training-grade data capture

Research question

What trace and tool-call schema must be written at time-of-execution so that expert corrections and outcome linkage survive retrofitting, rather than being reconstructed from ops logs?

Why

It converts step 4 of the recipe from a debugging concern into the data source for error analysis, SFT, and rubric reward. Retrofitting a training pipeline onto operational logging is where the moat's raw material is most commonly lost.

Steps 1-4 are the easy part that fits on one screen; iteration and evaluation are where projects fail.
Open source video
Tools, prompts, run loops, and basic observability can be built by a single 10x engineer in minutes.
Open source video
Source video

Autonomy boundary as an output contract: candidate generator vs. decider

Research question

What capability threshold, if any, would justify inverting the loop back from AI-in-the-loop toward Human-in-the-Loop automation in a causal-reasoning domain?

Why

The boundary determines interface, latency budget, confidence reporting, and whether autonomous action is withheld by design or by limitation. Treating it as a capability question rather than a design choice risks shipping a decision contract backed by a correlational model.

Do not attempt to fully automate expert judgment in specialized verticals too early.
Open source video
Finance: flag when a bond looks mispriced
Open source video
Pharma: rank which molecules to test next
Open source video
Source video

Article ideas

4

Your LLM Judge Is a Style Critic: Why Jargon-Heavy Verticals Break Automated Evaluation

An LLM judge in finance or pharma does not degrade gracefully — it inverts, rewarding fluent plausibility and penalizing correct-but-unconventional outputs, so any eval loop built on it optimizes for jargon and calls it quality. The fix is not a better model but an expert-authored rubric plus a probe set that detects the inversion.

Angle

Show the failure mechanism (next-word probability as the scoring function), then give a concrete diagnostic: fluent-but-wrong and awkward-but-correct probe pairs, and the agreement curve against expert scores.

Source video

Nothing Fits on One Screen: The Moat Is the Reward Signal, Not the Agent Stack

Agent scaffolding (problem formulation, data identification, prompts, observability) has commoditized to a single engineer's afternoon, so competitive advantage has relocated entirely to who can construct a verifiable reward signal in a domain where outcomes are unpublished and delayed. Teams still budgeting for stack capability are funding the part that is already free.

Angle

Contrast the one-screen step 1-4 build against the persistent, human-staffed eval service that actually gates release cadence, and trace where the training signal physically lives.

Source video

Stop Automating the Expert: The AI-in-the-Loop Output Contract

In domains requiring causal rather than correlational reasoning, an agent that emits decisions is a liability; an agent that emits ranked candidate hypotheses is an asset. The autonomy boundary is an interface decision driven by what the model's verification story can support, not a maturity ladder to climb.

Angle

Reframe the HITL/AITL debate as an output-contract problem: decision contract vs. candidate contract, and how each changes UI, latency budget, confidence reporting, and which actions remain deliberately unavailable.

Source video

Own the Data You Will Never Publish

Unpublished trade theses and failed trial results are simultaneously the competitive moat and the reason base models cannot help — secrecy removes the signal from every public corpus, so the vertical AI must manufacture its own training data through first-party expert workflows before any model improvement can occur.

Angle

Trace the causal chain from secrecy to absence from training corpora to the requirement for first-party data-generation pipelines, and check the claim that models 'have never seen it' against registry and filing channels.

Source video

Project ideas

4

Verifier Coverage Audit

gatehouse

For most candidate vertical tasks, a partial programmatic verifier exists (backtests, registry-reported endpoints, rejection-rate proxies) and identifying it is a stronger predictor of project success than model or stack quality.

Proof of concept

For a set of narrow tasks (flag mispriced bond, rank molecules, triage a case), enumerate candidate verifiers and score each on automation level, latency to label, and confounder load; then correlate verifier score with whether an eval loop actually converges in a two-week trial.

Measurement

Fraction of tasks with at least one usable partial verifier, median label latency, and eval-score variance reduction per week of iteration for verified vs. unverified tasks.

Source video

Judge Inversion Probe Suite

beyond-evals

LLM-judge agreement with experts degrades measurably faster on fluent-but-wrong outputs than on awkward-but-correct outputs, and the gap is a reliable detector of judge drift before it corrupts an eval loop.

Proof of concept

Build paired probes per domain: stylistically confident nonsense vs. correct-but-ungrammatical expert output. Score with several LLM judges and with one or more domain experts; compute sensitivity and specificity of the judge for each probe class.

Measurement

Judge-expert agreement delta between probe classes, and change in that delta across successive judge prompt versions.

Source video

Expert-Hours Yield Instrumentation

movement-lab

Eval improvement per expert-hour follows a decaying curve that plateaus once the cheap proprietary signal is exhausted, and the plateau point is predictable from inter-expert disagreement rates.

Proof of concept

Instrument a vertical deployment to log expert minutes against each eval item authored, each agreement decision, and each resulting eval-score change; plot expert-hours per unit eval improvement and fit the curve.

Measurement

Expert-hours per unit of eval-score improvement over time, and correlation between plateau onset and inter-expert disagreement rate.

Source video

Training-Grade Trace Schema

new

A trace schema that captures expert corrections and outcome linkage at write time yields usable SFT and error-analysis data from the first deployment week, whereas retrofitting a training pipeline onto operational logging loses a substantial fraction of the signal.

Proof of concept

Run two parallel instrumentations of the same agent — a standard ops log and a training-grade schema capturing per-step expert adjudication, tool calls, and a deferred outcome pointer — then measure how much recoverable eval/SFT data each produces after four weeks.

Measurement

Recoverable training examples per week and fraction of expert corrections attributable to a specific agent step, per instrumentation.

Source video

Architectural implications

3

Observability is described as a step-4 build item, but the learning loop requires traces and tool calls to become training and error-analysis artifacts.

Before

Trace logging is an operational concern for debugging and incident response.

After

Traces, tool calls, and expert corrections are structured as the raw material for error analysis, SFT, and rubric-based reward.

Consequence

Logging schemas must capture expert judgment and outcome linkage at write time; retrofitting a training-grade data pipeline onto ops logging is expensive and often loses the signal.

Source video

The evaluation component is an expert-authored judge rather than an LLM judge in domains where quality is not text-recoverable.

Before

An LLM-as-judge rubric scales evaluation cheaply across the agent's outputs.

After

Domain experts author rubrics and adjudicate outputs; LLM judges are trusted only where their scores correlate with expert scores.

Consequence

Evaluation becomes a persistent human-staffed service with its own queue, throughput limits, and cost line, and it gates release cadence.

Source video

The agent boundary shifts from deciding to proposing when the domain requires causal rather than correlational reasoning.

Before

Agent emits a decision or action that is executed.

After

Agent emits ranked hypotheses or narrowed option sets that an expert consumes and decides on.

Consequence

Interfaces, latency budgets, and confidence reporting must be redesigned around candidate ranking; the autonomous action surface is deliberately withheld.

Source video

Tradeoffs and failure modes

4

Autonomous expert replacement vs. AI-in-the-loop assistance

Benefit

Full automation maximizes throughput and removes the human cost per decision.

Cost or risk

Models reason from correlation, not causation, so automating expert judgment too early produces confident but causally wrong decisions in high-stakes settings.

Do not attempt to fully automate expert judgment in specialized verticals too early.
Open source video
Source video

LLM-as-judge for scalable evaluation

Benefit

Cheap, fast, automated scoring of agent outputs.

Cost or risk

In jargon-heavy domains the judge rewards surface plausibility, creating an echo chamber and letting fluent nonsense pass as valid expert output.

An LLM judge cannot tell alpha from jargon because it predicts the next probable word rather than understanding real value creation.
Open source video
Source video

Proprietary data kept private as competitive advantage

Benefit

Unpublished trade theses and failed trial results preserve alpha and competitive position.

Cost or risk

The same secrecy removes the signal from any public corpus, so base models never learn it and the vertical AI must source the data through first-party expert workflows.

Proprietary data is the moat because base models have never seen it.
Open source video
Source video

Narrow task scoping for evaluability

Benefit

A scoped task like flagging a mispriced bond or ranking molecules is gradable by an expert and therefore iterable.

Cost or risk

A narrow scope limits initial product surface and perceived value, and forces a roadmap of incremental expansions rather than a general assistant.

You must pick a very narrow task rather than asking an agent to do everything.
Open source video
Source video

Open questions

4

How do you validate an expert-authored rubric or reward model so it does not simply encode one expert's biases and recreate the echo chamber it was meant to replace?

Why unresolved

The summary identifies experts as the judge but provides no mechanism for calibrating inter-expert agreement or drift.

Research direction

Measure agreement rates between multiple domain experts on agent outputs and treat disagreement as an eval-case prioritization signal.

Source video

When the ground truth arrives only months or years later (did the molecule work, did the trade make money), how should the learning loop be structured around delayed, sparse, and confounded rewards?

Why unresolved

The summary states the outcomes are proprietary and unverifiable, but does not describe how long-horizon outcomes are attributed back to the agent's proposal.

Research direction

Design credit-assignment and delayed-label pipelines that link agent proposals to realized outcomes with confounder tracking (market regime, other interventions).

Source video

How can a team detect that an LLM judge has shifted from tracking expert judgment to tracking stylistic plausibility?

Why unresolved

The summary asserts the failure mode exists but gives no diagnostic.

Research direction

Build adversarial probe sets of fluent-but-wrong and awkward-but-correct expert outputs and measure judge sensitivity to each.

Source video

What is the minimum expert involvement per unit of model improvement, and does the loop converge or plateau once the easy proprietary signal is exhausted?

Why unresolved

The summary claims 89% of enterprise agents never reach production for lack of training signal but does not characterize the yield curve of expert curation.

Research direction

Instrument expert-hours-per-eval-improvement as a first-class metric across a vertical deployment.

Source video

Key claims

7
causalVerification needed

Reinforcement learning via verifiable rewards works well only in domains like math and code, so models cannot self-verify in niche verticals.

Evidence

The model cannot verify itself in niche domains because reinforcement learning via verifiable rewards is only great at math and code.

Question

Which vertical domains have been shown to lack usable programmatic verifiers, and are there partial verifiers (e.g., backtests) that partially close the gap?

Source video
factualVerification needed

Trade theses at hedge funds and failed pharma experiments are proprietary and unpublished, so they are absent from model training corpora.

Evidence

The trade theses in finance and failed experiments in pharma are proprietary and never published.

Question

What fraction of failed trial results are actually unpublished versus reported through registries and filings?

Source video
causalVerification needed

89% of enterprise AI agents never reach production because they lack proprietary training signals for vertical accuracy.

Evidence

Explains why 89% of enterprise AI agents never reach production: they lack the proprietary training signals needed for vertical accuracy.

Question

What is the source and methodology of the 89% figure, and is the cause attributed to training-signal scarcity or to other production barriers?

Source video
causalVerification needed

An LLM judge cannot distinguish real alpha from jargon because it models next-word probability rather than value creation.

Evidence

An LLM judge cannot tell alpha from jargon because it predicts the next probable word rather than understanding real value creation.

Question

On a benchmark of expert-scored outputs in finance or pharma, how much does LLM-judge agreement with experts degrade relative to math/code domains?

Source video
causalVerification needed

An LLM's inability to reason from cause rather than correlation requires experts to do the primary work while AI assists.

Evidence

In high-stakes domains, experts do the primary work and AI assists by generating candidate hypotheses or narrowing options, because models reason from correlation, not causation.

Question

Does causal reasoning capability change this boundary, and at what capability threshold could the loop invert back toward automation?

Source video
comparativeVerification not requested

Steps 1-4 of building an agent (problem formulation, data identification, prompting, observability) are trivially buildable by one strong engineer; iteration and evaluation are where projects fail.

Evidence

Steps 1-4 are the easy part that fits on one screen; iteration and evaluation are where projects fail.

Source video
factualVerification needed

Pharma companies hide nearly a third of clinical trial results where trials failed, and hedge funds publishing long positions quarterly see lower returns.

Evidence

Pharma companies hiding nearly a third of clinical trial results where trials failed

Question

What are the cited studies or figures for the one-third unpublished-failed-trials estimate and the publication/return relationship?

Source video

Connections

5