Graham Neubig · Published 2026-09-08

CMU AI Agents 2026: 2. Tool Use for Language Model Agents

privateReview: unreviewedConfidence: high
tool useagentsCodeActconstrained decodingMCPtool-useconstrained-decodingorchestrationjson-schema
Open on YouTube ↗

Summary

Overview

  • Speaker: Graham Neubig
  • Channel: Graham Neubig
  • Main topic: Tool Use and API integration for Language Model Agents
  • Purpose: To educate students and researchers on how language model agents interface with external programs, APIs, and tools, detailing the mechanics, execution, constraints, and evaluation methods. This lecture covers the fundamentals of tool use for language model agents, exploring how models turn token predictions into actions. Graham Neubig discusses tool definitions, benefits of tools (extending capabilities and facilitating computation), scenarios for tool use, programmatic tool calling (CodeAct), mechanics of providing tools (tokens, chat templates, tool definitions), constrained decoding (JSON schema and push-down automata), REST API calling, Model Context Protocol (MCP), orchestration of multiple calls (serial vs. parallel), and evaluation methodologies like the Berkeley Function Calling Leaderboard.

Topic Map

Basic Idea and Definition of Tools

  • Explanation: Defines what a tool is for a language model agent and why they are necessary to bridge the gap between token-predicting language models and external computer programs.
  • Key claims:
    • A tool is an interface through which a language model can invoke an external computer program.
    • Language models cannot perform tool invocation on their own because they are fundamentally token-predicting machines.
  • Examples:
    • Search, calculator, browser, Python, arbitrary APIs.
  • Terminology:
    • Tool
    • Language Model Agent
    • Token-predicting machine
  • Why it matters: Establishes the foundational distinction between static language models and active agents.

Benefits of Tools

  • Explanation: Outlines the core benefits of giving language models access to tools: extending capabilities beyond model parameters and facilitating operations that models could do but less efficiently.
  • Key claims:
    • Extending allows access to information or actions unavailable in model parameters (e.g., getting current time).
    • Facilitating allows delegating external operations that the model can do in text but much more slowly or unreliably (e.g., multi-digit multiplication).
  • Examples:
    • Getting current time via API; using a calculator for arithmetic.
  • Terminology:
    • Extend
    • Facilitate
    • Action space
  • Why it matters: Explains why tool use is necessary for real-world agent utility.

Tool-Use Scenarios

  • Explanation: Categorizes the practical needs for tool use into current information, exact computation, private state, and external action.
  • Key claims:
    • Call tools only when their benefit outweighs latency, cost, failure, and risk.
  • Examples:
    • Search for current information; calculator for exact computation; account API for private state; browser for external action.
  • Terminology:
    • Current information
    • Exact computation
    • Private state
    • External action
  • Why it matters: Helps developers determine when and what tools to integrate.

Examples of Tools

  • Explanation: Reviews specific examples of tools commonly integrated into language model agents such as ChatGPT.
  • Key claims:
    • Text-in/text-out models have evolved into all-encompassing applications via tool integration.
  • Examples:
    • Textual responses (finish), information retrieval (search_web), code execution (execute_code), image generation (generate_image), custom functions (create_grocery_cart).
  • Terminology:
    • Textual response
    • Information retrieval
    • Code execution
    • Image generation
  • Why it matters: Provides concrete architectural patterns for tool implementations.

Programmatic Tool Calling and CodeAct

  • Explanation: Discusses using code as a meta-tool, enabling control flow, variables, and libraries in one action.
  • Key claims:
    • Code acts as a meta-tool that improves success rates and reduces interaction turns.
    • Code is expressive but introduces challenges with broader action spaces, harder constraints, and higher impact/security risks.
  • Examples:
    • Using Python and pandas for data analysis tasks.
  • Terminology:
    • CodeAct
    • Meta-tool
    • Programmatic tool calling
    • Sandbox
  • Why it matters: Demonstrates how code execution significantly enhances agent capability and efficiency over rigid single-function calls.

Mechanics of Providing Tools

  • Explanation: Explains how tool calls are structured as tokens, serialized into prompt inputs, and dispatched by execution harnesses.
  • Key claims:
    • Tool calls are expressed as tokens within model outputs.
    • Different models use different tool call protocols and serialization formats.
  • Examples:
    • Qwen (function + parameter tags), Mistral Small 4 (control token + call ID), DeepSeek V3 (DSML invocation block).
  • Terminology:
    • Tool call tokens
    • Chat template
    • Tool definition
    • Call ID
  • Why it matters: Crucial for understanding how different LLM providers handle tool calling under the hood.

Constrained Decoding

  • Explanation: Covers how to ensure language models generate valid tool calls using JSON schema, context-free grammars, push-down automata, and token masking.
  • Key claims:
    • Language models are not guaranteed to generate well-formed tool calls without constraints.
    • Token masking based on grammars ensures 100% syntactic and schema validity during decoding.
  • Examples:
    • Using JSON schema to validate types, required fields, and enum values; XGrammar and push-down automata for token masking.
  • Terminology:
    • Constrained decoding
    • JSON schema
    • Context-free grammar
    • Push-down automaton
    • Token masking
  • Why it matters: Prevents malformed outputs and runtime parsing errors in production agent systems.

REST API Calling and Model Context Protocol (MCP)

  • Explanation: Examines how to expose functions over HTTP using FastAPI, publish OpenAPI descriptions, and connect hosts to servers via the Model Context Protocol (MCP).
  • Key claims:
    • REST APIs use OpenAPI descriptions for schema sharing.
    • MCP provides a standardized protocol for connecting AI hosts to local and remote client servers with separate credentials.
  • Examples:
    • FastAPI weather endpoint with API key authentication; FastMCP server wrapping OpenAPI.
  • Terminology:
    • REST API
    • OpenAPI
    • FastAPI
    • Model Context Protocol (MCP)
    • FastMCP
  • Why it matters: Provides industry standards for securely exposing external capabilities to agents.

Orchestrating Multiple Calls: Serial vs. Parallel

  • Explanation: Compares serial tool execution with parallel execution using asyncio.gather, highlighting dependencies and efficiency.
  • Key claims:
    • Parallel tool calls can significantly reduce task latency when calls are independent.
    • Calls with data dependencies must be sequenced.
  • Examples:
    • Executing weather, calendar, and flight lookups in parallel versus finding a customer ID before fetching orders.
  • Terminology:
    • Serial tool calling
    • Parallel tool calling
    • Asyncio.gather
    • Dependency
  • Why it matters: Directly impacts agent speed and cost efficiency.

Evaluating Tool Use

  • Explanation: Explores evaluation metrics and leaderboards for assessing language model tool use capabilities across selection, arguments, trajectory, and task levels.
  • Key claims:
    • Evaluation requires measuring benchmark accuracy across tool selection, arguments, and trajectory, alongside operational metrics like latency, reliability, and safety.
    • Provider error rates vary significantly based on inference engine quantization and decoding implementations.
  • Examples:
    • Berkeley Function Calling Leaderboard; OpenRouter tool-call error rate analytics.
  • Terminology:
    • Berkeley Function Calling Leaderboard
    • Tool selection
    • Trajectory
    • Operational failure
    • Quantization
  • Why it matters: Provides a rigorous framework for benchmarking and selecting LLM providers for agent workflows.

Key Points

Tools Extend and Facilitate LLM Capabilities

  • Explanation: Tools allow agents to access external data/actions and perform complex computations more reliably and quickly.
  • Evidence: Distinction between fetching live time/weather vs. performing multi-digit multiplication.
  • Practical implication: Developers should use tools for real-time data retrieval and precise computation tasks rather than relying purely on model weights.

Code as a Meta-Tool (CodeAct)

  • Explanation: Allowing agents to write and execute code provides control flow, variables, and library access in a single unified action.
  • Evidence: CodeAct research shows higher success rates and fewer interaction turns.
  • Practical implication: Code execution interfaces make powerful general-purpose agent environments but require strict sandboxing for security.

Constrained Decoding Guarantees Syntax

  • Explanation: Using JSON schemas and grammar-based token masking prevents models from generating malformed tool calls.
  • Evidence: Push-down automata and XGrammar mask invalid logits during inference.
  • Practical implication: Always implement constrained decoding to eliminate parsing failures in production applications.

Model Context Protocol (MCP) Secures Agent Tool Integration

  • Explanation: MCP standardizes discovery and invocation while maintaining separate credentials between agent hosts and upstream servers.
  • Evidence: FastMCP and static token verifiers in FastAPI integration.
  • Practical implication: Use MCP to safely connect AI applications to external data sources without exposing master API keys to the model.

Parallel Tool Execution Reduces Latency

  • Explanation: Independent tool calls can be executed concurrently using asynchronous gathering to dramatically speed up agent workflows.
  • Evidence: Comparing serial round-trip costs (~6.4 seconds) with parallel execution (~2.8 seconds).
  • Practical implication: Design agent execution harnesses to batch independent tool calls whenever possible.

Frameworks, Models & Processes

CodeAct

  • How it works: Treats executable code (e.g., Python) as the primary action space, allowing the model to compose functions, control loops, and use external libraries.
  • Components:
    • Built-in functions
    • External libraries (pandas)
    • Utility functions
  • When to use: When agents need to perform complex multi-step data manipulation, calculations, or programmatic workflows.

Model Context Protocol (MCP)

  • How it works: Connects an AI application host to client servers (local or remote) using a standardized protocol for discovering tools at runtime and handling secure authentication.
  • Components:
    • MCP Host
    • Clients
    • Local/Remote servers
  • When to use: When building secure, modular enterprise agent architectures that integrate multiple third-party services.

Grammar-Based Constrained Decoding (XGrammar)

  • How it works: Maps a JSON schema to a context-free grammar parsed by a push-down automaton, masking model output logits at each step to allow only valid tokens.
  • Components:
    • Finite control
    • Stack memory
    • Token mask cache
  • When to use: In production inference engines where 100% valid schema compliance is mandatory.

Examples & Case Studies

An agent needs to find the mean of 18, 23, 41, and 54.

  • Illustrates: Exact computation via code execution tool vs. mental calculation.
  • Lesson: Delegating math to Python execution ensures 100% reliability.

An agent needs weather, calendar, and flight information.

  • Illustrates: Parallel vs. serial tool calling performance.
  • Lesson: Independent tool calls should be executed concurrently to minimize latency.

An agent receives malformed JSON missing a closing bracket from a model output.

  • Illustrates: The danger of unconstrained decoding in tool use.
  • Lesson: Harnesses fail if decoding is not constrained by grammars or schemas.

Actionable Takeaways

  • Immediate:
    • Implement constrained decoding (JSON schema / grammar masking) for all production tool-use agents.
    • Use asynchronous execution (asyncio.gather) for independent tool calls to reduce latency.
    • Leverage FastAPI and FastMCP to expose safe, credential-separated tools to language models.
  • Strategic:
    • View code as a meta-tool (CodeAct) to maximize agent expressiveness and efficiency.
    • Evaluate agents across the full stack: selection, arguments, trajectory, and operational metrics (latency, cost, reliability, safety).
    • Be aware that provider-level quantization and decoding differences dramatically impact tool-call error rates.
  • Questions to investigate:
    • How do different LLM quantization levels (FP8 vs FP4) affect tool-calling robustness?
    • What are the best sandboxing practices for executing arbitrary code generated by language model agents?
    • How can we effectively measure semantic correctness beyond syntactic schema validation?

Claims Worth Verifying

  • CodeAct achieves higher success rates and fewer interaction turns even on tasks that didn't traditionally require code. (Research finding)
  • FP8 quantization models get significantly fewer mistaken tool calls than FP4 models. (Empirical observation)

Notable Quotes

"A tool is an interface through which a language model can invoke an external computer program." "Call tools only when their benefit outweighs latency, cost, failure, and risk." "Code is kind of a special... it's one of the special things that... it's different from other tools in the way that it like opens up your ability to call tools in new ways."

Compressed Summary

  • Tools extend LLM capabilities and facilitate efficient computation.
  • CodeAct uses code execution as a powerful meta-tool for agents.
  • Constrained decoding with JSON schemas and grammars guarantees valid tool calls.
  • Model Context Protocol (MCP) standardizes secure tool discovery and invocation.
  • Parallel tool execution drastically reduces agent latency for independent tasks.
  • Keywords: tool use, agents, codeact, constrained decoding, mcp
  • Core insight: Language model agent tool use relies on a structured, layered stack spanning capability design, token-level protocol serialization, grammar-constrained decoding, secure interface protocols, and multi-call orchestration.

Core insights

6
Mechanismhigh noveltystrong evidence

Tool-call well-formedness can be guaranteed at the decoding layer rather than the prompt layer: grammar-derived token masking (JSON schema, context-free grammars, push-down automata) enforces schema and syntactic validity during generation instead of relying on the model's compliance plus post-hoc parsing and retries.

Why it matters

Moves the correctness boundary for tool invocation out of prompt engineering and output validation/retry loops and into the inference stack, which changes where you invest engineering effort (serving/decoding config) and removes an entire class of production failure.

Generalization

Whenever a model must emit structured artifacts consumed by a deterministic system, the guarantee belongs to the decoder, not the prompt.

Language models are not guaranteed to generate well-formed tool calls without constraints.
Open source video
Token masking based on grammars ensures 100% syntactic and schema validity during decoding.
Open source video
Tradeoffhigh noveltymoderate evidence

Code is treated as a meta-tool: a single programmatic action (e.g., Python with pandas) can carry control flow, variables, and library calls, improving success rates and reducing interaction turns, but it simultaneously widens the action space, makes constraints harder to enforce, and raises impact/security risk.

Why it matters

Frames the central architectural fork in agent tooling: many narrow typed functions with tight, constrainable schemas versus one general executor with high capability but a much harder sandbox and blast-radius problem.

Generalization

Increasing expressiveness of a single agent action increases capability per turn but shifts verification burden from schema check to sandbox and resource containment.

Code acts as a meta-tool that improves success rates and reduces interaction turns.
Open source video
Code is expressive but introduces challenges with broader action spaces, harder constraints, and higher impact/security risks.
Open source video
Architecturehigh noveltymoderate evidence

Tool-call protocols are model-family specific — Qwen uses function and parameter tags, Mistral Small 4 a control token plus call ID, DeepSeek V3 a DSML invocation block — so tool calls are just tokens embedded in the model output stream rather than a uniform API surface.

Why it matters

Any harness supporting multiple models needs a per-model serialization/parsing adapter paired with the chat template, and swapping models is a protocol migration, not a config change; it also explains parsing brittleness when templates mismatch.

Generalization

Treat the model's tool-call format as a wire protocol with vendor-specific dialects, and isolate it behind an adapter layer.

Tool calls are expressed as tokens within model outputs.
Open source video
Different models use different tool call protocols and serialization formats.
Open source video
Practicemedium noveltymoderate evidence

Orchestration is a data-dependency scheduling problem: independent calls can be dispatched concurrently (e.g., asyncio.gather) for large latency wins, while calls with data dependencies must be strictly sequenced.

Why it matters

Latency and cost of an agent turn are dominated by tool round-trips; distinguishing independent from dependent calls is a concrete, implementable optimization with a correctness constraint attached.

Generalization

Agent execution loops should build a dependency graph over tool calls and schedule maximally parallel waves, rather than iterating strictly turn-by-turn.

Parallel tool calls can significantly reduce task latency when calls are independent.
Open source video
Calls with data dependencies must be sequenced.
Open source video
Mental Modelmedium noveltymoderate evidence

Tool portfolios should be justified by two distinct value sources: extending the model (information/actions absent from parameters, e.g., current time) versus facilitating (operations the model can do in text but far slower or less reliably, e.g., multi-digit multiplication) — with the decision rule that a tool is only worth calling when its benefit outweighs latency, cost, failure, and risk.

Why it matters

Gives a principled filter for tool selection and for when to skip a call entirely, rather than adding tools opportunistically; the presence of failure and risk in the cost calculus implies every tool adds a new failure surface.

Generalization

Adding capability via external calls always trades deterministic in-model computation for network/execution overhead and new failure modes.

Extending allows access to information or actions unavailable in model parameters (e.g., getting current time).
Open source video
Facilitating allows delegating external operations that the model can do in text but much more slowly or unreliably (e.g., multi-digit multiplication).
Open source video
Call tools only when their benefit outweighs latency, cost, failure, and risk.
Open source video
Architecturemedium noveltymoderate evidence

MCP is positioned as a standardized host-to-server connection protocol that spans local and remote servers with separately scoped credentials, layered on top of REST/OpenAPI-style schema exposure (e.g., FastAPI + OpenAPI, FastMCP wrapping OpenAPI).

Why it matters

Standardizing the host-client-server boundary with per-server credentials decouples tool providers from agent hosts and localizes the security boundary, replacing bespoke per-integration auth and discovery code.

Generalization

Tool integration is converging on a protocol boundary analogous to a package/driver interface, with credential isolation as a first-class design element.

MCP provides a standardized protocol for connecting AI hosts to local and remote client servers with separate credentials.
Open source video
REST APIs use OpenAPI descriptions for schema sharing.
Open source video

Deep dives

4

Constrained decoding vs. end-task accuracy for tool calls

Research question

Does grammar-constrained decoding improve end-task success beyond syntactic and schema validity, or does it constrain semantic flexibility and argument quality compared with unconstrained generation plus retries?

Why

Teams are moving schema enforcement into the inference stack, but if constrained decoding only guarantees parse validity while harming task success, the engineering tradeoff changes; this determines whether decoding-time constraints are a net win or an incomplete fix.

Language models are not guaranteed to generate well-formed tool calls without constraints.
Open source video
Token masking based on grammars ensures 100% syntactic and schema validity during decoding.
Open source video
Source video

Code-as-tool capability vs. sandbox containment tradeoff

Research question

Under what task distributions does code-as-tool outperform typed function calling on success rate and turn count, and what sandbox containment profile preserves most of that advantage while bounding security and resource risk?

Why

The architectural fork between many narrow tools and one general executor is a safety decision as much as a capability decision; without measured tradeoffs, teams cannot choose a defensible containment model or justify CodeAct adoption.

Code acts as a meta-tool that improves success rates and reduces interaction turns.
Open source video
Code is expressive but introduces challenges with broader action spaces, harder constraints, and higher impact/security risks.
Open source video
Source video

Per-model tool-call protocol adapters and portability

Research question

How can a single agent harness abstract per-model tool-call protocols while preserving constrained decoding guarantees, and what test suites are needed to detect silent parsing failures across model versions?

Why

If tool-call formats are vendor-specific token dialects, then model swaps become protocol migrations; without adapters and protocol-level tests, multi-model agents accumulate brittle parsing code and hidden failure modes.

Tool calls are expressed as tokens within model outputs.
Open source video
Different models use different tool call protocols and serialization formats.
Open source video
Source video

Dependency-aware scheduling of parallel tool calls

Research question

Can static argument-reference analysis or planner-declared dependencies reliably infer tool-call dependencies to schedule maximally parallel waves without correctness loss in realistic agent workflows?

Why

Latency gains from parallel tool calls are only safe where dependencies permit; if dependency inference is accurate, agent loops become workflow engines, but mis-inference can silently corrupt results.

Parallel tool calls can significantly reduce task latency when calls are independent.
Open source video
Calls with data dependencies must be sequenced.
Open source video
Source video

Article ideas

4

The Decoder Is the Contract: Why Tool-Call Correctness Belongs in the Inference Stack

Prompt engineering and retry loops are the wrong place to guarantee tool-call structure; grammar-constrained decoding collapses schema validity into the serving layer, but it also couples deployment to schema versioning and shifts the remaining failure surface to semantic mismatches between valid arguments and intended actions.

Angle

Argue that schema enforcement is an inference-layer concern, then map the resulting organizational consequences: schema-as-decoding-artifact, version coupling between serving and tools, and a reduced but altered error budget.

Source video

Code as a Meta-Tool Is a Security Architecture Decision

Adopting CodeAct is not merely a capability upgrade; it replaces schema-bounded safety with sandbox containment as the primary control, forcing teams to design filesystem, network, resource, and credential boundaries before they can responsibly ship expressive code execution.

Angle

Frame the CodeAct vs. typed-function choice as a trade of verification burden from static schema check to dynamic sandbox and resource containment, with concrete containment dimensions as the real design surface.

Source video

Your Agent Harness Has a Wire Protocol Problem

Tool-call formats are vendor-specific token dialects, not a uniform API, so multi-model agents require protocol adapters and protocol-level test suites; treating tool calls as portable leads to brittle parsing and silent failures that only appear after model swaps.

Angle

Reframe tool-call serialization as a wire protocol with dialects, then derive the adapter layer and test strategy needed to keep harnesses model-agnostic.

Source video

Parallel Tool Calls Are a Scheduling Problem, Not a Prompt Trick

Latency wins from concurrent tool execution require dependency-aware scheduling, not just a prompt asking for parallel calls; without an explicit call graph, parallelism is unsafe, and with one, the agent loop becomes a workflow engine with partial-failure semantics.

Angle

Show that the serial/parallel decision is a data-dependency scheduling problem, then outline how static argument-reference analysis can derive call graphs and what correctness and failure-handling constraints follow.

Source video

Project ideas

4

Grammar-Constrained Decoding Accuracy Harness

beyond-evals

Enforcing tool-call schemas via grammar-constrained decoding during generation increases end-task success rates and reduces retry count compared with unconstrained generation plus JSON repair, without degrading semantic argument quality.

Proof of concept

Implement a constrained decoder using outlines or equivalent for a set of JSON-schema tools, then run a tool-calling benchmark with and without decoding-time constraints on the same model and tasks.

Measurement

Parse validity rate, end-task accuracy, average retries per tool call, and human- or rubric-based argument semantic correctness.

Source video

CodeAct Sandbox Containment Tradeoff Lab

gatehouse

For data-analysis and retrieval tasks, code-as-tool with a strict sandbox achieves higher task success per turn than typed function calling, but the advantage shrinks as containment tightens; there exists a containment profile that retains most benefit with bounded risk.

Proof of concept

Build a CodeAct agent with configurable sandbox policies (filesystem, network, CPU/memory, credential scoping) and run matched tasks from pure computation to external data fetching across policy levels.

Measurement

Task success rate, turns per task, containment strictness score, attempted violation rate, and capability loss versus unrestricted execution.

Source video

Model-Protocol Adapter Test Suite

new

A shared schema plus per-model protocol adapters for Qwen, Mistral, and DeepSeek tool-call formats can maintain high parse validity across model swaps, but cross-model portability of constrained decoding requires adapter-specific grammar handling; without adapters, failure rates spike on model swaps.

Proof of concept

Implement adapters for three model families and run identical tool-calling suites, measuring parsing and schema violations under shared constraints versus no adapter layer.

Measurement

Parse error rate, schema violation rate, latency overhead of adapter, and failure-rate delta after model swap.

Source video

Dependency-Aware Parallel Tool Scheduler

new

Static argument-reference analysis over tool-call plans can correctly identify independent calls to parallelize, reducing agent wall-clock latency by more than 30% on read-heavy workflows while preserving correctness; mis-inferred dependencies are rare and can be handled by conservative sequencing.

Proof of concept

Build a scheduler that parses tool-call plans, constructs a dependency graph from argument references, and dispatches independent calls via asyncio.gather while sequencing dependent ones.

Measurement

Wall-clock latency reduction versus serial baseline, number of incorrectly parallelized calls, task success rate, and partial-failure handling behavior.

Source video

Architectural implications

4

Schema validity is enforced during decoding via token masking from grammars and push-down automata rather than by prompting and post-hoc JSON validation.

Before

Prompt instructions plus output parsing, validation, and retry loops to repair malformed tool calls.

After

Grammar/schema-constrained decoding integrated into the serving stack, with schemas as a decoding-time artifact.

Consequence

Retry logic and error-handling code shrink, but the inference/serving layer becomes coupled to tool schemas and must be versioned with them.

Source video

Code execution acts as a single meta-tool replacing many narrow function tools.

Before

A large catalog of typed single-purpose tools, each individually described and schema-constrained.

After

A sandboxed code interpreter as one tool, with expressive multi-step programs executed in a single turn.

Consequence

The critical engineering boundary moves from tool-schema design to sandboxing, resource limits, and auditability, since the action space is no longer bounded by a fixed schema.

Source video

Tool-call formats are model-family specific and expressed as tokens within the output stream.

Before

A portable, provider-agnostic tool-calling interface assumed interchangeable across models.

After

A per-model protocol adapter (chat template plus parser for tags, control tokens, or invocation blocks) sitting between the model and the execution harness.

Consequence

Model swaps become integration projects; teams need protocol-level test suites and model-specific harness code.

Source video

Multiple tool calls are scheduled as an orchestration problem with dependencies rather than a strictly sequential loop.

Before

Turn-by-turn serial execution, waiting for each tool result before deciding the next call.

After

Dependency-aware scheduling that dispatches independent calls concurrently (asyncio.gather) and sequences only dependent ones.

Consequence

Agent latency drops substantially for read-heavy workflows, but the harness must model call graphs and handle partial failures of concurrent calls.

Source video

Tradeoffs and failure modes

4

Code as a meta-tool (CodeAct) versus typed function calls

Benefit

Higher task success rates and fewer interaction turns through control flow, variables, and libraries in one action.

Cost or risk

Broader action space, harder-to-enforce constraints, and higher impact/security risk requiring sandboxing.

Code is expressive but introduces challenges with broader action spaces, harder constraints, and higher impact/security risks.
Open source video
Source video

Adding a tool to an agent's action space

Benefit

Extends capability beyond model parameters and offloads operations the model performs slowly or unreliably.

Cost or risk

Latency, cost, potential failure, and risk that must be outweighed by the tool's benefit before calling.

Call tools only when their benefit outweighs latency, cost, failure, and risk.
Open source video
Source video

Unconstrained model-generated tool calls

Benefit

Natural, flexible generation without decoding-time machinery.

Cost or risk

No guarantee of well-formed tool calls, producing syntax or schema errors that surface as runtime parsing failures.

Language models are not guaranteed to generate well-formed tool calls without constraints.
Open source video
Source video

Parallel tool execution

Benefit

Significantly reduced task latency when calls are independent.

Cost or risk

Incorrect or impossible when calls have data dependencies, which must be sequenced.

Parallel tool calls can significantly reduce task latency when calls are independent.
Open source video
Source video

Open questions

5

Does grammar-constrained decoding change model task accuracy, not just syntactic validity, compared with unconstrained generation plus retries?

Why unresolved

The summary asserts 100% syntactic and schema validity from token masking but gives no accuracy comparison or evaluation against unconstrained decoding.

Research direction

Run matched tool-calling benchmarks with and without constrained decoding and measure end-task success, not only parse validity.

Source video

How should tool use be evaluated — what metrics capture correct tool selection, argument fidelity, and outcome quality?

Why unresolved

The evaluation section of the topic map is truncated and only names the Berkeley Function Calling Leaderboard as an example.

Research direction

Build a taxonomy of tool-use metrics (selection, argument correctness, sequencing, outcome) and validate them against the leaderboard-style benchmarks.

Source video

What containment model is adequate for code-as-tool when the action space is arbitrarily wide?

Why unresolved

The summary flags broader action space and higher impact/security risk for code execution but does not describe sandboxing mechanisms or limits.

Research direction

Prototype sandbox profiles (filesystem, network, resource, credential scoping) and measure capability loss versus containment strength.

Source video

Can an agent harness automatically infer the dependency graph among tool calls to decide serial versus parallel execution?

Why unresolved

The summary states the serial/parallel rule but does not describe how dependencies are detected or represented at runtime.

Research direction

Experiment with static argument-reference analysis or planner-declared dependencies to derive call graphs and measure latency and correctness.

Source video

How do per-model tool-call protocol differences interact with constrained decoding and chat templates in practice?

Why unresolved

The summary shows model-specific formats (Qwen, Mistral, DeepSeek) but does not address whether constraints work uniformly across them.

Research direction

Implement per-model adapters with shared schema constraints and test cross-model portability and failure rates.

Source video

Key claims

7
factualVerification needed

Language models cannot invoke tools on their own because they are fundamentally token-predicting machines, so a tool is an external interface the model can only call by emitting tokens.

Evidence

Language models cannot perform tool invocation on their own because they are fundamentally token-predicting machines.

Question

Is tool invocation ever achievable without an external execution harness mediating the model's emitted tokens?

Source video
causalVerification needed

Grammar-based token masking in constrained decoding yields 100% syntactic and schema validity of tool calls.

Evidence

Token masking based on grammars ensures 100% syntactic and schema validity during decoding.

Question

Does token masking achieve full schema validity across all supported schemas, including nested and conditional constraints?

Source video
comparativeVerification needed

Using code as a meta-tool improves agent success rates and reduces the number of interaction turns, at the cost of a broader action space, harder constraints, and higher security risk.

Evidence

Code acts as a meta-tool that improves success rates and reduces interaction turns.

Question

Under what task distributions and sandbox configurations does code-as-tool outperform typed function calling on success rate and turn count?

Source video
factualVerification needed

Different models use different tool-call protocols and serialization formats, so tool calls are tokens embedded in model outputs.

Evidence

Different models use different tool call protocols and serialization formats.

Question

How many distinct tool-call serialization protocols exist across current model families, and how stable are they across versions?

Source video
causalVerification needed

Parallel tool calls significantly reduce task latency when calls are independent, while calls with data dependencies must be sequenced.

Evidence

Parallel tool calls can significantly reduce task latency when calls are independent.

Question

What latency reduction is achievable in realistic multi-tool agent workflows once dependency constraints are applied?

Source video
factualVerification needed

MCP standardizes connections between AI hosts and local or remote client servers with separate credentials, while REST APIs share schemas via OpenAPI descriptions.

Evidence

MCP provides a standardized protocol for connecting AI hosts to local and remote client servers with separate credentials.

Question

Does per-server credential separation in MCP meaningfully reduce credential-leak blast radius compared with bespoke API-key integrations?

Source video
opinionVerification not requested

A tool should only be called when its benefit outweighs latency, cost, failure, and risk.

Evidence

Call tools only when their benefit outweighs latency, cost, failure, and risk.

Source video

Connections

5