Demystifying Agentic AI Using Small Language Models [PyCon DE & PyData 2026]

Overview

• Speaker: Serhii Sokolenko
• Channel: PyData
• Main topic: Agentic AI Using Small Language Models
• Purpose: To educate developers and data scientists on how to build, run, and evaluate agentic AI applications locally and cost-effectively using Small Language Models. The presentation demystifies agentic AI by demonstrating how small language models (SLMs), such as Salesforce xLAM-2 (32B), can effectively replace large language models for agentic workflows and tool-calling tasks at a lower cost, with enhanced privacy, and on local consumer hardware. The talk covers the evolution of LLMs into decisioning engines, the core capabilities required for agentic workflows, a comparison between SLMs and LLMs using benchmark leaderboards (like Berkeley Function Calling Leaderboard), and practical setup and deployment using local runtimes like llama.cpp and Tower.

Topic Map

Introduction & Motivation: The Agentic AI Hype

• Explanation: Introduction of the speaker, company Tower.dev, and the current hype around agentic AI (salesforce Agentforce, Postman, Satya Nadella quotes).
• Key claims:
  • AI agents are becoming the primary way of interacting with computers.
  • Developers need to 'get their hands dirty' and test ideas on laptops using small language models.
• Examples:
  • Historical Gold Rush vs. AI Agent Gold Rush
• Terminology:
  • Agentic AI
  • Small Language Models (SLMs)
• Why it matters: Sets the stage for moving beyond LLM hype into practical local experimentation.

How LLMs Became Decisioning Engines

• Explanation: The progression from GPT-3 era training data containing reasoning-like patterns to Chain-of-Thought prompting and intermediate reasoning steps.
• Key claims:
  • LLMs and SLMs contain reasoning patterns from their training data.
  • Chain-of-thought prompting injects intermediate reasoning steps into the model's generation loop.
• Examples:
  • Stack Overflow, code tutorials, forum answers
• Terminology:
  • Chain-of-Thought Prompting
  • Decisioning Engines
  • Reasoning Loops
• Why it matters: Explains the underlying mechanism that transforms a next-token predictor into an autonomous agent.

Small vs. Large Language Models for Agentic Workflows

• Explanation: Comparing SLMs and LLMs on parameter count, memory requirements, strengths, and weaknesses.
• Key claims:
  • LLMs are an overkill for agentic reasoning because they contain irrelevant knowledge and expensive inference.
  • SLMs (<30B parameters) fit into consumer-grade GPU memory, offering better privacy, lower cost, and local experimentation capabilities.
• Examples:
  • Using a sledgehammer to crack a walnut
• Terminology:
  • SLM
  • LLM
  • Quantization
  • Inference Cost
• Why it matters: Justifies the use of smaller models specifically for agentic tasks.

Important Capabilities for Agents & Benchmark Evaluation

• Explanation: Detailed breakdown of the 4 key capabilities required for agentic workflows (reasoning, tool calling, code generation, instruction following) and review of Berkeley Function Calling Leaderboard (BFCL).
• Key claims:
  • SLMs of size ~32B are now as accurate on agentic tasks as LLMs 10x their size.
  • Open-source SLMs like Salesforce xLAM-2 (32B) rank competitively with proprietary models on function calling.
• Examples:
  • Berkeley Function Calling Leaderboard V4
• Terminology:
  • Tool Calling
  • Parameter Passing
  • Function Calling (FC)
  • BFCL
• Why it matters: Provides empirical proof from benchmarks that SLMs are viable for production agentic workflows.

Experimentation Setup & Demo

• Explanation: Walkthrough of the exact technical stack needed for local and cloud experimentation: SLMs, local inference servers, Python runtimes, and agentic frameworks.
• Key claims:
  • Local experimentation is feasible using llama.cpp with 4-bit quantization on Apple Silicon.
  • Tower runtime allows easy packaging, local testing, and cloud deployment of Python-native data pipelines and agents.
• Examples:
  • Building a stock price data agent using Claude Code, LangChain, Apache Iceberg, and llama.cpp
• Terminology:
  • llama.cpp
  • Tower
  • LangChain
  • Apache Iceberg
  • Model Context Protocol (MCP)
• Why it matters: Gives developers a concrete blueprint to start building and deploying agents immediately.

Key Points

LLMs are an overkill for agentic reasoning

• Explanation: Large language models contain excessive compressed world knowledge that is irrelevant for specific tool-calling and reasoning loops, making inference slow and expensive.
• Evidence: Comparison of general knowledge vs. task-specific agent requirements.
• Practical implication: Switching to SLMs drastically reduces cloud infrastructure costs and data privacy risks.

Small models can match large model performance on agentic tasks

• Explanation: Recent benchmarks like BFCL show that open-source 32B SLMs achieve accuracy comparable to 100B+ proprietary models when evaluated specifically on tool calling and reasoning.
• Evidence: Berkeley Function Calling Leaderboard rankings showing Salesforce xLAM-2 (32B) near top proprietary models.
• Practical implication: Developers can run high-performing agentic apps locally without depending on costly external APIs.

Business rules replace rigid data pipelines for agents

• Explanation: Agentic workflows replace traditional Airflow DAGs with natural language business rules and guardrails, allowing agents to dynamically choose tools based on context.
• Evidence: Prompt examples contrasting deterministic DAGs with conditional rules (e.g., check database cache before calling external API).
• Practical implication: More flexible data ingestion and processing systems that handle free-form requests.

Frameworks, Models & Processes

Agentic Reasoning & Tool Calling Loop

• How it works: The user prompt goes to the LLM/SLM, which enters a reasoning loop to break down the task, selects appropriate tools (APIs, databases), executes them, and continues until a final answer token is produced.
• Components:
  • Prompt
  • Reasoning Loop
  • Tools (APIs, Analytical/Operational DBs)
  • Final Answer
  • Memory
• When to use: When building autonomous agents that need to interact with external systems to answer complex queries.

Local Agent Experimentation Stack

• How it works: A modular stack combining an open-source quantized SLM, a local inference server, a Python runtime, and an agent framework.
• Components:
  • SLM (Salesforce xLAM-2 32B)
  • Local inference server (llama.cpp)
  • Python runtime service (Tower)
  • Agentic Framework (LangChain)
• When to use: For local development, testing, and production deployment of cost-effective agents.

Examples & Case Studies

New York lawyers submitted a legal brief generated entirely by ChatGPT containing hallucinated case citations.

• Illustrates: The danger of using LLMs without access to real structured business data and proper grounding.
• Lesson: Agents need access to reliable data sources (like Apache Iceberg lakehouses) and cached business data rather than relying solely on parametric model memory.

Building a stock price data agent using Claude Code and Tower, querying Yahoo Finance or Apache Iceberg lakehouse based on business rules.

• Illustrates: The practical creation and deployment of an agentic app using Python, LangChain, and MCP.
• Lesson: Agents can make intelligent decisions on whether to fetch data from local caches or external APIs based on explicit system prompt instructions.

Actionable Takeaways

• Immediate:
  • Install llama.cpp and test serving an SLM like Salesforce xLAM-2 locally.
  • Use LangChain for writing agentic workflows with tool calling.
  • Incorporate business rules and guardrails instead of rigid DAGs.
• Strategic:
  • Leverage Small Language Models for agentic workloads to cut inference costs and ensure data privacy.
  • Pair inference engines with data lakehouses like Apache Iceberg to prevent hallucinations and ground agent decisions in real business data.
  • Transition from traditional ETL pipelines to agentic data workflows.
• Questions to investigate:
  • How do infinite looping issues in agentic reasoning get mitigated effectively?
  • What are the latency tradeoffs between local quantized SLMs and cloud-hosted large models in high-concurrency production environments?

Claims Worth Verifying

• SLMs with ~32B parameters and 4-bit quantization can fit into 16GB-32GB RAM/VRAM on consumer hardware. (technical hardware claim)
• Salesforce xLAM-2 (32B) scores comparably to GPT-4 on tool calling and function calling benchmarks. (benchmark performance claim)

Notable Quotes

"AI agents will become the primary way we interact with computers in the future. They will be able to understand our needs and preferences, and proactively help us with tasks and decision making." "More human than human is our motto." "An entity that perceives its environment, makes decisions, and takes actions to achieve goals." "I'm gonna have to science the shit out of this."

Compressed Summary

• SLMs (<30B parameters) match LLM performance on agentic workflows and tool calling while being vastly cheaper and runnable locally.
• Chain-of-thought prompting turned language models into decisioning engines by incorporating intermediate reasoning steps.
• Effective agentic workflows require strong reasoning, tool calling, code generation, and instruction following capabilities.
• Pairing SLMs with local inference servers (llama.cpp) and data lakehouses (Apache Iceberg) enables privacy, cost reduction, and factual grounding.
• Keywords: agentic ai, small language models, function calling, llama.cpp, apache iceberg, langchain
• Core insight: Small language models are sufficiently powerful, flexible, and economical to replace large models for agentic workflows and tool-calling applications.