AI Engineer · Published 2026-07-22

Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer — Emil Eifrem, Neo4j

privateReview: unreviewedConfidence: high
OntologySemantic layeragentsneo4jknowledge graphsemantic-layeragent-architectureknowledge-graphexecution-tracesenterprise-aiself-learning
Open on YouTube ↗

Summary

Overview

  • Speaker: Emil Eifrem
  • Channel: AI Engineer
  • Main topic: Ontology-based semantic layers for scalable agentic AI
  • Purpose: To explain why enterprise agentic AI architectures fail when relying on 'thick' agents and manual data wiring, and to propose an ontology-based semantic layer as the architectural solution. Emil Eifrem, Founder and CEO of Neo4j, presents a solution blueprint for scaling agentic AI in enterprise environments. By decoupling business logic and intent from hardcoded, manually wired data sources, organizations can use an ontology-based semantic layer consisting of business ontology, technical ontology, and execution traces. This approach allows agents to remain thin while leveraging a smarter, shared, and self-learning substrate.

Topic Map

The Enterprise Agent Problem

  • Explanation: When building AI agents for complex enterprise tasks (like account opening), developers typically wire business logic (intent, planning, looping) directly to hardcoded data sources inside the agent code.
  • Key claims:
    • Every agent builder has to find all data sources from scratch.
    • Violates DRY (Don't Repeat Yourself) principle across enterprise agents.
    • Lacks cross-agent learning and data source discovery.
  • Examples:
    • Account opening agent wired to motor vehicle records and passport verification APIs.
  • Terminology:
    • agentic AI
    • thick agents
    • business logic
    • data sources
  • Why it matters: As enterprises scale from single applications to hundreds of databases and agents, manual wiring becomes unmaintainable and repetitive.

Markdown Files and Vibe Coding Limitations

  • Explanation: Many engineering teams attempt to solve data discovery and context problems using markdown files and prompt engineering.
  • Key claims:
    • Markdown files are part of the solution but not the complete solution.
    • You cannot vibe code with just markdown files in enterprise databases.
  • Examples:
    • Reference to swyx on Latent Space podcast warning about relying solely on markdown files.
  • Terminology:
    • markdown files
    • vibe coding
    • prompt engineering
  • Why it matters: Enterprises have massive database heterogeneity (Snowflake, Databricks, Oracle, S3) that markdown cannot dynamically govern.

The Solution: Thin Agents on a Smarter Substrate

  • Explanation: To achieve agentic AI at scale, agents should be thin and sit on top of an ontology-based semantic layer.
  • Key claims:
    • Enterprise AI requires thin agents on a smarter shared substrate.
    • Three pillars of the ontology-based semantic layer solve discovery, trust, DRY, and learning.
  • Examples:
    • F20 Bank, global tech platforms, and leading fintechs implementing ontology layers.
  • Terminology:
    • thin agents
    • semantic layer
    • ontology-based semantic layer
  • Why it matters: Decoupling agents from data wiring allows organizations to add hundreds of agents without re-engineering database connections.

Key Points

Business-Facing Ontology

  • Explanation: A simple description of real-world business concepts and how they relate, articulated in human terms familiar to company employees.
  • Evidence: Using concepts like Customer and First Name rather than database column names like f_name.
  • Practical implication: Enables intuitive alignment between business intent and underlying data models.

Technical Ontology

  • Explanation: Metadata model of the data asset landscape representing where data lives, schemas, attributes, and physical locations across enterprise data systems.
  • Evidence: Mapping Oracle databases, Neo4j graphs, Snowflake warehouses, and S3 buckets.
  • Practical implication: Provides automated discoverability of technical data sources for agents.

Execution Traces

  • Explanation: Runtime signals capturing agent execution paths, decisions, outcomes, context, and errors to evaluate success and score paths.
  • Evidence: Recording whether an agent successfully resolved a government-issued ID via DMV lookup or passport verification.
  • Practical implication: Powers cross-agent self-learning and optimization over time.

Frameworks, Models & Processes

Ontology-Based Semantic Layer

  • How it works: Combines a business ontology, a technical ontology, and execution traces into a single governed layer that maps intent to data sources and learns from runtime execution.
  • Components:
    • Business-Facing Ontology
    • Technical Ontology
    • Execution Traces
  • When to use: When scaling multiple AI agents across a complex enterprise data landscape with heterogeneous databases.

Examples & Case Studies

An account opening agent needs to run compliance checks using government-issued IDs.

  • Illustrates: How business ontology connects to technical ontology (DMV registry and passport verification).
  • Lesson: Execution traces record which data source resulted in success, guiding future agent invocations.

Actionable Takeaways

  • Immediate:
    • Stop hardcoding data source connections directly inside agent business logic.
    • Explore Neo4j's startup program and solution engineering resources.
  • Strategic:
    • Adopt an ontology-based semantic layer to achieve enterprise agentic AI at scale.
    • Separate business-facing concepts from technical database schemas to adhere to DRY principles.
  • Questions to investigate:
    • How are business and technical ontologies initially populated and maintained?
    • How do execution traces automatically update agent scoring algorithms in production?

Claims Worth Verifying

  • In order to do agentic AI at scale, we need thin agents on a smarter shared substrate. (architectural thesis)
  • Ontology-based semantic layers solve discovery, trust, DRY, and cross-agent learning. (architectural benefit)

Notable Quotes

"To do agentic AI at scale, we need thin agents on a smarter shared substrate" (at 6:59) "Learn your databases. You can't vibe code with just markdown files." (at 5:55)

Compressed Summary

  • Thick agents with manually wired data sources do not scale in enterprise environments.
  • Ontology-based semantic layers provide a governed substrate for agentic AI.
  • The three pillars are business ontology, technical ontology, and execution traces.
  • Graphs in AI enable cross-agent learning and structured data discovery.
  • Keywords: ontology, semantic layer, agents, neo4j, knowledge graph
  • Core insight: Scaling enterprise AI agents requires decoupling business logic from data sources via an ontology-based semantic layer featuring business ontology, technical ontology, and execution traces.

Core insights

5
Architecturemedium noveltymoderate evidence

The ontology-based semantic layer is decomposed into three distinct pillars with separate responsibilities: a business-facing ontology (human concepts like Customer, First Name), a technical ontology (metadata model of where data physically lives across Oracle/Neo4j/Snowflake/S3), and execution traces (runtime path/outcome signals). This separates intent semantics from physical data mapping from empirical performance.

Why it matters

It gives a concrete responsibility split for the shared substrate, so teams can build and evolve each layer independently instead of collapsing business meaning and physical schema into agent code or a single monolithic metadata store.

Generalization

Separating business semantics, technical data mapping, and runtime telemetry into distinct layers applies to any agent platform needing to decouple intent from heterogeneous data sources.

Combines a business ontology, a technical ontology, and execution traces into a single governed layer that maps intent to data sources and learns from runtime execution.
Open source video
Using concepts like Customer and First Name rather than database column names like f_name.
Open source video
Mapping Oracle databases, Neo4j graphs, Snowflake warehouses, and S3 buckets.
Open source video
Mechanismhigh noveltymoderate evidence

Execution traces are the self-learning feedback loop: recording which data source resolved a task (e.g., DMV lookup vs. passport verification) and scoring execution paths so future agent invocations reuse winning routes. This turns runtime telemetry into a substrate-level learning signal rather than agent-local logging.

Why it matters

It provides a concrete mechanism for cross-agent learning — improvement accrues to the shared substrate, so a new agent benefits from outcomes recorded by prior agents without retraining or re-wiring.

Generalization

Recording and scoring execution paths at a shared layer converts operational traces into reusable routing/optimization knowledge across otherwise independent agents.

Execution traces record which data source resulted in success, guiding future agent invocations.
Open source video
Powers cross-agent self-learning and optimization over time.
Open source video
Failure Modemedium noveltymoderate evidence

Hardwiring data sources into agent business logic forces every agent builder to independently rediscover data sources, violating DRY and preventing cross-agent discovery/learning. The failure is architectural, not a prompt-quality issue.

Why it matters

Identifies that agent sprawl compounds redundant data-discovery work and blocks shared learning, so per-agent optimization cannot fix a substrate-level defect.

Generalization

Any per-agent duplicated integration logic (not just data sources) will not scale and should be externalized to a shared, discoverable layer.

Every agent builder has to find all data sources from scratch.
Open source video
Violates DRY (Don't Repeat Yourself) principle across enterprise agents.
Open source video
Lacks cross-agent learning and data source discovery.
Open source video
Tradeoffmedium noveltymoderate evidence

Markdown files and vibe coding are explicitly insufficient for enterprise data governance: markdown is 'part of the solution but not the complete solution' because enterprise database heterogeneity cannot be dynamically governed by static text artifacts.

Why it matters

Directly challenges the popular markdown-as-context practice, warning engineers not to over-rely on it as the context/discovery layer for heterogeneous enterprise data estates.

Generalization

Static, hand-maintained context artifacts degrade as the number and heterogeneity of backing systems grows; a governed metadata layer is needed for dynamic discovery.

Markdown files are part of the solution but not the complete solution.
Open source video
You cannot vibe code with just markdown files in enterprise databases.
Open source video
Learn your databases. You can't vibe code with just markdown files.
Open source video
Mental Modelmedium noveltymoderate evidence

The core architectural thesis is 'thin agents on a smarter shared substrate': moving logic/context/learning out of the agent and into a governed shared layer is what allows adding hundreds of agents without re-engineering database connections.

Why it matters

It reframes the agent as a thin client over a knowledge-rich substrate, shifting investment from individual agent sophistication to the shared layer — a different build/buy allocation.

Generalization

Scaling agentic systems favors investing in shared substrate intelligence over making individual agents fatter.

To do agentic AI at scale, we need thin agents on a smarter shared substrate
Open source video
Enterprise AI requires thin agents on a smarter shared substrate.
Open source video

Deep dives

5

Execution traces as a substrate-level learning loop

Research question

Can centrally recorded, scored execution traces (which data source resolved a task) measurably improve routing decisions for agents that never generated those traces, without reinforcing early bad routes?

Why

If traces can serve as a cross-agent learning signal, improvement accrues to the shared substrate rather than to individual agents, changing where teams invest (shared layer vs. per-agent tuning) and making every new agent cheaper to onboard. If they cannot, the 'self-learning substrate' claim collapses into ordinary per-agent logging.

Execution traces record which data source resulted in success, guiding future agent invocations.
Open source video
Powers cross-agent self-learning and optimization over time.
Open source video
Source video

Separating business ontology, technical ontology, and execution traces

Research question

What is the minimal responsibility boundary between business semantics, physical data mapping, and runtime telemetry that lets each layer evolve independently without breaking agent intent resolution?

Why

The three-pillar split is the concrete design artifact of the talk; getting the seams right determines whether ontologies can be owned by different teams (business analysts vs. data engineers vs. platform/SRE) and whether schema drift in one pillar propagates into the others.

Combines a business ontology, a technical ontology, and execution traces into a single governed layer that maps intent to data sources and learns from runtime execution.
Open source video
Using concepts like Customer and First Name rather than database column names like f_name.
Open source video
Mapping Oracle databases, Neo4j graphs, Snowflake warehouses, and S3 buckets.
Open source video
Source video

Latency and cost of ontology-mediated resolution vs. hardwired data access

Research question

What is the runtime overhead per agent invocation of resolving business concepts through the ontology layer compared with hardcoded data-source wiring, and at what fleet size does the maintenance saving dominate that overhead?

Why

The talk asserts discovery, trust, DRY, and learning benefits but presents no performance data. Without a latency/cost baseline, adopting a semantic layer is a bet against an unmeasured regression on the critical path of every agent call.

Enterprise AI requires thin agents on a smarter shared substrate.
Open source video
Decoupling agents from data wiring allows organizations to add hundreds of agents without re-engineering database connections.
Open source video
Source video

Bootstrapping and drift maintenance of enterprise ontologies

Research question

Can the business and technical ontologies be populated from schema introspection plus targeted human review, and how does the layer detect and handle drift when business concepts or physical schemas change?

Why

The architecture is only as good as its freshness; an ontology that silently diverges from the actual warehouses reintroduces the exact trust failure the semantic layer was meant to eliminate, and this is the least specified part of the proposal.

As enterprises scale from single applications to hundreds of databases and agents, manual wiring becomes unmaintainable and repetitive.
Open source video
Ontology-based semantic layers solve discovery, trust, DRY, and cross-agent learning.
Open source video
Source video

Where markdown-as-context stops working

Research question

At what data-source count and heterogeneity does hand-maintained markdown context degrade enough that discovery accuracy and freshness fall below a governed metadata layer's baseline?

Why

Markdown-plus-prompting is the current default practice for agent context, so finding an empirical degradation threshold converts a rhetorical warning into an engineering decision rule for when to build the semantic layer.

Markdown files are part of the solution but not the complete solution.
Open source video
You cannot vibe code with just markdown files in enterprise databases.
Open source video
Learn your databases. You can't vibe code with just markdown files.
Open source video
Source video

Article ideas

3

Thin Agents, Fat Substrate: Why Per-Agent Optimization Cannot Fix an Architectural Defect

Hardwiring data sources into agent business logic is a substrate-level DRY violation, not a prompt-quality problem, so no amount of per-agent refinement (better prompts, better markdown, better tool descriptions) removes the redundant data-discovery work or enables cross-agent learning; the only durable fix is externalizing intent-to-data resolution to a shared governed layer.

Angle

Reframe the common 'make the agent smarter' instinct as misallocated investment, and show the accounting: cost per new agent falls only when discovery moves out of the agent.

Source video

Your Execution Traces Are a Routing Table, Not a Log File

Runtime telemetry becomes an asset only when it is recorded at a shared layer, attributed to the specific data source that resolved the task, and fed back into source selection for future invocations; observability semantics (store everything, query later) leave that value on the table and allow per-agent learning silos to persist.

Angle

Argue for a design change: execution traces should be modeled as first-class substrate state that affects routing, with explicit safeguards against reinforcing earlier bad routes.

Source video

Three Pillars, Three Owners: Assigning Business, Technical, and Trace Ontologies to Different Teams

The value of an ontology-based semantic layer depends less on picking a graph database than on drawing clean responsibility seams between business semantics, physical data mapping, and runtime telemetry so each can evolve independently; collapsing them into one monolithic metadata store recreates the coupling the architecture was meant to remove.

Angle

Operational and organizational reading of the three-pillar split, with an emphasis on who owns drift in each pillar and how change propagates across seams.

Source video

Project ideas

3

Thin-vs-Thick Resolution Benchmark

beyond-evals

For a fixed task set against a heterogeneous data estate, a thin agent resolving sources through a business+technical ontology matches hardwired thick-agent correctness while reducing per-new-agent integration effort, without exceeding an acceptable per-invocation latency budget.

Proof of concept

Implement the same enterprise task (e.g., account opening with DMV and passport verification sources) twice: once with data sources hardcoded in agent logic, once with a thin agent over an ontology layer backed by a graph store; stub three heterogeneous backends (SQL warehouse, graph, object store) behind both.

Measurement

Task success rate and answer correctness; p50/p95 resolution latency and token/tool-call cost per invocation; lines of integration code and time-to-add a fifth data source or a second agent.

Source video

Trace-Scored Source Router

movement-lab

Scoring execution traces by outcome and using them to rank candidate data sources improves task success or cost versus static source selection, and does so for a newly added agent that contributed none of the traces.

Proof of concept

Record per-invocation traces mapping task type to resolving data source and success/cost outcome in a shared store; run a path-scoring router for a new agent against a static-selection baseline on the same task distribution, with guards against low-sample routes.

Measurement

Delta in task success rate and cost per task for the new agent (zero-shot transfer of traces); rate at which the router converges on the winning source vs. a human-curated mapping; regression rate from early bad-route reinforcement.

Source video

Ontology Drift Detector

gatehouse

Automated schema introspection plus a small reviewed mapping set can keep the technical ontology current enough that concept-to-physical resolution accuracy stays above a fixed threshold as source schemas change, outperforming hand-maintained markdown context of comparable authoring effort.

Proof of concept

Bootstrap the technical ontology from introspected schemas of several heterogeneous sources with human review of a sample; inject schema changes (renamed column f_name to first_name, new warehouse) and compare resolution accuracy and staleness against a markdown-plus-prompting context baseline.

Measurement

Concept-resolution accuracy over time; detection latency for injected schema changes; refresh effort in human minutes per change; accuracy crossover point as source count/heterogeneity grows.

Source video

Architectural implications

2

Business logic and data-source wiring are currently co-located inside agent code.

Before

Each agent embeds intent, planning, looping, and hardcoded data-source connections.

After

Agents express intent against a shared ontology layer that resolves data sources dynamically.

Consequence

New agents can be added without re-implementing data discovery, and the substrate becomes the locus of improvement and governance.

Source video

Cross-agent learning requires a shared, persistent substrate rather than per-agent state.

Before

Runtime outcomes are logged locally per agent and do not inform other agents.

After

Execution traces are recorded centrally and used to score paths and guide future invocations.

Consequence

Operational telemetry becomes an optimization asset for the whole fleet, not just observability for one agent.

Source video

Tradeoffs and failure modes

2

Markdown-based context vs. governed semantic layer

Benefit

Markdown files and prompt engineering are convenient and are part of the solution.

Cost or risk

They cannot dynamically govern heterogeneous enterprise databases, so relying on them fails at scale.

You cannot vibe code with just markdown files in enterprise databases.
Open source video
Source video

Thick agents vs. thin agents on a shared substrate

Benefit

Thick agents keep logic and data access local, simpler for a single task.

Cost or risk

Manual data wiring duplicates discovery work across agents, violates DRY, and blocks cross-agent learning as the fleet grows.

Violates DRY (Don't Repeat Yourself) principle across enterprise agents.
Open source video
Source video

Open questions

3

How are the business and technical ontologies initially populated and maintained over time?

Why unresolved

The summary states the layer's components and benefits but gives no mechanism for bootstrapping or keeping ontologies current as schemas and business concepts change.

Research direction

Prototype automated ontology population/curation from schema introspection plus human review, and measure drift handling.

Source video

How do execution traces automatically update agent scoring algorithms in production?

Why unresolved

The summary claims traces power self-learning and path scoring but does not specify the update mechanism, frequency, or safeguards.

Research direction

Define and benchmark a path-scoring update loop, including how it avoids reinforcing earlier bad routes.

Source video

What is the runtime cost and latency of resolving business concepts to physical data sources through the ontology layer versus hardcoded wiring?

Why unresolved

The claimed benefits (discovery, trust, DRY, learning) are asserted, but no performance or cost data is presented.

Research direction

Build a benchmark comparing thin-agent-over-ontology against thick-agent hardwired resolution on latency, cost, and correctness.

Source video

Key claims

5
predictionVerification needed

Enterprise agentic AI requires thin agents on a smarter shared substrate to scale.

Evidence

To do agentic AI at scale, we need thin agents on a smarter shared substrate

Question

Does a thin-agent-over-ontology architecture empirically scale to hundreds of agents and data sources better than thick agents, measured by maintenance cost and reliability?

Source video
causalVerification needed

An ontology-based semantic layer solves discovery, trust, DRY, and cross-agent learning.

Evidence

Ontology-based semantic layers solve discovery, trust, DRY, and cross-agent learning.

Question

Which of these four benefits are measurably delivered, and under what enterprise conditions do they fail?

Source video
causalVerification needed

Hardcoding data sources inside agent logic violates DRY across enterprise agents and blocks cross-agent learning.

Evidence

Violates DRY (Don't Repeat Yourself) principle across enterprise agents.

Question

How much redundant data-discovery effort is actually eliminated per new agent when a shared layer is introduced?

Source video
opinionVerification needed

Markdown files alone cannot govern data discovery in heterogeneous enterprise databases.

Evidence

You cannot vibe code with just markdown files in enterprise databases.

Question

At what data-source count or heterogeneity does markdown-based context degrade enough to require a governed metadata layer?

Source video
causalVerification needed

Execution traces record which data source succeeded, guiding future agent invocations.

Evidence

Execution traces record which data source resulted in success, guiding future agent invocations.

Question

Do trace-based routing decisions improve task success or cost relative to a static source selection baseline?

Source video

Connections

3