Databricks · Published 2026-09-17

Genie Agents Tutorial: Build & Customize AI Data Agents

privateReview: unreviewedConfidence: high
genie agentsdatabricksunity catalogai data agentsNatural language queryinggenie-agentsunity-catalogcontext-groundingsemantic-layertext-to-sqlagent-observabilityinstruction-tuning
Open on YouTube ↗

Summary

Overview

  • Speaker: Hanlin Sun, Product Management, Databricks
  • Channel: Databricks
  • Main topic: Genie Agents in Databricks
  • Purpose: To demonstrate how data analysts and domain experts can build, customize, tune, and monitor Genie Agents to empower sales and business teams with natural language data querying. Hanlin Sun from Databricks Product Management introduces Genie Agents: domain-specific assistants that enable users to answer questions about their data using natural language. The video covers agent creation, connecting data sources from the Unity Catalog, configuring general instructions and curated examples for join relationships, using Genie Code to automatically improve instructions, and monitoring agent usage via the activity digest.

Topic Map

Introduction to Genie Agents

  • Explanation: Genie agents are domain-specific assistants defined and curated by domain experts to help users answer data questions using natural language.
  • Key claims:
    • Genie agents are domain-specific assistants helping users answer questions via natural language.
    • They are defined and curated by domain experts to ensure specific behavioral manners.
  • Examples:
    • An analyst building sales assets and creating an agent for the sales team.
  • Terminology:
    • Genie Agents
    • Unity Catalog
    • domain-specific assistants
  • Why it matters: Bridges the gap between raw data tables and non-technical business users by leveraging expert-curated context.

Creating and Querying a Genie Agent

  • Explanation: Demonstrates creating a new Genie agent by selecting tables from Unity Catalog and asking natural language questions like open sales opportunities in APAC.
  • Key claims:
    • Genie agents inherit context from Unity Catalog including table descriptions and join relationships.
    • Previous queries and dashboards are automatically loaded into the agent context.
  • Examples:
    • Asking 'show me all open sales opportunities in APAC' and getting a breakdown by stage.
  • Terminology:
    • Unity Catalog
    • open sales opportunities
    • pipeline breakdown
  • Why it matters: Enables rapid deployment of data agents directly connected to trusted enterprise datasets.

Customizing Instructions and Examples

  • Explanation: How domain experts can tune agent behavior using General Instructions and Examples to enforce rules like including sales rep names.
  • Key claims:
    • General instructions allow authors to guide agent output format and terminology.
    • Examples allow defining granular join relationships and query behaviors.
  • Examples:
    • Adding an instruction to always include the sales rep name when asking for performance breakdowns.
  • Terminology:
    • General Instructions
    • Examples tab
    • join relationships
  • Why it matters: Ensures business logic and domain terminology are correctly reflected in agent responses.

Using Genie Code for Continuous Improvement

  • Explanation: Genie Code assists authors in editing instructions and improving agent knowledge based on user interactions and ambiguities.
  • Key claims:
    • Genie Code reads existing agent context to suggest new instructions.
    • Helps refine definitions such as customer churn rate.
  • Examples:
    • Adding an instruction for churned customers using Genie Code suggestions.
  • Terminology:
    • Genie Code
    • churn rate
    • instructions
  • Why it matters: Streamlines the iterative tuning of agents based on real-world edge cases.

Monitoring Agent Usage

  • Explanation: The Monitoring tab provides an activity digest showing past questions, user ratings, and conversation details to further refine the agent.
  • Key claims:
    • Monitoring tab tracks historical questions, user ratings, and conversation logs.
    • Authors can inspect individual queries and results to improve agent performance over time.
  • Examples:
    • Reviewing questions asked by users in the last 30 days in the activity digest.
  • Terminology:
    • Activity digest
    • Monitoring tab
    • conversation type
  • Why it matters: Provides visibility into how end users interact with the agent, enabling continuous quality improvement.

Key Points

Domain-Specific Assistance

  • Explanation: Genie agents combine LLMs with domain expert curation and Unity Catalog metadata.
  • Evidence: Inherits table descriptions, join relationships, and prior query context from Unity Catalog.
  • Practical implication: Reduces hallucinations and ensures answers align with organizational definitions.

Custom Instructions & Genie Code

  • Explanation: Agents can be precisely tuned via text instructions and AI assistance.
  • Evidence: Genie Code suggests instructions based on user intent and ambiguities.
  • Practical implication: Makes agent maintenance scalable and collaborative.

Monitoring & Governance

  • Explanation: Comprehensive monitoring tools track usage and feedback.
  • Evidence: Activity digest shows questions, unique users, feedback, and logs.
  • Practical implication: Allows data teams to measure adoption and audit agent accuracy.

Examples & Case Studies

An analyst created a sales forecasting agent and asked for open opportunities in APAC.

  • Illustrates: Natural language to data querying and pipeline breakdown by stage.
  • Lesson: Genie agents automatically map open opportunities across discovery, demo, validation, and procurement stages.

An analyst added a general instruction to always include sales rep names in performance breakdowns.

  • Illustrates: Customizing agent instruction behavior.
  • Lesson: Agents respect domain instructions to return human-readable names instead of raw IDs.

Actionable Takeaways

  • Immediate:
    • Create a Genie agent from Unity Catalog tables for your business domain.
    • Use General Instructions to enforce terminology and formatting rules.
    • Leverage Genie Code to automatically draft instructions for ambiguous definitions.
  • Strategic:
    • Empower business teams with natural language access to governed data assets.
    • Establish a continuous improvement loop using the monitoring activity digest.
  • Questions to investigate:
    • How do join relationships affect complex multi-table queries?
    • What metrics should be monitored in the activity digest to assess agent ROI?

Claims Worth Verifying

  • Genie agents inherit context from Unity Catalog including table descriptions and join relationships. (technical)

Notable Quotes

"Genie agents are domain-specific assistants that can help your users answer questions on top of their data by simply asking in natural language."

Compressed Summary

  • Genie agents are natural language domain-specific data assistants.
  • Built on Unity Catalog with automatic context inheritance.
  • Supports general instructions and curated examples for tuning.
  • Genie Code assists in editing instructions based on usage.
  • Monitoring tab tracks historical queries, ratings, and user feedback.
  • Keywords: genie agents, databricks, unity catalog, ai data agents, natural language querying
  • Core insight: Genie agents combine Unity Catalog governance with natural language interfaces and expert-curated instructions to make enterprise data accessible and accurate for business teams.

Core insights

5
Architecturehigh noveltymoderate evidence

Genie agents derive their semantic layer automatically from Unity Catalog metadata: table descriptions and join relationships are inherited, and prior queries and dashboards are auto-loaded into agent context. The grounding substrate is therefore the governed catalog plus historical query artifacts, not hand-written prompts or a separate BI semantic model.

Why it matters

Shifts where domain semantics live: instead of duplicating a semantic model inside the agent, the agent consumes existing governance metadata and prior analytics artifacts as context. This changes the work of deploying a data agent from prompt engineering to catalog curation and query-history hygiene.

Generalization

Any enterprise agent over structured data can treat the existing metadata/governance layer (descriptions, declared relationships, historical queries) as its context source, making catalog quality the primary determinant of agent accuracy.

Genie agents inherit context from Unity Catalog including table descriptions and join relationships.
Open source video
Previous queries and dashboards are automatically loaded into the agent context.
Open source video
Mechanismmedium noveltymoderate evidence

Agent tuning is split across two distinct abstraction levels: General Instructions for global output format and terminology policy, and Examples for granular join relationships and per-question query behavior. Behavioral policy and query-level exemplars are separate, separately editable artifacts.

Why it matters

Gives a concrete decomposition for configuring a data agent: durable global rules versus case-specific demonstrations. This is the difference between a policy layer that should be stable and a test-case layer that grows with observed failures, which affects maintainability and review workflows.

Generalization

Instruction-following agents generally benefit from separating global behavioral policy from example-driven, case-level guidance, because the two have different change frequencies and different review/ownership needs.

General instructions allow authors to guide agent output format and terminology.
Open source video
Examples allow defining granular join relationships and query behaviors.
Open source video
Mechanismhigh noveltymoderate evidence

A second AI component, Genie Code, reads the existing agent context and existing instructions and proposes new instructions, targeting ambiguities surfaced by real user interactions (e.g., refining a definition such as customer churn rate). Instruction authoring is performed by an agent rather than by a human writing prompts from scratch.

Why it matters

This is a meta-agent pattern: the maintenance surface of an agent (its instruction set) is itself maintained by an LLM grounded in the agent's current context and observed traffic. It makes spec drift correction cheaper, but also means the agent's specification is co-authored by a model.

Generalization

As instructed agents accumulate, the bottleneck moves from 'writing the prompt' to 'keeping the instruction set consistent with observed failures' — a task well suited to a context-aware authoring agent operating over production traces.

Genie Code reads existing agent context to suggest new instructions.
Open source video
Helps refine definitions such as customer churn rate.
Open source video
Practicemedium noveltymoderate evidence

The Monitoring tab's activity digest (historical questions, user ratings, conversation details, unique users) is positioned as the input to the next tuning cycle, closing a loop from production usage back into the agent's instructions.

Why it matters

Treats production interaction logs and explicit user feedback as a first-class, structured input to agent specification changes rather than as passive dashboards. This is the operational loop that makes an agent improve instead of silently degrading.

Generalization

Agent reliability engineering requires a defined path from observed conversation traces and ratings back into a change in the agent's configuration or instructions; observability without a write-back path to the spec is insufficient.

Monitoring tab tracks historical questions, user ratings, and conversation logs.
Open source video
Allows data teams to measure adoption and audit agent accuracy.
Open source video
Tradeoffmedium noveltyweak evidence

The mechanism for handling domain ambiguity is natural-language instruction authoring rather than executable/validated logic: ambiguous business definitions (e.g., 'churned customers', 'churn rate') are resolved by writing instructions, and the claimed benefit is reduced hallucination and alignment with organizational definitions.

Why it matters

Free-text instructions become the correctness substrate for business logic. That is fast and collaborative but offers no type checking, test harness, or deterministic verification described in the summary — so correctness depends on curation discipline and post-hoc monitoring.

Generalization

Expressing business semantics as prose instructions trades verifiability for iteration speed; systems that do this need compensating mechanisms (exemplar tests, monitored feedback, or a coded semantic layer) to avoid silent semantic errors.

Reduces hallucinations and ensures answers align with organizational definitions.
Open source video
Examples allow defining granular join relationships and query behaviors.
Open source video

Deep dives

4

Catalog metadata grounding contribution to text-to-SQL accuracy

Research question

Do inherited table descriptions and join relationships measurably reduce hallucination on multi-table questions, or do they mainly improve fluency on simple single-table questions?

Why

This determines whether catalog curation is load-bearing for agent reliability or merely cosmetic, and it gives a falsifiable basis for prioritizing governance work.

Genie agents inherit context from Unity Catalog including table descriptions and join relationships.
Open source video
Reduces hallucinations and ensures answers align with organizational definitions.
Open source video
Source video

Stale-context contamination from auto-loaded query history and dashboards

Research question

Does auto-loading prior queries and dashboards act as useful few-shot grounding or as a source of stale-context contamination, and does filtering by recency/ownership improve answer quality?

Why

Auto-loaded context can silently steer answers toward superseded or idiosyncratic artifacts, and the summary provides no validation or filtering step.

Previous queries and dashboards are automatically loaded into the agent context.
Open source video
Source video

Instruction-set scalability and conflict management for model-suggested instructions

Research question

At what instruction-set size does model-suggested instruction authoring become counterproductive through conflicts, redundancy, or drift from intent?

Why

If Genie Code can grow instructions faster than review capacity, the agent's specification can self-corrupt, making consolidation and conflict detection necessary.

Genie Code reads existing agent context to suggest new instructions.
Open source video
Helps refine definitions such as customer churn rate.
Open source video
Source video

Diagnostic metrics for agent ROI and accuracy beyond question counts and ratings

Research question

Which activity-digest metrics (repeat-question rate, rating trend, escalation/abandonment, unique-user retention) correlate with expert-reviewed answer correctness?

Why

Without diagnostic metrics, monitoring measures activity rather than quality, and tuning may optimize for the wrong signals.

Monitoring tab tracks historical questions, user ratings, and conversation logs.
Open source video
Allows data teams to measure adoption and audit agent accuracy.
Open source video
Source video

Article ideas

4

Catalog Quality Is Agent Reliability

For data agents over governed tables, accuracy is bounded by catalog metadata quality, so catalog curation—descriptions, declared joins, shared query hygiene—is a reliability engineering task, not prompt engineering.

Angle

Reframe data agent deployment as governance-dependent reliability, with a testable plan for measuring metadata contribution.

Source video

The Two-Layer Spec: Separating Agent Policy from Test Cases

Durable global instructions and per-query examples should be maintained as separate artifacts because they change at different rates and require different review, preventing policy drift and example sprawl.

Angle

Engineering decomposition for tuning instructed agents, with concrete ownership and change-management implications.

Source video

Who Writes the Spec? Model-Assisted Instruction Authoring and Its Review Burden

When a meta-agent suggests instructions from production traces, the bottleneck moves from prompt writing to reviewing, deduplicating, and regression-checking accumulated instructions; without conflict detection, quality degrades as the spec grows.

Angle

Analyze the shift from human-authored to model-assisted specifications and the governance it requires.

Source video

Observability Without Write-Back Is Just Dashboards

Monitoring activity digests only improve agents if they have a defined path into instruction changes; passive telemetry without a write-back path to the spec cannot prevent silent degradation.

Angle

Operational loop design for agent reliability, using the Genie activity digest as a case study.

Source video

Project ideas

4

Catalog Grounding Ablation Harness

beyond-evals

Inherited join relationships and table descriptions measurably reduce incorrect multi-table answers compared with no metadata or deliberately wrong metadata.

Proof of concept

Build a benchmark of multi-table questions across three conditions: metadata present, metadata absent, and metadata wrong; compare generated SQL/results to gold SQL/result sets.

Measurement

Answer correctness rate, hallucination/incorrect-join rate, and difference between conditions on single-table vs multi-table questions.

Source video

Stale-Context Filter for Auto-Loaded Query History

gatehouse

Filtering prior queries and dashboards by recency and ownership before ingestion improves answer quality compared with unfiltered auto-loading.

Proof of concept

Ablation where prior query history is filtered by recency/ownership before entering agent context, compared to unfiltered ingestion on the same question set.

Measurement

Expert-judged answer correctness, user rating, and rate of answers citing superseded artifacts.

Source video

Instruction Conflict Detector for Genie Code

new

As an instruction set grows via model-suggested additions, contradiction and redundancy rates increase, and a conflict-detection/consolidation step reduces them without losing coverage.

Proof of concept

Track instruction growth, contradiction rate, and redundancy over iterative Genie Code sessions; add automated conflict detection and consolidation, then remeasure.

Measurement

Contradiction rate, redundancy ratio, instruction count, and regression in answer quality after consolidation.

Source video

Activity Digest to Spec Write-Back Pipeline

movement-lab

A defined write-back path from activity-digest metrics (repeat-question rate, rating trend, abandonment) to instruction changes improves repeat-question rate and rating trend compared with passive dashboard review.

Proof of concept

Instrument the activity digest, derive candidate instruction changes from low-rated/repeat questions, and route them through review into the agent spec.

Measurement

Repeat-question rate, rating trend, abandonment rate, and time from observed failure to spec change.

Source video

Architectural implications

3

Semantic context for the agent is sourced from catalog metadata and prior query/dashboard artifacts rather than authored separately.

Before

Domain semantics are encoded in a dedicated BI semantic model or hand-written prompt preamble owned by the analytics team.

After

The agent reads descriptions, declared join relationships, and historical queries directly from the governance catalog as its context.

Consequence

Data governance work (describing tables, declaring relationships, curating shared queries) becomes load-bearing for agent accuracy, and catalog quality becomes an agent reliability concern.

Source video

Instruction maintenance is delegated to a model-assisted authoring loop over existing context and observed ambiguities.

Before

A human author reads failure reports and manually rewrites prompt instructions.

After

A secondary model proposes instruction edits by reading the agent's current context and user interactions.

Consequence

Instruction sets can grow faster than human review capacity, raising the need for review, deduplication, and regression checking of instructions as they accumulate.

Source video

Agent quality management is organized around an explicit usage-and-feedback loop (activity digest) tied to authoring.

Before

Agent quality is assessed ad hoc or only at build time.

After

Questions, ratings, and conversation logs from production are routinely reviewed and drive instruction changes.

Consequence

Agent configuration becomes a continuously edited artifact rather than a one-time deployment, requiring versioning and change-tracking of instructions alongside the data.

Source video

Tradeoffs and failure modes

4

Automatic context inheritance from prior queries and dashboards

Benefit

Rapid deployment: the agent is productive without manually authoring join logic or examples, because prior queries and dashboards are loaded into context.

Cost or risk

Inherited artifacts are treated as valid context without any stated validation step; stale, superseded, or idiosyncratic prior queries could steer answers in ways authors did not intend.

Previous queries and dashboards are automatically loaded into the agent context.
Open source video
Source video

Free-text instructions as the locus of business logic

Benefit

Domain experts can tune terminology, formatting, and definitions directly, and Genie Code makes this maintenance scalable and collaborative.

Cost or risk

Business rules expressed as prose have no described verification mechanism; correctness relies on expert curation and is only audited after the fact through monitoring.

Makes agent maintenance scalable and collaborative.
Open source video
Source video

Model-suggested instructions (Genie Code)

Benefit

Shortens the loop from observed user ambiguity to a concrete instruction, including refining definitions like churn rate.

Cost or risk

Suggested instructions are proposed by a model reading current context, which risks self-reinforcing or contradictory instructions if suggestions are accepted without review.

Genie Code assists authors in editing instructions and improving agent knowledge based on user interactions and ambiguities.
Open source video
Source video

Trusting catalog metadata as grounding

Benefit

Claims reduced hallucination and answers aligned with organizational definitions by combining the LLM with expert curation and catalog metadata.

Cost or risk

The reliability guarantee is asserted rather than demonstrated in the summary; the strength of grounding depends on how complete and accurate table descriptions and join relationships actually are.

Genie agents combine LLMs with domain expert curation and Unity Catalog metadata.
Open source video
Source video

Open questions

5

Do automatically inherited join relationships and table descriptions measurably reduce hallucination, or do they mainly improve fluency on simple single-table questions?

Why unresolved

The summary asserts reduced hallucination as a practical implication but provides no measurement, benchmark, or failure comparison.

Research direction

Build a benchmark of multi-table questions where join metadata is present, absent, and deliberately wrong, and measure answer correctness against a gold SQL/result set.

Source video

How do join relationships affect complex multi-table queries?

Why unresolved

Listed as an open investigation in the summary with no described mechanism for how declared relationships are combined with inferred ones.

Research direction

Probe agents with questions requiring 3+ table joins, vary whether relationships are declared in metadata versus added as curated Examples, and log where the generated queries diverge from expected join paths.

Source video

What should be measured in the activity digest to assess agent ROI and accuracy, beyond question counts and user ratings?

Why unresolved

The summary names the tracked fields but does not define which metrics are diagnostic of quality or adoption.

Research direction

Define candidate metrics (repeat-question rate, rating trend, escalation/abandonment, unique-user retention) and test which correlate with expert-reviewed answer correctness.

Source video

Does auto-loading prior queries and dashboards act as useful few-shot grounding or as a source of stale-context contamination?

Why unresolved

The summary states loading occurs automatically but never characterizes when inherited artifacts help versus mislead.

Research direction

Run an ablation where prior query history is filtered by recency/ownership before ingestion, and compare answer quality against unfiltered ingestion.

Source video

At what instruction-set size does model-suggested instruction authoring become counterproductive (conflicts, redundancy, drift from intent)?

Why unresolved

Genie Code's suggestion behavior is described qualitatively with no limits, conflict handling, or review policy specified.

Research direction

Track instruction growth and contradiction rate over iterative Genie Code sessions, testing whether conflict detection or consolidation steps are needed.

Source video

Key claims

7
factualVerification needed

Genie agents inherit context from Unity Catalog including table descriptions and join relationships.

Evidence

Genie agents inherit context from Unity Catalog including table descriptions and join relationships.

Question

What specific catalog artifacts are ingested into agent context, and how are join relationships represented and updated when the schema changes?

Source video
factualVerification needed

Previous queries and dashboards are automatically loaded into the agent context.

Evidence

Previous queries and dashboards are automatically loaded into the agent context.

Question

What scope of history is loaded (per user, per workspace, time window), and is it filtered or sanitized before entering context?

Source video
factualVerification needed

General instructions allow authors to guide agent output format and terminology, while Examples define granular join relationships and query behaviors.

Evidence

Examples allow defining granular join relationships and query behaviors.

Question

How do instruction-level rules and example-level behaviors interact when they conflict, and which takes precedence?

Source video
factualVerification needed

Genie Code reads existing agent context to suggest new instructions and refines ambiguous definitions such as customer churn rate.

Evidence

Genie Code reads existing agent context to suggest new instructions.

Question

Are suggested instructions accepted unchecked, and how are conflicting or redundant suggestions handled as the instruction set grows?

Source video
causalVerification needed

Combining LLMs with expert curation and Unity Catalog metadata reduces hallucinations and aligns answers with organizational definitions.

Evidence

Reduces hallucinations and ensures answers align with organizational definitions.

Question

What measured reduction in incorrect answers is attributable to catalog metadata versus curation versus the base model?

Source video
factualVerification needed

The Monitoring tab tracks historical questions, user ratings, and conversation logs, enabling audit of agent accuracy and adoption measurement.

Evidence

Allows data teams to measure adoption and audit agent accuracy.

Question

What retention window applies (the example cites the last 30 days), and does rating data correlate with independently judged answer correctness?

Source video
opinionVerification not requested

Genie agents are domain-specific assistants defined and curated by domain experts so their behavior conforms to specific expectations.

Evidence

They are defined and curated by domain experts to ensure specific behavioral manners.

Source video

Connections

5