A Semantic Layer for AI Agents: Inside Datapipelines.

A query can run successfully and still answer the wrong question. The column exists. The join executes. The numbers look reasonable. But the amount was in cents, the timestamp was local time, or “active customer” meant something different to the person asking.

Those are the problems I want a semantic layer to address in Datapipelines. An AI agent needs more than a database schema. It needs the meaning behind the data, the evidence supporting that meaning, and a way to recognize when its knowledge needs another look.

Our implementation makes that knowledge part of the metadata an agent already reads. An observation learned while building one pipeline can become useful context for the next session, with its origin and limitations still attached.

This is a walkthrough of the implementation: where semantics live, how agents record them, how they reach the next reader, and what human review currently does—and still needs to do.

Implementation snapshot: based on the source reviewed on 8 November 2026. Agent procedures below describe the supplied instructions; they do not guarantee that every model follows them. This walkthrough does not establish which source revision is running in a particular deployment.

What is a semantic layer? #

A semantic layer connects technical data structures to the meaning people use when asking questions. It describes concepts such as revenue, customers, units, time zones, relationships, and the grain of a dataset: what one row represents.

For example, a schema might tell you that amount is a decimal. A semantic definition tells you whether it represents dollars or cents, whether tax is included, and which transactions belong in revenue.

Many semantic-layer implementations focus on centrally defined metrics and dimensions. The dbt Semantic Layer, for example, lets teams define business metrics in their modeling layer and reuse them across applications.

In Datapipelines, we are building around another part of the same problem: the knowledge an agent needs while authoring a data pipeline. That includes business definitions, but also details that can invalidate an otherwise well-formed query: sampling, timestamp conventions, coded values, join assumptions, and exceptions in the source data.

Here, a learned fact means a stored statement about data or its interpretation. It is different from a fact table in dimensional modeling. The word “fact” also does not mean “automatically true.” Every statement carries a trust status.

Meaning lives in several places #

A single description field cannot carry the whole model. We keep structural metadata, human descriptions, learned observations, and business choices distinguishable.

These sources have different authority. A comment is useful context, but it may be outdated. A column’s name suggests a meaning, but does not establish it. Statistics may be absent or old. The agent must read the available evidence before deciding what a value means.

We also keep dataset knowledge out of the generic agent instructions. Those instructions teach the procedure; the connected datasource supplies the particulars. A new customer’s database should not inherit assumptions from somebody else’s sample dataset.

The facts we record #

The implementation has a closed list of twelve fact kinds. The kind determines the allowed scope.

Datasource facts travel with access to the datasource. Other workspaces granted that datasource can reuse them. Workspace rules stay within their workspace. Two organizations can use the same source data while defining revenue differently.

A datasource fact must reference at least one table, optionally a column. A join can reference multiple objects. A workspace rule may have no table references, which matters when a definition spans datasources. Such a rule is still recorded against a particular datasource so it has a discovery and visibility context.

Scope also determines what belongs in the store. An observation about one report’s selected quarter belongs in that pipeline’s description unless it establishes something reusable about the source. A tool timeout or platform behavior belongs in operational knowledge, rather than a dataset’s semantics.

How an agent decides to create a fact #

The server does not automatically turn every query result into knowledge. The agent’s instructions and tool descriptions tell it when to call semantics_record; the server validates that request.

The agent instructions prescribe this sequence:

1. Read the datasource listing. It includes descriptions, datasource-wide facts, and existing workspace definitions. Reuse an existing definition before inventing another one.
2. Discover schemas and tables. Read their remarks and any table-level facts.
3. Read every relevant column. Inspect types, comments, warnings, and attached facts before writing SQL against those columns.
4. Read catalog statistics. Use bounds and indexes to understand coverage and plan useful probes.
5. Probe the unresolved questions. Examine values that will be filtered, grouped, or joined. A claim about a whole table needs evidence covering that claim, rather than a convenient handful of rows.
6. Record reusable meaning that introspection cannot provide. Include the supporting SQL when available. Keep question-specific interpretation in the pipeline description as well.

For example, seeing temperature as DECIMAL does not establish its unit. Reading a companion unit field and checking source documentation may establish the convention. The agent can then record that convention with references to the relevant columns and the query it used.

An unresolved assumption remains an assumption. If the agent cannot establish a time zone from the available information, it must say so. Recording the assumption without evidence gives it asserted status; it does not turn the guess into an observation.

Business choices need a separate decision. A distribution of customer activity can inform a proposed definition of “active,” but no query can decide that the organization approves a 30-day threshold. That distinction is enforced in the trust assignment.

What happens inside semantics_record #

The MCP adapter passes the request through a shared application service and a recorder. The recorder is responsible for validating the fact and its evidence before insertion.

The main checks are:

• The caller has permission to record semantics, and the datasource is visible to the active workspace.
• The kind belongs to the requested scope. Fact text is between 8 and 1,000 characters; an optional evidence summary is at most 300 characters.
• Structural references resolve against live schema introspection. Unknown tables or columns are refused.
• The store checks for an identical live fact using scope, workspace, datasource, kind, normalized references, and text.
• Supplied evidence SQL executes successfully through the read-only probe path. Refused SQL, execution errors, and timeouts prevent the record from being created.

There is also a name-agreement check in the MCP adapter. When the catalog listing is available, an exact table name mentioned in the text but omitted from the references can be added automatically; the response reports refs_added. A near-match can be refused with semantics.ref_mismatch. This helps stop a useful sentence from being attached to the wrong object. It is a catalog-assisted check, not a general natural-language proof.

Evidence runs once at recording time. The result is capped at five returned rows for summarization, with a timeout. That return cap does not promise a five-row database scan: a grouped query can still examine many rows. If the caller supplies no summary, the first returned rows are formatted into a compact one.

An illustrative record #

Suppose an example weather datasource contains readings.temperature and readings.unit. After investigating the unit convention, an agent might call semantics_record with:

{
  "scope": "DATASOURCE",
  "datasource": "weather",
  "kind": "unit",
  "fact": "readings.temperature uses the unit named by readings.unit.",
  "refs": [
    {"table": "readings", "column": "temperature"},
    {"table": "readings", "column": "unit"}
  ],
  "evidence_sql": "SELECT unit, COUNT(*) AS rows_seen, MIN(temperature) AS low, MAX(temperature) AS high FROM readings GROUP BY unit"
}

This is an illustrative payload, not a claim about a production dataset. Its SQL provides inspectable observations. The agent still has to justify the interpretation: the server checks that evidence runs, not that it logically proves the accompanying prose.

That limitation matters. An irrelevant query—or a query returning no rows—must not be mistaken for proof merely because the record is labeled observed.

What a learned-fact record contains #

Facts are persisted in the metadata database’s learned_facts table. The important fields fall into a few groups:

References are structural data, rather than names that a future reader must extract from prose. They provide the attachment points for metadata enrichment and the objects against which drift can be checked.

Provenance also has a visibility boundary. A shared datasource fact can be useful to another workspace without exposing the originating workspace’s pipeline. The response includes that pipeline link only when the reader can access it; from_this_workspace distinguishes local from externally recorded knowledge.

How semantics reach the next agent #

The most consequential design choice is where facts are delivered. An agent should encounter them while reading the objects it needs to use.

The shared LearnedFactsEnricher loads visible facts and attaches them to the existing discovery responses. MCP tools and their corresponding REST metadata endpoints use the same enrichment and wire-format helpers.

Workspace definitions are present at the beginning of discovery deliberately. If the definition of “active” only appeared after the agent had selected a column, it could arrive after the agent had already chosen its own interpretation.

A shortened, illustrative column response looks like this:

{
  "name": "temperature",
  "type": "DECIMAL",
  "source_type": "numeric",
  "nullable": false,
  "warnings": [],
  "remarks": "Sensor reading",
  "facts": [
    {
      "kind": "unit",
      "scope": "DATASOURCE",
      "fact": "readings.temperature uses the unit named by readings.unit.",
      "trust": "observed",
      "evidence_summary": "unit=C, rows_seen=1200, low=-3.5, high=31.2",
      "recorded_via": "mcp",
      "from_this_workspace": true
    }
  ]
}

The full inline fact also has its ID and recording timestamp. Optional fields include drift explanations, conflict markers, and visible source-pipeline information. Full records from semantics_list expose the evidence SQL and history fields needed to investigate further.

This retrieval uses datasource and object references. The learned-fact path does not require an embedding search, vector database, or a separate conversational memory lookup. It also does not automatically attach every learned fact to arbitrary SQL result rows: enrichment happens at the discovery surfaces above.

Connecting a definition to executable logic #

Remembering a business definition is only part of reuse. An agent should also be able to find the code that already computes it.

Script-template versions can cite workspace rules through an implements list of fact IDs:

{
  "implements": ["<workspace-definition-id>"]
}

The reverse relationship is returned as implemented_by, containing template IDs and versions. It appears on workspace rules in semantics_list and in the datasource listing’s definitions. Template discovery can also filter by a fact ID.

That gives the next agent a useful chain:

Business question → recorded definition → implementing script version → pipeline reuse.

The link is limited to workspace rules visible in that workspace. It is a citation, not a proof that the script correctly implements the prose. When a cited rule is retired or superseded, the citing version is marked needs_review, with retired_facts identifying the affected rule and its successor where present.

The system does not rewrite the script automatically. A person or agent must examine the successor, check the implementation, and update the citation. A pipeline release that pins such a version receives a review warning; that warning is not itself a release refusal.

Trust, human review, and the current approval boundary #

The trust model makes uncertainty visible:

Human approval is the intended authority for business choices, but the approval workflow is not complete in the implementation described here. The database and domain model include verified, verified_by, and verified_at. The implemented MCP actions are record, list, and retire. The facts UI lets people inspect the knowledge; it does not yet offer verify, edit, or retire controls.

That means I cannot describe this as “every fact is approved before agents use it.” Asserted and observed facts are already served to readers with their status. The agent instructions require readers to consider that status, investigate uncertainty, and revisit stale or review-required facts. A workspace definition remains asserted even if its supporting SQL executes successfully.

Today, a human can review the statement and evidence, decide whether the definition is appropriate, and direct an authorized agent to record a correction or retire a wrong statement. That review does not automatically produce a persisted verified stamp. Completing that explicit confirmation action is separate work.

Authorization still applies to the existing actions. Recording and retiring require the corresponding permissions; the author role and administrative roles have those capabilities, while the promoter role does not gain them merely by being allowed to promote artifacts. Explicitly retiring a shared datasource fact created from another workspace additionally requires datasource-management permission.

Maintaining facts as data and definitions change #

Detect structural drift on reads #

At record time, the recorder computes a SHA-256 fingerprint for each referenced table from its sorted column names and canonical types. Per-table fingerprints remain separately addressable, so a fact referencing two tables can be checked when either table is read.

When columns are introspected again, the enricher compares the current structure with the recorded fingerprint. A missing referenced column makes the fact stale. A changed fingerprint with references still present makes it need review. A complete table listing can detect a missing table; a filtered or truncated listing cannot justify that conclusion.

These demotions are persisted. A later read does not automatically restore a fact’s trust, even if a similarly named column reappears. The system cannot safely distinguish a rename from an unrelated replacement.

There are deliberate limits: this check uses column names and canonical types. Comments, nullability, and precision/scale are not separate inputs to the fingerprint; a change that leaves the column names and canonical type labels unchanged will not be detected by it. It does not fingerprint data contents either. A currency convention can change while the schema remains identical. Evidence is not continuously rerun, and a rule without structural references has no table fingerprint to check. Semantic maintenance still requires judgment.

Keep competing statements visible #

Within the facts considered by a metadata-enrichment call, multiple live facts with the same kind and reference set are flagged with conflict: true. Both remain visible. The store does not choose a winner based on recency or an agent’s confidence. The separate semantics_list projection does not currently calculate this conflict marker.

This is a structural conflict signal. It does not attempt to decide whether two sentences logically contradict each other. Exact duplicates are checked separately during recording.

Replace explicitly and retain history #

To correct a fact, record a replacement with supersedes pointing to the predecessor. The record path inserts the replacement and retires the predecessor with the reason superseded. To withdraw a statement without replacing it, use semantics_retire with a reason.

Retired records remain available through semantics_list with include_retired. Keeping that history allows a reviewer to understand both what the system currently believes and what it used to believe.

Recording and explicit retirement also produce audit events with identifiers and operation details. The recording event includes kind, scope, datasource, references, trust, and whether evidence was supplied; it deliberately omits the fact text and evidence SQL. The fact record itself retains the evidence for authorized readers.

Managed lake columns have a separate semantics projection #

There is another implementation detail worth separating from agent-learned facts.

A managed lake table definition includes stable field IDs, names, logical types, requiredness, documentation, and tags. Applying a definition rebuilds lake_column_semantics from that applied version. Each projected row includes the field ID, ordinal, name, logical and Iceberg types, requiredness, documentation, tags, and applied-version number.

The settlement machinery can rebuild this projection when its applied version is out of step with the committed definition, under a table lease. This keeps derived column metadata tied to the definition that was applied.

This projection and learned_facts have different jobs. One is derived from declared table definitions; the other stores learned observations and workspace decisions with evidence and trust. The generic column-response shape described above does not automatically expose every lake projection field as a learned fact. Keeping that distinction prevents a declared tag from being presented as an agent-verified observation.

What I want the next session to inherit #

The useful outcome is concrete: an agent investigating a datasource should discover what an earlier session learned, see why it was recorded, find the rule and implementation already in use, and recognize when something needs review.

That requires more than saving a sentence. It requires the right scope, structural attachments, evidence, visible uncertainty, replacement history, and an honest human-review boundary.

Those are the foundations of the learned semantic layer in Datapipelines. The next agent starts with inspectable knowledge, and the person responsible for the business definition can see what is being carried forward.

If you are working through similar questions around data platforms or agent context, get in touch.