Why AI Gives Wrong Answers About Your Business Data—and How to Diagnose the Problem

An AI agent can run valid SQL and return a convincing answer that is wrong. Before replacing the model or rewriting its prompt, locate the error: interpretation, business definition, query, source data, or explanation.

These failures require different fixes. A stronger model may still use an ambiguous metric. A perfect query cannot recover transactions that have not arrived in the warehouse.

First, reproduce the answer

Save the original question, the user's permitted scope, the metric definition, the generated query or tool request, the returned rows, and the final response. Record the data refresh time and model configuration too. Store traces under appropriate access controls because they can contain sensitive data.

Re-run the calculation against the same data snapshot. Comparing yesterday's AI answer with today's dashboard can create an apparent error when the underlying records have changed.

If you cannot identify the query behind an answer, the first improvement is traceability.

Failure 1: the business term is ambiguous

“Sales” could mean gross order value, paid merchandise, invoiced revenue, or cash received. Each may have a different date and treatment of refunds.

For a fictional retailer, define net merchandise sales as paid, non-test merchandise amounts after discounts, minus completed merchandise refunds attributed to the original order. Tax and shipping are excluded; all amounts use USD.

Two eligible orders contain 100 and 60 in merchandise, with a 20 refund on the first. The correct total under this definition is 140. An answer of 160 may be arithmetically correct for gross merchandise sales while answering the wrong question.

Fix: expose named metrics and their definitions. Ask for clarification when the question could reasonably map to more than one. Our semantic-layer introduction explains how this contract supports AI queries.

Failure 2: a join multiplies the amount

Suppose the first order has two line items and the second has one. Joining an order-level net amount to the line items creates three rows:

Joined row	Order-level net amount
A100, first item	80
A100, second item	80
A101, only item	60

A simple sum now returns 220. Nothing about the query's syntax signals that it has counted the first order twice.

Fix: identify the grain of every input and check row counts before and after joins. Calculate at the intended grain, or use correctly configured modeling features for that relationship. SUM(DISTINCT amount) is not a general repair: two different orders can legitimately have the same amount.

Looker's relationship documentation explains why join cardinality is part of the model. For additional examples, see our SQL fanout guide.

Failure 3: the periods do not mean the same thing

An agent may compare the current incomplete month with the full previous month, use UTC where the business reports in local time, or attribute refunds differently from the dashboard.

Fix: include explicit start and end boundaries, timezone, and date field in the query contract. Decide whether historical reports restate when late refunds arrive. State the convention in the answer instead of hiding it behind “last month.”

Failure 4: missing data looks like a business change

A delayed connector can make sales appear to collapse. An inner join can remove orders whose customer record has not loaded. A currency conversion table can lack a required rate.

Fix: check freshness, completeness, missing join keys, and conversion coverage before drawing conclusions. If those checks fail, the correct response may be “the dataset is incomplete,” with the affected period identified.

Failure 5: the number is right but the story is invented

An agent sees a decline in sales and says a price increase caused it. The available data may show only that the events occurred at similar times.

Fix: separate observed changes, candidate explanations, and evidence needed to investigate them. A useful answer can say which region contributed most to the decline without asserting an untested cause.

Build a small regression set

Include the two-order example, an order with several items, a cancelled order, a test order, a refund posted later, an empty period, and a request for another user's restricted data. For each case, specify the expected number or expected behavior.

Try multiple phrasings of the same question. Score numeric correctness, definition selection, permission enforcement, and explanation support separately. An overall average can hide a serious failure in one category.

The next step is to build a semantic layer with explicit tests. If your team is struggling to reconcile AI answers with trusted reports, discuss an AI implementation assessment with Labs4Change.