The most dangerous AI data analysis failure is usually not a broken sum. It is a plausible-looking number produced with the wrong denominator, time window, or post-join grain. When the retrieval logic exists only inside a chat, the team may not even be able to identify where the error occurred.
Correct Arithmetic Can Still Answer the Wrong Question
Consider a common request: “Compare payment conversion in June and May.” It contains at least five unresolved conditions:
- Denominator: Visitors, registrants, people who created an order, or people who reached the payment page?
- Time window: Grouped by the month of the event, or by the cohort's registration month?
- Grain: One row per person, session, or order?
- Join: After orders are joined to line items, is each order multiplied by its number of products?
- Freshness: Is June finalized, and have refunds and cancellations been backfilled?
Suppose SQL joins 10,000 orders to 28,000 line items and then applies COUNT(*). The database precisely returns 28,000. The arithmetic is not wrong; the analyst has treated item rows as orders. If the model then writes a fluent interpretation, the error becomes even harder to notice.
Research on enterprise text-to-SQL also shows that real tasks go far beyond generating SQL from a visible schema. The 632 enterprise workflow questions in Spider 2.0 often require metadata, dialect documentation, and project code. In the latest revision of the paper, an o1-preview code-agent baseline solved 21.3%. That number applies only to that dataset and setup and should not be treated as the accuracy of every AI product. See the Spider 2.0 paper.
EntSQL, introduced in 2026, focuses more directly on internal metrics, reporting conventions, and organizational rules. Most of its questions require domain knowledge beyond the question and schema. The practical implication is clear: a model must have more than SQL ability. It needs the right business context, and that context must itself be reviewable. See the original EntSQL paper.
Four Lines of Defense That Expose Errors Before a Decision
1. Plan: State “How We Will Calculate It” in Plain Language
At minimum, the Plan should name the decision question, denominator, grain, time range, join keys, exclusions, and data cutoff. This is not paperwork for its own sake. It lets product, finance, and operations reviewers stop a bad definition before formal execution without needing to read SQL.
2. SQL: Preserve Reviewable Evidence of Data Retrieval
SQL should expose FROM, JOIN, WHERE, GROUP BY, and time boundaries. Natural language is well suited to explaining intent; SQL is well suited to answering exactly which rows were counted. Even when AI drafts the SQL, the query should not exist only in a hidden model execution environment.
3. Checkpoints: Deliberately Search for Counterevidence
Add minimal validation before reaching a formal conclusion:
SELECT
COUNT(*) AS rows_after_join,
COUNT(DISTINCT order_id) AS distinct_orders
FROM joined_orders;
Also reconcile the numerator and denominator, NULL rates, minimum and maximum dates, and a sample of five source records. A checkpoint is not meant to prove that the model must be right. It creates opportunities to see fan-out, missing data, or an unfinished reporting period.
4. Limitations: Deliver What Is Not Known Too
The Report should state the data cutoff, missing fields, proxy metrics, sampling bias, and causal conclusions that cannot be supported. The U.S. NIST generative AI risk framework treats false or fabricated content as a risk to be managed through measurement, monitoring, and documentation—not merely by asking the model to “be more careful.” See NIST AI 600-1: Generative AI Profile.
How Lantide Data Makes the Process Inspectable
In Lantide Data Project Analysis, the agent first explores the data and drafts a Plan. Users can comment on denominators, time ranges, and checkpoints, then choose Execute only when the Plan is acceptable. SQL tabs, query steps, and the Report together preserve the evidence of the analysis. See the analyst workflow guide for the SQL-first division of work and the agent architecture guide for state and tool boundaries.
These mechanisms reduce the risk that an error cannot be discovered; they do not turn AI into an infallible analyst. Business definitions, data quality, and the final decision remain human responsibilities. If source data is incomplete, Plan and SQL can expose the limitation but cannot invent the missing truth.
Conclusion
Do not ask only whether AI calculated the answer accurately. Ask whether you can reconstruct its denominator, data scope, and each retrieval step. Before giving a CSV or Excel workbook to AI, require a Plan, visible SQL, counterevidence checkpoints, and Report limitations. The answer may arrive a little later, but it will be far more qualified to inform a decision.