A PM's responsibility in reviewing AI analysis is not to prove every SQL line correct. Before Execute, confirm six things: the decision being supported, who is analyzed, the denominator, the time range, exclusions, and how the result will be validated. These form the business contract; an analyst or Agent translates them into SQL.
1. Which decision should this analysis change?
"Analyze paid performance" is too broad. A better question is: "Decide whether to extend the trial for new customers by examining payment conversion and refunds within 30 days of their first trial."
Knowing the intended action reveals the required segments, comparison, and evidence. Curiosity alone may not justify formal-analysis overhead.
2. Who are we analyzing, and what does one row represent?
Clarify whether the unit is an account, user, order, or event. One user can have several orders; one order can have several items. Without grain in the Plan, duplicate rows after JOINs can inflate results.
A PM need not specify SQL. A comment such as "calculate final conversion at account grain; count repeated payments by one account once" is enough.
3. What exactly are numerator and denominator?
"Paid users" may mean ever paid, currently subscribed, or paid without a refund. The denominator may be all signups, checkout entrants, or eligible new customers. Write the formula in plain language:
30-day paid conversion = accounts completing their first valid payment within 30 days ÷ eligible accounts starting their first trial in the period.
If the team cannot agree on that sentence, it should not Execute.
4. When does time begin and end?
Confirm time zone, calendar versus cohort window, and whether the last cohort has a complete observation period. Someone registering July 31 plainly lacks 30 days of data on August 1. When comparing versions, check promotions, holidays, and tracking-rule changes.
5. What should be excluded?
Test accounts, employees, refunds, cancellations, duplicate events, bots, and missing data can all change the conclusion. More exclusions are not always better; every rule needs a decision rationale.
An effective Plan comment is concrete:
Exclude fully refunded payments and internal test accounts from
paid users; keep partial refunds but disclose them in limitations.
6. How will we know the result did not drift silently?
Require at least one validation checkpoint:
- explain the scale difference from the existing signup report;
- ensure distinct account count is stable before and after JOINs;
- measure NULLs, duplicate keys, and unknown statuses;
- manually inspect ten accounts against the definition;
- show old and new definitions side by side and attribute the difference.
Validation does not prove AI is always right. It creates checks capable of falsifying an error.
A pre-Execute PM review card
[ ] Decision: which choice will this result support?
[ ] Population: who is the unit and eligible population?
[ ] Denominator: can the formula be stated in one sentence?
[ ] Time: are time zone, dates, and observation windows complete?
[ ] Exclusions: how are refunds, tests, and missing values handled?
[ ] Validation: which checkpoint can detect a definition or JOIN error?
How Lantide Data lets non-SQL stakeholders review
In Lantide Data Project Analysis, an Agent drafts a readable Plan. A PM comments directly on selected text, and the analyst or Agent revises it. Formal execution begins only when a user presses Approve & Execute. See Review a Plan without SQL and the quality bar before Execute.
PMs can focus on intent, definitions, and acceptance instead of pretending to be SQL reviewers. SQL remains for analyst review, while the Report includes numbers, evidence, and limitations. Plan approval also does not raise the Observe/Execute/Admin ceiling of an external MCP connection.
Lantide supplies review and authorization points, but cannot create business definitions for the owner or guarantee reviewed Plans contain no data defects. High-risk decisions still require reviewers with domain and data expertise.
Conclusion
For the next AI analysis Plan, six comments may be enough: decision, population, denominator, time, exclusions, and validation. A PM's value is not prettier SQL; it is getting the team to agree on what a number means before it is produced.