Working with AI on analysis · 1/7
Imagine an AI agent finishing a retention analysis in ten minutes. It looks ready, until someone in the meeting asks: “Why did we use this denominator?”
The rationale may be buried in an earlier message, an abandoned query, or a decision made halfway through. The person responsible for the number still has to reconstruct it.
After more than a year of working with AI on data analysis, I keep coming back to this: the more analysis AI does for us, the less analytical context we may retain.
A polished answer can hide what was never made explicit
Language models can make an answer sound complete, even when an assumption is questionable. The conclusion can be clean and confident without adequately exposing its assumptions or uncertainty. Reading it does not tell me what was tested, why one definition won, or what the agent never considered.
When working with a junior analyst, questions usually help me understand where their knowledge ends. Reviewing an AI analysis requires the same attention: why did it make this judgment, what assumptions did it use, and which choices remain unverified? Fluent wording is not enough.
Faster output can create slower ownership
When I run an analysis myself, much of the context stays in my head:
- Why I chose this denominator.
- Which paths I tested and then abandoned.
- Where to drill down if a PM questions an anomaly.
With an agent, the output can arrive faster than that understanding develops. The person preparing the work for delivery gets a second job: analysis archaeology.
If rejected paths, changing assumptions, and validation steps are scattered across the chat, that can mean rework, reusing an already rejected definition, or missing an important clue entirely.
The AI will not be in the meeting to take responsibility for the decision. Less execution time does not necessarily mean less total time from question to a result someone can confidently defend.
My approach: make decisions and evidence part of the work product
I started asking not only how to get a better answer, but how the person and the agent could share the same analytical context.
For the retention example, I want three things to remain visible:
- The Plan preserves decisions: the intended denominator, rejected alternatives, and why.
- Execution records preserve evidence: actual queries, results, and validation tied to that decision.
- The Report preserves the delivery: the accepted conclusion, supporting evidence, and limitations.
If the PM challenges the denominator, I want to see which definitions were considered, why one was chosen, and what might change with another. Rereading the same confident answer does not help.
Agent state separates exploration, planning, execution, and delivery. The agent can try ideas while exploring, but the method must become a reviewable Plan before formal execution. Delivery then converges in a Report. Polished wording alone does not make a result verified or approved.
This is why I have been building Lantide Data around this way of working. I want AI to accelerate analysis while preserving the decisions and evidence a person needs to understand, challenge, and take responsibility for it. Not every thought needs a record, but the person presenting the analysis should not have to reverse-engineer it before the meeting.
Further reading
Try this way of working in Lantide Data: Download Lantide Data.
Next：Why I Chose SQL-first After Working with AI on Data Analysis