Working with AI on analysis · 3/7
After completing an analysis with AI, can I save it as a Skill and use it directly next time?
Once we have agreed on definitions, fixed the SQL, and worked through a few traps, preserving that effort seems natural. I tried using Skills, only to find more conversations beginning with: “You are using the definition from another project. Here it should be…”
I started wondering why knowledge selection that felt obvious to me was harder for the agent.
A complete Skill can still carry the wrong knowledge forward
The hope is straightforward: ask the agent to turn a completed analysis into a Skill, then spend less time explaining the next similar task.
Consider retention analysis for paying users:
- For cumulative paid conversion from acquisition, anyone who has ever paid might count. For recent use of a particular feature, we might only include people who paid in the last 30 days.
- For user value, a payment on any server might qualify. For a promotion on one game server, only payments on that server might count.
Suppose we analyze a server-specific promotion and decide to include characters that paid during the event, counted by account × server. That definition may be entirely reasonable. The Skill records it correctly. But it may be inappropriate for the next analysis of churn across all historical paying users.
I think of this as a metaphorical kind of knowledge overfitting: conditions valid for one case become a supposedly reusable method. The agent may not distinguish the general approach from a decision made only for that occasion. Every word was remembered correctly; the application is still wrong, which can be harder to catch than forgetting.
Parameters help, but scope still matters
An obvious response is to turn the payment period, counting unit, and cross-server deduplication into parameters and require confirmation each time.
For simple, stable analyses, that may be enough. But as the same Skill moves across projects, defaults, conditions, and exceptions accumulate. One project uses lifetime payments; another uses the last 30 days. One deduplicates across servers; another keeps them separate.
To explain the parameters, I also have to include the business context behind each choice. A retention method gradually becomes a manual for operating across multiple projects.
Parameterization is useful when the method has a stable, clear scope. When unrelated project rules are mixed together, adding parameters starts to feel like patching holes. Splitting the Skill and copying shared definitions into every copy creates another problem: updates can be missed.
Traditional analysis has similar problems. We copy a Notebook or edit last month’s SQL. When it is our own work, we often remember which conditions to change. The confusion may only surface at handoff or months later: “Why did I exclude those users?”
With agents, output and task switching both accelerate. Knowledge handoff becomes daily work. Alongside reviewing results, I have to check which historical assumptions the agent inherited today.
Separate knowledge by type and scope
Looking through the knowledge mixed into my Skills, I found four useful destinations:
| Knowledge type | What belongs there | Example |
|---|---|---|
| Method: Skill | Reusable methods and checks across projects | How to analyze retention, choose an observation window, and validate results |
| Shared rules: User knowledge | Stable knowledge within the user’s working scope | The relationship between accounts and characters, or how test accounts are identified |
| Project rules: Project knowledge | Project background and definitions used over time | How one server’s operations define a paying user; not automatically valid elsewhere |
| Analysis decisions: Plan | Conditions and choices for this particular question | The period, exclusions, and reasons used in this analysis |
This does not mean writing four documents before every task. It means routing knowledge when the agent saves it. One-off conditions stay in the Plan; genuinely reusable content can later become a method or shared rule.
The pieces must also connect during use. A Skill reminds the agent to confirm the payment definition. Project knowledge offers the relevant candidate definition. The version adopted for this analysis goes into the Plan.
The agent can know that several definitions exist. It should also know where each applies, and ask when the scope is unclear rather than silently choose.
Recording decisions during analysis has always been more reliable than reconstructing them afterward. It has also been hard to sustain while investigating an anomaly or chasing a deadline.
Agent collaboration creates a more natural opportunity. Each phase already involves discussion, validation, and confirmation. At its end, I can ask the agent to organize the knowledge we just confirmed, suggest where it belongs, and let me review it.
Analyze, classify, confirm, and save while the context is still available. That seems more practical than waiting until delivery to excavate what we actually did.
Further reading
Try this way of working in Lantide Data: Download Lantide Data.
Previous：Why I Chose SQL-first After Working with AI on Data Analysis · Next：Is Data Infrastructure Enough for AI to Take Over a Team’s Analysis?