Data lineage is the traceable relationship between sources, transformations, and results. It is not only for large data platforms. Any local report that will be rerun, handed off, debugged, or asked "where did this number come from?" needs minimal lineage. Company size changes the depth, not the need.
Without lineage, judgment time is what gets wasted
Imagine a monthly metric suddenly falls. You have orders_july.xlsx, clean_orders.csv, three SQL drafts, and final HTML, but cannot tell which source version the clean file used, where refunds were excluded, or which queries must rerun after an update.
Without lineage, people guess from filenames, modification times, and memory. Lineage makes dependencies explicit:
orders.xlsx
→ 01_clean_orders (types and status cleanup)
→ 02_monthly_revenue (month and revenue definition)
→ 03_segment_summary (market grouping)
→ Report / HTML
OpenLineage models lineage as Dataset, Job, and Run: a Job consumes or produces a Dataset, and a Run is one execution of a Job. This reminds us that knowing A depends on B is insufficient; we should also know which execution and transformation produced the result. See the OpenLineage Object Model.
What minimal lineage should retain
You do not need an enterprise metadata platform first. Keep five things for reusable analysis:
- Source identity: filename or connection, worksheet/table, and analysis time.
- Transformation logic: rerunnable SQL, not only an output CSV.
- Intermediate names:
clean_orders, notfinal_v3_new. - Input/output relationships: what every step reads and produces.
- Execution and limitations: when it ran, success state, freshness, and known gaps.
These answer what must rerun after an upstream change, which downstream outputs a defect contaminates, and how a Report number maps back to SQL.
When is a DAG worthwhile?
| Scenario | Suggested record |
|---|---|
| One-off, one clear JOIN | Keep the SQL and source |
| Multi-step cleanup with reused intermediates | Name persistent SQL and dependencies |
| Frequently updated upstream data | Create a DAG / Source Run |
| Formal report or high-impact metric | Add Plan, execution evidence, and limitations |
Do not split one simple query into ten nodes for appearance. Lineage should reduce debugging and rerun cost; if the graph is harder to read than the logic, it has been overbuilt.
How Lantide Data carries local SQL lineage
In Lantide Data, persistent SQL tabs preserve transformations, and query results can become cached intermediate tables in the same workspace. When downstream SQL references upstream caches, Source Run reruns them in dependency order, and Lineage Graph shows relationships among files, persistent tabs, and caches. See Cache and Source Run and User Guide §8.
Two boundaries matter:
- Cache is a workspace intermediate and may be cleared after switching workspaces or restarting; it is not a permanent warehouse table.
- Background Agent query caches may not support Source Run. For stable reruns, create explicit dependencies with persistent SQL tabs.
Lantide keeps local files, SQL, Plans, and Reports in one analytical workspace so lineage supports review and delivery. It does not replace an enterprise catalog, scheduler, or company-wide metadata governance.
A naming convention you can use now
Name tabs with "sequence + action/output + grain":
01_clean_orders_order_grain02_revenue_by_month03_retention_by_signup_cohort
Record each step's input, output, grain, and checkpoint in the Plan. Those extra words save the next owner from reconstructing the whole analysis.
Conclusion
Choose one local report rerun every month. Draw four layers—raw file → SQL → intermediate result → Report—and ensure every arrow has rerunnable logic. That is enough lineage to create value. Upgrade to a platform solution when dependencies and team scale warrant it.