In one sentence: Lantide Data is a local-first data analysis IDE with a built-in Agent runtime (Harness) designed for analysts—reviewable deliverables, governed execution authorization, and querying and tooling unified with a DuckDB workspace, rather than "a chat window where you paste a CSV."
For step-by-step instructions, see the User Guide. This series explains why it is designed this way, and the product and engineering trade-offs behind it. For terminology, see the Appendix: Glossary at the end.
Series Position
| Order | Article | Topic |
|---|---|---|
| 0 | This article (Series Guide and Product Positioning) | Category, market gap, reading map |
| 1 | Agent-Era Data Analysis Workflow | Reviewable deliverables and execution authorization |
| 2 | Governable Agent Memory | Governable Agent Memory |
| 3 | Prompt and Context Engineering | Prompt and Context |
| 4 | AI Agent Architecture | Agent architecture (runtime / Harness) |
| 5 | Unified Query Layer | Unified query layer |
Suggested order: Series Guide and Product Positioning → Agent-Era Data Analysis Workflow → Governable Agent Memory → Prompt and Context Engineering → AI Agent Architecture → Unified Query Layer.
1. What Gap Are We Filling
"AI can do data analysis" is no longer novel; the pain point is the lack of an Agent execution environment (Harness) purpose-built for analytical work.
Many people still cobble together an analysis chain using a general-purpose chat model plus an Excel / SQL client: fragmented tools, intermediate results that are hard to trace, and AI-generated SQL they dare not treat as a formal deliverable. To make "a dedicated analysis environment" clear, we can roughly divide the market into three tiers (for comparison, not a complete competitive map):
| Tier | Common form | Typical gap |
|---|---|---|
| A. General-purpose Agent | ChatGPT, Claude, Cursor, etc. | Can write SQL, but has no access to your workspace, cache lineage, Plan state, or a local unified query layer |
| B. AI + data features | BI Copilot, NL2SQL, notebook assistants | Mostly one-off Q&A or one-off SQL generation; rarely build a complete, reviewable, execution-authorized analysis flow |
| C. Data analysis Agent Harness | Julius AI, Lantide Data, etc. | Agent + data execution environment + chart/report delivery; products in this tier make different trade-offs on traceability, definitions, and governance |
A Strong Representative in Tier C: Julius AI
Julius AI is a highly mature member of Tier C: upload CSV/Excel or connect Snowflake / BigQuery → natural-language Q&A → charts, statistics, notebooks, scheduled reports. Its product narrative emphasizes "no need to write SQL."
Behind the scenes, the model generates and executes Python / R / SQL inside a session sandbox—this can be called an analysis abstraction layer (the hidden DA, where the Data Analyst execution layer is concealed): trading hidden code for speed, with its strength in the Insights endpoint—getting quickly from data to charts and conclusions.
Common Gaps in This Kind of Harness
Belonging to Tier C does not mean the upstream and governance are already sufficient. The common trade-offs of Julius-style products are:
- Upstream preparation is mostly done inside the session: cleaning and alignment are hard to trace and hard to rerun as formal intermediate results.
- Delivery leans toward charts and conclusions: shareable, but the definition logic is hard to review, annotate, and sign off like a document.
- Workflow is conversation-driven: even with a notebook, there is still little boundary of "write the contract first, then formally run only after a human presses the button."
These gaps do not deny Tier C's value; they illustrate that the Harness already exists, but there is another route that puts more weight on traceable intermediate results, collaboratable definitions, and authorized Execute (the Silver / Insights stages of Medallion below)—and this is the route Lantide Data takes.
Therefore: What Lantide Data Aims to Fill
Our position is: analysis is not a one-off answer, but a set of work products that can be discussed, versioned, and rerun; the Agent should explore, draft, and execute within the same IDE, while humans retain the pace and authorization. Rather than building yet another "chat Harness that produces charts," we make the path leading up to insight a governable artifact within Tier C.
Differences from Julius-Style Harnesses
| Dimension | Julius-style (same Tier C track) | Lantide Data |
|---|---|---|
| Upstream preparation | Silver is mostly in-session and implicit; hard to trace, hard to rerun | Analysis-ready layer — SQL tabs, cache lineage, Source Run (engineering definition in Unified Query Layer §7) |
| Delivery language | In-session Python/R/SQL, code as a secondary role | SQL tabs as the primary artifact; Python is limited to a governed analysis tool (activate_analysis) |
| Analysis quality | Often just the final charts and text; denominator/grain/limitations may not be verifiable | Plan / Report spell out denominator, grain, checkpoint, limitations |
| Workflow | Mostly conversation-driven; has a notebook but still chat-centric | Plan → annotate → user Execute → Report; SQL + Plan can be annotated and signed off |
To help readers place the category and trade-offs, this is not a feature evaluation or market ranking. For the full SQL-first methodology, see Agent-Era Data Analysis Workflow §3.
An Analyst's Day: Pipeline Stages and Work Products
We borrow Medallion terminology to mark pipeline stages (not a claim that Lantide is in-warehouse ETL). The table below aligns a typical day with each stage—and also shows that Lantide's home turf is the traceable Silver and the sign-off-ready Insights, not merely competing for the endpoint charts:
| Stage | Meaning | The analyst in Lantide |
|---|---|---|
| Bronze | Raw landing, little processing | Load CSV / external database / MCP; Quick Analysis to verify columns and distributions |
| Silver | Cleaned, aligned, reusable intermediate results | SQL tabs to explore definitions, columns, JOIN keys, and grain; large mapping tables go in Reference docs (see Workflow §4.1); use cache and Source Run to rerun upstream when needed |
| Gold | Business metrics, subject-area wide tables | In the Plan, spell out metric definitions, denominator, and validation methods; align definitions with colleagues before Execute |
| Insights | Narrative, hypothesis testing, decision recommendations | Draft a Plan → annotate → you press Execute → Report / optional HTML; the Report should include numbers, limitations, and traceable evidence |
The whole flow happens within the same IDE, not scattered across chat windows and spreadsheets.
Product boundaries (to tighten the scope): We are also not a general-purpose coding agent decoupled from the workspace (great at editing a repo, but unconcerned with Plan, Report, or cache lineage), nor a memory product that auto-feeds conversations into a vector store by default. In form, it is a local-first desktop analysis IDE (a local DuckDB carrying CSV / ATTACH / MCP) with an embedded ReAct Agent runtime (a state-filtered tool matrix + UICommand); cross-conversation definitions are written to the knowledge file only after approval via Queued Knowledge review, rather than accumulating in a black box.
Portability (.lantide): For sharing analytical context and results, use a Project export; for collaborative handoff or switching machines, use a Workspace (a Full backup moves multiple workspaces + the Application at once). Workspace / Full backup do not include local CSV/Parquet by default, but you can opt in to package physical data (choose either Extract on the new machine or Rebind to a local directory); all four export types can optionally be passphrase-encrypted at the outer layer. For selection and operation, see User Guide §11.1.0.
2. Series Structure
The main body has five articles: two on methodology (Workflow, Memory) + three on engineering support (Context, Agent architecture, Query layer). Together they explain how the Tier C (analysis Agent Harness) above is designed and implemented; missing any one link would degrade it into "a smart chatbot." For each article's core question, see §4.
3. What These Articles Answer
Lantide Data puts "querying local files with SQL" and "doing analysis with AI" in the same desktop IDE. What readers truly care about is often not just whether the model can write SQL, but:
- Can the analysis process be seen, reviewed, and traced?
- Where is the division of labor between humans and the Agent (who can Execute, who can change the Plan)?
- How are memory and context kept controllable, rather than piling up in a black box?
- Why build a custom Agent loop instead of adopting a general-purpose Agent framework?
This series answers the above with two articles on methodology (Agent-Era Data Analysis Workflow, Governable Agent Memory) and three on engineering support (Prompt and Context Engineering, AI Agent Architecture, Unified Query Layer); this article (Series Guide and Product Positioning) is responsible for the category and the reading map.
flowchart TB
guide[Series Guide and Product Positioning]
subgraph methodology [Methodology]
workflow[Agent-Era Data Analysis Workflow]
memory[Governable Agent Memory]
end
subgraph engineering [Engineering Support]
context[Prompt and Context Engineering]
agent[AI Agent Architecture]
query[Unified Query Layer]
end
ug[User Guide_Operations]
guide --> workflow
guide --> memory
workflow --> context
memory --> context
context --> agent
agent --> query
ug -.->|operational details| workflow
4. Overview of the Main Articles
| Order | Article | Core question | In one sentence |
|---|---|---|---|
| 0 | Series Guide and Product Positioning (this article) | Who are we, and how are we different? | Analysis Agent Harness, reading path, capability map |
| 1 | Agent-Era Data Analysis Workflow | How are deliverables made reviewable? | Plan / Report / HTML, SQL-first, annotation, Execute boundary |
| 2 | Governable Agent Memory | How is memory kept controllable? | Layering, approval, Reorganize, catalog injection |
| 3 | Prompt and Context Engineering | What does the model see? | Summary / Detail, layered Prompt, knowledge and annotation injection |
| 4 | AI Agent Architecture | How does the model do it? | ReAct, state routing, tool matrix, UICommand |
| 5 | Unified Query Layer | What to query, and how to chain it? | DuckDB rewriting, tab cache, Source Run, lineage |
5. Capability Map
| Capability | Methodology | Engineering implementation | Operation (USER_GUIDE) |
|---|---|---|---|
| Projectized Plan → Report flow | Agent-Era Data Analysis Workflow | AI Agent Architecture state and tools, Prompt and Context Engineering Context | §11 |
| Markdown annotation collaboration | Agent-Era Data Analysis Workflow | Prompt and Context Engineering §11 | §11.7–11.9 |
| Reference docs | Agent-Era Data Analysis Workflow §4.1 | AI Agent Architecture read_reference |
§11.3.1 |
| HTML Report delivery | Agent-Era Data Analysis Workflow | AI Agent Architecture HTML tools | §10 |
| Knowledge extraction and approval | Governable Agent Memory | Prompt and Context Engineering injection, AI Agent Architecture expand tool | §15 |
| Explore / Execute SQL | Unified Query Layer | AI Agent Architecture validate_query / run_* |
§12.6 |
| Cache tables and Source Run | Unified Query Layer | AI Agent Architecture run_sql_tab / source_run_sql_tab |
§7–8 |
| Agent operating the IDE (opening tabs, running queries) | Agent-Era Data Analysis Workflow execution boundary | AI Agent Architecture UICommand | §12.6 |
| Analysis quality gate | Agent-Era Data Analysis Workflow | Prompt and Context Engineering + D3 runner smoke | — |
6. How to Read This Series
By Role and Time
| You are… | Suggested reading |
|---|---|
| Product decision-maker (about 5 minutes) | This article §1–§2; then skim the opening of Agent-Era Data Analysis Workflow |
| Want to confirm "is it a Harness" (about 10 minutes) | This article + AI Agent Architecture §1 |
| Analyst / PM / business & operations (about 30 minutes) | Try the onboarding: documentation tour (Analyst / PM / Business & Operations introductions); design depth: Workflow + Memory; operations reference User Guide §11–13 |
| Platform enablement (data engineering) (about 45 minutes) | Learn Platform Enablement (Data Engineering) introduction + User Guide §3–4, §11.1 |
| Engineer (about 120 minutes) | In order: Workflow → Memory → Context → Agent architecture → Query layer; pair the cache with §7–8 |
| Want to try the product directly | documentation tour → User Guide §2 Quick Start |
By Question
| Your question | Read from here |
|---|---|
| How is this product different from ChatGPT / BI Copilot? | This article §1 (the three-tier market and product boundaries) |
| How is it different from Julius AI / chat-style analysis tools? | This article §1 (Julius-style Harnesses and Lantide's trade-offs) |
| Why SQL-first instead of Python as the workhorse? | Agent-Era Data Analysis Workflow §3 |
| How does the analysis flow run, and who can press Execute? | Agent-Era Data Analysis Workflow |
| Queued Knowledge—won't knowledge be recorded haphazardly? | Governable Agent Memory (first cross-reference §15) |
| Why does the AI sometimes keep querying tables? | AI Agent Architecture §4 + Prompt and Context Engineering §3–§6 |
| How are multi-tab SQL and cache chained? | Unified Query Layer §8 + §7–8 |
When does the Agent materialize, run run_query, and expose SQL? |
Unified Query Layer §8.1 + User Guide §7.4 |
| How is the Harness implemented? | AI Agent Architecture → Unified Query Layer |
Each article opens with a "Series Position" table; for the complete guide, this article is authoritative.
7. Division of Labor with the User Guide
| Document | Reader | Content |
|---|---|---|
| User Guide | All users | Where the buttons are, how to click them, what the status prompts mean |
| This series (Articles_v2) | Those who want to understand the design | Category positioning, design motivation, architectural layering, trade-offs, and anti-patterns |
When there is overlap, the User Guide is authoritative; this series uses "see User Guide §X" to bring in operational details, without repeating lengthy step-by-step instructions.
8. Series Maintenance Conventions
A cross-article topic is written in full in only one article (SSOT), with the rest getting 1–2 sentences + a link. When patching in a new feature:
- First check the table below to determine the authoritative article, then write it there.
- Other articles only add a pointing link, without repeating the table.
- Operational steps are still written only in USER_GUIDE.
| Topic | Authoritative article |
|---|---|
| Plan / Execute / annotation / HTML product story | Agent-Era Data Analysis Workflow |
| Analysis quality guardrails and release hard gate | Agent-Era Data Analysis Workflow §11 |
| Knowledge governance | Governable Agent Memory |
| Summary / Detail, Optimizer, QEM semantics | Prompt and Context Engineering |
Ledger, [[QUERY_STEP]], ReAct, exploration budget |
AI Agent Architecture |
| Conditional materialized tables, sqlglot, Source Run DAG | Unified Query Layer |
| Competitive comparison, pipeline depth (the Medallion metaphor), product boundaries | Series Guide and Product Positioning §1 |
| SQL-first methodology (analysis / maintenance / collaboration) | Agent-Era Data Analysis Workflow §3 |
| Analysis-ready layer (engineering definition) | Unified Query Layer §7 |
Appendix: Glossary
| Term | In one sentence | Authoritative article | Operation / FAQ |
|---|---|---|---|
| persist cache | The cache table materialized by a persistent tab Run / run_sql_tab; supports Source Run |
Unified Query Layer §8.1 | §7.4, FAQ: agent vs persist |
| agent cache | run_query conditional materialization; can be referenced via FROM, but does not enter the Source Run DAG |
Unified Query Layer §8.1 | Same as above |
| validate vs run_query | The former only validates with no data rows; the latter previews ≤200 rows | Prompt and Context Engineering §7–§8 | FAQ |
| Query Execution Model | Cross-state semantic contracts such as Run / Source Run / no DDL | Prompt and Context Engineering §7 | §12.6 |
| conditional materialization | An agent cache may be created for multi-step, cache-referencing, or more complex SQL | Unified Query Layer §8.1 | FAQ: run_query materialization |
| optimization hint | A non-blocking reminder: a multi-table JOIN may consider cache, but analytical correctness comes first | Prompt and Context Engineering §7 | — |
| Query Step Ledger | The in-conversation query step index and [[QUERY_STEP]] |
AI Agent Architecture §6 | — |
| Queued Knowledge | Not injected before approval; written to the knowledge file after Apply | Governable Agent Memory | §15.5 |
| flat / catalog | Injecting the full knowledge text vs. a table of contents + expand | Governable Agent Memory §7 | §15.4 |
| exploration budget | When show_tables and the like exceed the soft limit, ask_user is required to continue |
AI Agent Architecture §4.2 | — |
| Execute | Only the user can start the formal execution of a Plan | Agent-Era Data Analysis Workflow §7 | §11.6 |
| analysis-ready layer | The Silver metaphor: rerunnable intermediate results within the workspace | Unified Query Layer §7 | §7–8 |
| Insights endpoint | Downstream deliverables such as charts, narrative, and decision recommendations | This article §1 | — |
| the hidden DA | The model writes SQL/code on the user's behalf within the session, and the user sees only the conclusions | This article §1 | — |
| SQL-first | Definitions and data retrieval use reviewable SQL as the primary contract; Python is a governed statistical extension | Agent-Era Data Analysis Workflow | §12.9 |