Why AZ-Five runs on DataFusion and Delta Lake, not ClickHouse

Every analytics platform makes one decision that shapes all the others: where do the tables live, and how many copies of them are there? The industry default for a product like ours is two copies — an object-store lake for the “real” data, and a ClickHouse (or similar OLAP store) fed from it, because that’s where queries are fast. We decided early that AZ-Five would have one copy, and that decision is now load-bearing for almost everything the product promises.

The one-copy constraint

AZ-Five stores every table — uploads, connector syncs, product events, logs, model predictions — as standard Parquet files with a Delta Lake transaction log, on object storage. Queries run through Apache DataFusion, a Rust query engine that reads those tables in place. There is no second store, which means there is no pipeline feeding a second store, no schema drift between the two, and no moment where the dashboard and the SQL editor disagree because they read different copies.

That constraint is also what makes the honesty claims on this site cheap to keep: “your data is exportable” means copying Parquet files, because Parquet files are all there is.

What the two-copy default buys, and what it costs

To be fair to the default: a dedicated OLAP store is genuinely fast, and nothing we have built makes that untrue. What it costs is coherence. Once events live in one engine and business tables in another, every feature that spans the two — a funnel next to a revenue chart, a quality rule on an events table, lineage from a dashboard back to an upload — becomes an integration project. Most vendors resolve this by not spanning the two, which is how a team ends up with five tools.

We wanted the funnel and the revenue dashboard resolving the same governed measure over the same table. That only stays simple if there is one table.

Making one copy fast enough

The honest version of this story is that “query the lake directly” is slow until you do the work. The work, in our case:

• Arrow all the way down. DataFusion is Arrow-native, and so is our ingestion path — data moves between the ingest workers, the engine and the API as Arrow batches, not through serialization hops.
• Pruning before reading. Delta’s transaction log gives per-file column statistics; combined with partition layout and Parquet page indexes, most queries never open most files. Derived time partitioning keeps event tables laid out so time-range queries prune well by construction.
• A block cache under the engine. Object storage has real latency, so the engine runs over a native block cache — hot Parquet byte ranges are served from local disk, while the Delta log itself is always read fresh so a query can never see a stale table version.
• Rollups where they’re provable. For big event tables we materialize time-grain rollups (minute and coarser). The planner rewrites a chart query onto a rollup only when it can prove the answer identical — otherwise it reads raw. Speed is an optimization, never a different answer.
• Buffering the last mile. The freshest events sit in a hot buffer that queries merge with the lake, so “real-time” doesn’t require a streaming OLAP store either.

None of these is exotic. The point is that they compose into a single-store system that holds up under product-analytics workloads — funnels, retention, live tails — without a second engine.

Where this bet shows up in the product

Because every capability reads the same tables, the cross-cutting features stop being integrations: quality state computed at ingestion shows up on dashboard tiles; lineage traces a published number through its measure to the raw upload; an ONNX model’s predictions land as just another table watched by the same monitors. The transit-diagram metaphor on our homepage is not decoration — it’s a literal claim that the lines share track, and the shared track is the lakehouse.

The two-copy architecture can be made faster at the extreme end. We think the one-copy architecture is what makes a data platform coherent enough for a lean team to actually run. That’s the bet.