Why We're Not Convinced About Data Lakehouse Architecture

We've built 3 data lakehouses and we're not convinced about this architecture at all. We're not sure if we're doing it wrong but we don't think we want to try again.

Unifying the best of data warehouse and data lake sounds compelling, but all of our projects seemed to suffer from the following challenges.

You End Up Rewriting a Data Warehouse

The marketing materials make it sound easy, but you are inevitably thinking about how to layout data on object stores, maintenance jobs, concurrent access, transactions and/or reading the Iceberg spec. Low level non differentiating stuff.

This is engineering effort that could be spent on business logic rather than reinventing storage layer primitives.

More Expensive Than You Think

You save by using object storage but you end up writing ingestion logic, transformation jobs etc which are expensive to run (e.g., Databricks or EMR). You also pay for billions of S3 access calls. And then the engineering time costs more than license fees you are trying to avoid.

The total cost of ownership often exceeds what you'd pay for a managed data warehouse when you factor in:

• Compute costs for transformation engines
• API call costs for object storage
• Engineering time for custom tooling
• Operational overhead

Not Fast Enough for Real-Time Use Cases

You think you don't have a real time use case, but you inevitably want to run an app or something latency sensitive and the lakehouse soon turns out to be too slow for anything fast and interactive. You are then totally snookered and have to implement caching layers which make everything more complex.

This adds another layer of complexity to your architecture and creates cache invalidation challenges.

Ecosystem Tooling is Poor Compared to a Data Warehouse

Reporting, data quality, machine learning etc. Everything is made awkward vs SQL data warehouses.

The maturity of tooling around traditional data warehouses is hard to beat. Years of ecosystem development means better integrations, better debugging tools, and better operational workflows.

Our Conclusion

The technology is cool for data geeks and the idea of open standards based data is compelling, but with how well data warehouses can work with unstructured data and with how well they can export/interoperate with things like Iceberg or Parquet, we honestly think we'll just build around a data warehouse from now on unless we're operating at truly massive scale.

Modern data warehouses have evolved to handle many of the use cases that lakehouses were designed for, while maintaining the simplicity and performance that made them successful in the first place.