It Depends #64: Iceberg, Delta, Hudi, Polaris, Unity & Lakehouse - Vinoth Chandar Onehouse - Jun ’24

On the occasion of Onehouse’s $35M Series B funding, I had the pleasure of discussing the intricacies of lakehouse architecture and table formats with founder and CEO, Vinoth Chandar. He created the Hudi format while at Uber and then worked at Confluent, before starting Onehouse. Our conversation delved into the origins and differences between Apache Hudi, Apache Iceberg, and Delta Lake.
The complexity behind table formats and metadata is truly remarkable. Vinoth shared insights into achieving interoperability through Databricks UniForm and @Apache XTable, developed in collaboration with Microsoft and Google. We also explored technical aspects of meta stores and catalogs, such as Snowflake’s Polaris and Databricks’ Unity Catalog.
Vinoth also offered his perspective on Tabular (now part of Databricks) and the future of lakehouse management. I believe you'll find this episode insightful and valuable.
Feel free to share your thoughts and feedback
#data #datamanagement #ai #ml #cloud #multicloud #moderndatastack #cloudnative #opensource #llm #lakehouse #apacheiceberg #apachehudi #LFDelta

[@nandakishore5198]

transactional layer
iceberg - update is delete +insert
iceberg - mark the del records , create file of new inserts
hudi - mor
hudi - streaming workloads
liquid clustering
table metadata formats
hudi, delta - more writes
iceberg - more read - warehouse
read, write, table management
read optimised
write optimised
hudi
lakehouse
data rewriting
3 tables formats
table metadata formats
iceberg - snapshot
delta, hudi - log bassd
write, read, table management
hudi - built in table mgmt
metadata
cor ; mow
cow; mor
interoperability
uniform
read compatibility
write compatibility
manifest file
delta log
xtable and uniform
foundational db in sf
access control + technical metadata
polaris - sf
unity catalog - open source
iceberg rest api catalog
metastore - read and write plans
data not tied to one engine
data catalog being interoperable bw engines
lake view - ingestion, transformation, orchestration- observability