How To Integrate Databricks With Snowflake-Managed Iceberg Tables

Snowflake recently announced that Iceberg Tables is now in Public Preview. This is a huge milestone for Data Lakes and Lakehouses using open standards. To demonstrate the openness, flexibility, and interoperability of the Iceberg spec within Snowflake, this video demos how to integrate Snowflake-managed Iceberg tables within Databricks. As of today, you can use an Azure Databricks cluster to query Snowflake-Managed Iceberg data within AWS (or any supported Databricks / CSP-hosted storage provider.)
Check out this blog for the implementation details. Please leave comments/feedback.
/ how-to-integrate-datab...
Subscribe for more! www.snowflake.c...
Explore sample code, download tools, and connect with peers: developers.sno...

[@HetAcharya]

Amazing demo Paul!!

[@handekurtur2518]

Hi Paul, amazing demo many thanks! can we use Iceberg tables with Azure Data Lake Storage Gen2 instead of S3? only on azure platform

[@rajapathak13]

Great video!!
Is it possible to have my iceberg table created through databricks and have snowflake iceberg table to read the parquet seamlessly? Will Snowflake catalog gets updated everttime there are inserts happening on azure iceberg table

[@IrfanAnsari-ws7hv]

Is it possible to write data from Spark to a Snowflake managed iceberg table?

[@PaulNeedleman]

Unfortunately, no. There are two main patterns:
-Snowflake write to Managed Iceberg tables
-Spark write to Snowflake externally managed tables
as of today there is no way to to write externally to a managed iceberg table. Stay tuned tho!

[@nitinkansal]

nice demo. Can we access to iceberg tables using query federation in databricks instead of libraries to make connection to snowflake?

[@PaulNeedleman]

Using Federated Query is a different pattern and I have not explored it. That said, it should work to query Snowflake tables but it would not go through the Snowflake SDK. The SDK is a great option because it doesn't require Snowflake compute and my understanding is that the Federated query model would.

[@nitinkansal]

@@PaulNeedleman I replicated all the steps from the demo but I was not able to list the table. It was empty result from databricks end. Any clue?

[@nitinkansal]

@@PaulNeedleman I have workspace unity catalog enabled. Is that the reason?

[@PaulNeedleman]

@nitinkansal it's very possible. I tested without unity catalog.