Unity Catalog Demo of New Features with Zeashan Pappa at Data + AI Summit 2024

Get open access to your data no matter where it resides while applying unified governance - see the latest features of Unity Catalog.
Speaker:
Zeashan Pappa, Staff Product Manager, Databricks

[@chinmaykajalwa]

Nice video. You showcased an example of Duck DB integration with Unity Catalog. Can you please help me validate if my understanding captured in below points, about the behaviour of "open sourced Unity Catalog" (UC)?
1. Within UC, we can apply the Data Security changes like data masking. When this UC is accessed from Duck DB, the columns will appear as masked there too.
2. Similarly, other Security config such as row level Security, column level Security will also be visible in Duck DB.
3. Similar to "attach accounts_prod" command in Duck DB, we can integrate UC with other lakehouse implementations such as Microsoft Fabric and even on-prem Delta Lake too (or at least such integration is in roadmap).
4. Such tables are hosted/managed Within Databricks, but are accessed from Duck DB too, which is a reverse of what is done in case of "external table".

[@nirakar085]

Great features.

[@cobrider2]

2 reactions:
- by querying the table with duckdb, the authentication and permission is handled only by Unity Catalog, and not by the underlying storage solution (AWS S3, Azure ADLS, ...). right ?
- Applying column masks will only work for hosted compute like the databricks clusters, because querying with a local self hosted compute like DuckDB requires to download the parquet files (containing the PII data) locally then only execute the query... meaning you actually have PII data downloaded on your local machine. right ?