SBTB 2023: Alex Monahan, In Process Analytical Data Management with DuckDB.

DuckDB is a novel analytical data management system. DuckDB supports complex queries, has no external dependencies, and is deeply integrated into the data science and data engineering ecosystems. It is fast, easy to install and use, and handles larger than RAM datasets. Since DuckDB runs in the same process as the host language (Python, R, Java, JS, Rust, etc.), no serialization or socket communication has to occur, making data transfer virtually instantaneous. For example, DuckDB can directly query Pandas data frames faster than Pandas itself. It can even run in the browser with WebAssembly. In our talk, we will describe the value proposition of DuckDB, and how it can be used to improve users' day-to-day lives through automatic parallelization, efficient operators and out-of-core operations.
More details available here: www.scale.byth...