Analyze MILLIONS of points in SECONDS (on your computer) with DuckDB for GIS
Vložit
- čas přidán 8. 07. 2024
- DuckDB is an incredibly powerful analytical database that can run right on your computer and allows you to analyze LOTS of data. Plus it is very easy to install and it now has support for GIS and geospatial analysis. This coding tutorial will walk you through the setup and different use cases for DuckDB for GIS
0:00 Intro
2:55 Setting up
5:52 Installing spatial and H3 extensions
10:27 Reading geospatial data
18:17 Native DuckDB geometries speed test
24:34 Using H3 and DuckDB in Python
27:42 Resources and wrap-up
Download the data in the tutorials here:
➡️ NYC 311 Data: data.cityofnewyork.us/Social-...
➡️ NYC Neighborhoods Data: github.com/HodgesWardElliott/...
🔌 DuckDB spatial Extension: github.com/duckdblabs/duckdb_...
🔌 DuckDB H3 Extension: github.com/isaacbrodsky/h3-du...
Guides and resources from the video:
🌎 Geospatial DuckDB from Mark Litwintschik: tech.marksblogg.com/duckdb-ge...
☠️ Big Data is Dead: motherduck.com/blog/big-data-...
🐤 DuckDB for Beginners from @motherduckdb : • DuckDB Tutorial For Be...
✅ Check out my Notion Productivity Template here: mbforr.gumroad.com/l/notion-p...
📚 FREE CARTO Account for Students: docs.carto.com/faqs/categorie...
Check out some other posts and resources here:
Learn Spatial SQL: forrest.nyc/learn-spatial-sql/
40+ Resources for Spatial SQL: forrest.nyc/40-spatial-sql-re...
75+ Geospatial Python and Spatial Data Science Resources and Guides: forrest.nyc/75-geospatial-pyt...
Other courses to get started with GIS, spatial data science, and data science (affiliate links):
Geographic Information Systems (GIS) Specialization from UC Davis: imp.i384100.net/n1Kyxo
Spatial Data Science and Applications from Yonsei University: imp.i384100.net/9W6E6W
Python for Everybody from University of Michigan: imp.i384100.net/BX3P31
Remote Sensing Image Acquisition, Analysis and Applications: imp.i384100.net/Xxe4Ay
Google Data Analytics Professional Certificate: imp.i384100.net/oeKO9b
IBM Data Science Professional Certificate: imp.i384100.net/n1KyK6
#gis #python #sql
➕ Follow me on LinkedIn: / mbforr
🎙️Check out the Modern GIS episode on the MapScaping Podcast: mapscaping.com/podcast/what-i...
🌐 My personal website: forrest.nyc
🚀 My top links and resources: forrest.nyc/links-and-guides-... - Věda a technologie
Happy to see you, Matt! I would love to watch your videos more frequently.
Thanks so much that means a lot!
Thanks for the amazing demo!
Appreciate it!
Nice one! The geo data viewer extension is neat, learned something new :)
Thanks and thank you for your videos too! Learned a lot from you as well!
Woah, that's rad as duck! 😉
🦆 More and more I have been using it locally and it’s been great!
Wow, I learned something new. Thanks for the vid.
Glad to hear it!
I’m using postres like everyone :) but i’ll give a try to duckdb. It looks promising and nice.
Me too! I think the spatial capabilities are going to advance a lot but the good news is that you can keep some data in PostGIS and mix it with DuckDB since they have some Postgres connectivity too!
Does this support indexing? Not just your usual spatial index like GIST(geom) but indexes on custom expressions?
Also- great that the native geom types are fast. But do they handle change of CRS, proj, geog vs geom etc?
AKA what if you need high accuracy distance calc
Ah I see it’s GDAL and more coming native, cool :)
I think it doesn’t support CRS yet and yes you can do non spatial indexes.
Brightness of the video is 1 million lumens :D
I know…still working on my editing skills!
@@MattForrestnice.
He has a quack the folder 😮.
Haha
One can use DBeaver to connect duckdb.
Yes definitely
How to make h3? I'm stuck there
also spatial extension doesn't work
Spatial extension now comes native in DuckDB but H3 you have to build locally then use as an unsigned extension.
good lordy, can I buy you a new keyboard? sounds like you're typing on a relic. in all seriousness, thanks for the good videos.
Bad keyboard and bad mic but should be fixed now!
Not supposed to do pip install. Its supposed to be python -m pip install.
😮😢
Good point