Extending Postgres for High Performance Analytics (with Philippe Noël)

PostgreSQL is an incredible general-purpose database, but it can’t do everything. Every design decision is a tradeoff, and inevitably some of those tradeoffs get fundamentally baked into the way it’s built. Take storage for instance - Postgres tables are row-oriented; great for row-by-row access, but when it comes to analytics, it can’t compete with a dedicated OLAP database that uses column-oriented storage. Or can it?
Joining me this week is Philippe Noël of ParadeDB, who’s going to take us on a tour of Postgres’ extension mechanism, from creating custom functions and indexes to Rust code that changes the way Postgres stores data on disk. In his journey to bring Elasticsearch’s strengths to Postgres, he’s gone all the way down to raw datafiles and back through the optimiser to teach a venerable old dog some new data-access tricks.
-
ParadeDB: paradedb.com
ParadeDB on Twitter: / paradedb
ParadeDB on Github: github.com/paradedb/paradedb
pgrx (Postgres with Rust): github.com/pgcentralfoundatio...
Tantivy (Rust FTS library): github.com/quickwit-oss/tantivy
PgMQ (Queues in Postgres): tembo.io/blog/introducing-pgmq
Apache Datafusion: datafusion.apache.org/
Lucene: lucene.apache.org/
Kris on Mastodon: mastodon.social/@krisajenkins
Kris on LinkedIn: / krisjenkins
Kris on Twitter: / krisajenkins
-
0:00 Intro
2:03 Podcast
1:06:36 Outro
#softwaredevelopment #rdbms #programming #databases

[@matterhart]

He nailed it with the kitchen knife vs chainsaw analogy, with postgres+extensions fitting nicely in between. I think a lot of start-ups and b2b businesses would really benefit from the simplicity of 'just use postgres', and I think their work is going to bring that simplicity to more and more companies. The really big products don't need this, but I bet even really big companies have smaller, perhaps internal projects that should start with 'just use postgres'.

[@user-kt1iz4vc3x]

i came because i'm interested in paradedb and i'm staying because the the host has a cool vibe 👍

[@JulianHaeger]

This has rapidly become my favourite developer podcast. Kris seems to ask the exact questions I'm starting to think of, but far more articulately and precisely than I could, mixed in with lots of insightful questions I'd never think of.
Keep it up 👍

[@DeveloperVoices]

Thanks! Will do. 😁

[@maxwellflitton3973]

@@DeveloperVoices hey I'm the author of the O'Reilly book async rust, and currently writing the 3rd edition of Rust and web programming for Packt. I'm also helping build surrealDB for the day job. I'm loving your inerviews, it's clear you really understand fundamentals of programming and have experience. Would love to meet you sometime. I'm based in London

[@TheHubra]

Have no interest in postgres but I loved just having this in the background as I did my own coding. Thanks for another wonderful video!
I actually found myself happy knowing a little more about postgres despite not being interested in it. Just one little thing to have in my back pocket in case a situation arises where a good extension mechanism in a DB is required, in which case postgres will be the first thing I'll try!

[@hernantz]

Keep it up with these great interviews!

[@ArtemMelanich]

Are there any good options on sharded\distributed Postgres alternatives?

[@CaribouDataScience]

You misspelled duckdb 😊

[@DeveloperVoices]

Haha. Loving DuckDB at the moment. 😁

[@dingthe4th]

First!