Did I Pick The Right Database???
Vložit
- čas přidán 12. 06. 2024
- Databases are scary. Picking the right one is scarier. I hope this helps. Please don't use Firestore.
BECOME A MEMBER IF U LOVE ME:
/ @t3dotgg
Also check me out on other things, like Twitter / t3dotgg
And everything else: t3.gg/links
Idez you are a legend thank you so much
KEYWORDS JAVASCRIPT DATABASES FIREBASE FIRESTORE FAUNA PLANETSCALE AWS ECS EC2 FAUNA FAUNADB GRAPHQL SQL POSTGRES POSTGRESQL PSQL MYSQL MONGO MONGODB
Timestamps
00:00:00 intro
00:02:01 overview
00:03:26 what are the types of database?
00:06:15 where do I host my database?
00:10:01 how much of the concern do you want to own?
00:17:22 please don’t use mongo
00:25:02 database functionalities
00:30:05 firestore rant …
00:41:04 what did we learn so far?
00:44:08 firebase risks
00:47:31 what about bleeding edge???
00:50:47 hasura and faunadb are bad ideas
01:05:02 answering chats questions
01:06:13 outro - Věda a technologie
"MONGO" is an anagram for "OMG NO."
Lmao wow
I am using it just to learn it. Why is it so bad?
🤣
@@riccardotoniolo9604 This video goes over why Mongo is recommended against at 17:22. To reiterate, DocumentDBs have valid use cases, however their major problem is that they make relationships hard to handle, which makes them an inefficient tool for most applications, because lots of problems require relations.
😮😮😐🤭🤭
Just remember ya'll there really is no such thing as making a "wrong" choice. If you went with Firestore, Mongo, or any of the others spoken badly about that does not mean you need to rewrite everything. Some solutions may be suboptimal but all that goes down to trying out technologies and learning the intricacies of each solution. For 99.9 percent of the projects, they will never reach the scale of teasing out some of the problems with individual solutions. And if that .1% happens, take it as a learning for next time.
Yeah. Just chalk it up to experience. Using the wrong tool for the job is part of learning what the right tool is.
There were some incorrect facts about Hasura:
- It is not proprietary, but Open Source Apache 2.0 and MIT licensed
- You are not locked into proprietary infrastructure, you can host it yourself, with anything that can run a docker container (Kubernetes, Cloud Run, Railway, Render, Digital Ocean, AWS, etc)
- It connects to standard databases: Postgres, MS SQL Server, MySQL, Big Query, etc, so you can replace Hasura at any time, and use Prisma if you prefer.
- It provides both GraphQL and REST APIs for accessing data
- You can put a server in front of it anytime, just for some router, or right from the beginning, and treat it has a GraphQL abstraction layer over SQL, similar to what you would be doing with a non-standard ORM interface.
We are hosting Hasura ourselves and it is still a bad idea from our experience, but for entirely different reasons, where it seems to have a weirdly bad interaction with postgresql and pgbouncer, and the specific way in which it handles roles and schema separately from the database is really annoying. From what I hear, postgraphile is nicer.
We're replacing it with postgREST for the rest API bit and pg_graphql for the graphql query bit for those people that liked it. We haven't run into any scale issues with postgrest so far, and running on essentially the same stack as supabase is generally comforting.
I understand your issues with mongo, but there are reasons to use mongo apart from relationships.
- You cant treat relationships in DocumentDBs the same way you do in SQL based, the base for the relationships are different, in documentDB you design your relationships in a way that makes it easier to query and get the data you need for that query. based on your example you store the user reactions inside the comment with the user info inside the reaction object. If you don't need to update the information in a sub collections very frequently then separate them in a new collection otherwise if that data is mostly read them put them in the object it self.
- If your data structure is not consistent 1 object has 3 fields others have 5 for example, in this cases SQL based DBs are not the right option for you.
I may be wrong here but it seems you are looking at document databases with the same mentality and structure as you do for SQL based ones and that is the wrong way to go about it. In SQL based databases you store the user in a table and you create relationships to other tables to point to your user, in document databases you can put the user info inside the object you want to show. In you example the user info could be inside the reaction object, the comments object also has the user info for the user commenting. In documentDBs you dont separate data based on their relationships but rather based on the info accessibility. If you need to update the related info very frequently separate that into a different collection otherwise just put it inside the document, the concept of a single source of truth does not exist in documentDBs.
In my experience most of the codebases I see with mongoDB that have problems is mostly because people design their databases based on the SQL principles and dont want to repeat info inside the databases. Imagine a marketplace with 10M products, if you put this into SQL you will have a bunch of tables that need to be queried every time you want to list 1 or more products, if you use documentDB you can include all product info inside 1 object and get the catalog with 1 simple query, on the other hand if you want to change the name of a category in the product it is easier to use SQL, you change it in 1 place and all documents are updated, on a DocumentDB you will need to get all products that have that category and update all products info. Yes it is more complex to update but it is way faster to get the info. As a rule if you need to read 100 times for every write you do, them documentDB may be the best option for you.
What is your opinion on using SQL in the back and then cache the more "high level" results in Mongo, like using Mongo as a cache. All mutations go to SQL, then Mongo updates based on the new changes from SQL. Queries go straight to Mongo.
@@seanchen9771 I would not use mongo in front on a SQL database, unless you have a very specific situation where that makes sense, if you want to use cache use Memcached, or even Redis in front of your SQL database, another solution is to store the query results in elasticsearch. There are tons of solutions for caching SQL databases out there.
"our data structure is not consistent 1 object has 3 fields others have 5 for example, in this cases SQL based DBs are not the right option for you."
What? is this serious because if thats your problem just create 5 fields then just return 3 fields if thats the only thing you want.
{
id: 1,
name: John,
comments: [
{
id: 1.
description: Sample,
}
]
}
This kind of structure can only be found on simple backend. In real life example, comments are not tagged in users alone because its tagged in thread which has multiple comments from different users. How can you achieve that through documents?
@@eizen8774 in such a structure and if you follow proper SQL based design rules the comments in your example should be in a separate table, and you would have a table that would relate the comments with the users with many-to-many relationship.
Indexing array values in SQL is expensive. In a document DB you would list the comments as a collection and ever document would have the information regarding that comment inside of it including the user info:
{ id: "comentid", description: "your comment", user: { id: "userId", name: "john"}}
If there is another comment you repeat the user info for that new comment. A simple query like get all comments where user.id == "someUserId" is quite easy to write either in mongo, dynamo or even firestore.
It is just a different paradigm, it works in a different way you should not think about it the same way you do when structuring your data for SQL. There are good and bad use cases for both SQL or document based DBs. In general if you just need to get information about a certain document (for example products in an ecommerce scenario) if you have a catalog with 100 products is fairly safe to say you can do it with SQL but not at a big scale, those joins you use in SQL to grab related info will compound really quickly when you start entering the 100k to 1M marks specially on reading. This would give a much longer discussion than this comments allow to... But if you need to read a lot it is better to avoid dealing with relationships on every read.
Bad luck, he doesn't reply criticism only compliments. Proves he is very ignorant.
Hey Theo, I am an engineer working in the core FaunaDB team and I think you have gotten couple things wrong when describing Fauna. First of all, you can put a server in between your Client and FaunaDB nothing is preventing you from doing that. Second, Fauna is not like Hasura, yes we do have a GraphQL api and you can choose to connect to your database directly from Client but you don't have to. Fauna cloud offering is no different that DynamoDB or Mongo Atlas cloud offering. And I am not sure why you think you can't control data permission with Fauna.
I did get the feeling that he just threw Fauna into the Hasura category without fully knowing what it is
He talked random nonsense of mongo too, then proceeded to draw an extremelly dumb relational db diagram, he is clueless, probably just wanted to convince his audience of new devs that he knows stuff
@@matheus-kirchesch absolutely YES, and it is a reason why dislikes were extremely usefull on youtube.
@@user-sh3uq1ct8t he has 87 dislikes right now , so what are u on about ???
@@musashi542 People people don't dislike anymore, since it's pointless, since it doesn't show up. You can only get it through extensions (and the numbers aren't even correct), which very few people got through the trouble with, because (going back to my first point), people don't care to dislike anymore. So, yeah, that's why the dislikes were extremely useful on youtube.
For MongoDB or any other NoSQL, you design the database based on the questions you asked, ie. your queries. For SQL database, due to its relational nature, you design based on how the data relates to each other.
For NoSQL, designing data by questions means denormalising a lot of the data by access patterns. This also means that there can be data duplication, and you have to account for that upfront in terms of data cleanliness. That means more work that you might need to do for benefits that you might not need. You also giving up the very flexible nature of SQL for this.
The benefits of doing that is increased read throughput since all the data that is accessed together is clumped together. Is it worth it? I would say it depends on the business and should be a business question whether it’s worth it or not.
Now, can this be implemented in a SQL database anyway? Well yes. But the nature of something like MongoDB is that it’s much more seamless to use it this way since it’s designed around that.
Now, when you approach on using MongoDB like it is a SQL database of course it going to be complicated. It will feel like a hack rather than what’s it’s designed to do.
In other words, don’t use MongoDB like its some sort of a relational DB. It will just hurt.
Totally agree. I've been convinced by some very smart people that Mongo is ROUGHLY in the same category as Dynamo of "if you know what you're doing, good shit. Ignore my video and do that"
Real talk tho: how many Mongo devs have you seen using Mongo correctly? I literally never have in my life 😅
I think one particularly interesting thing that rarely gets touched upon here is that there are two subtly different way to do relational databases that are VERY different from each other.
On one hand you have traditional ER diagram models that naturally end up in 3NF. On the other hand, which includes the largest actual data sets, you have completely denormalized uses of relational DBs using star schemas where the dimension tables form a common bus (Kimball architecture), and where the data set is *a lot* easier to query and a lot easier to add/remove tables to/from, but where you have slightly weaker enforcement of constraints due to being denormalized.
Dimensional modeling and fact tables is imho a good fit for many cases where people go with noSQL just to denormalize. You do not need to abandon relational databases, you can just use them differently.
@@t3dotgg We use mongo at work and for a solid year we had good API design and we were able to fetch data logically. At some point it started getting so convoluted that it was far easier to work with TS and filter data with TS than it was to make a query. This was simply a skill issue.
@@t3dotgg In a volatile environment where it's difficult to know the schema beforehand I find it much easier to work with Mongo (or any document based database). At the company I previously worked for we had an entity called an Order. This Order had so many sub-entities linked to it that it was basically more than half of the database itself (40+ tables just for this entity and its subentities, and we might have had to add more). Any query, any modification of that Order was a huge pain in SQL because you always had to use all the subentities. The same thing in Mongo would've have been incredibly easy
You cannot write in the same sentence "NoSQL" and "design" !
I actually think MongoDB is far on the left of the spectrum and yes almost all applications data do have relations.
But MongoDB is NOT about having no relations. People who complain about MongoDB either embeds all relational data into a single document(this is not even possible to due to single document size limit of 16MB) making queries impossible or normalize all the data just like relational databases, ending up with way worse (JOIN) performance.
The super cool thing with MongoDB is that you can start out with a straight forward relational database like normalized design and can incrementally optimize according to the application needs.
For example, in the case of Users and Comments, yes u would definitely store comments in separate documents. There is absolutely no reason to embed ALL the user's comments in a single user document. BUT if you have an API that fetches user data with the recent 10 comments and this API is getting a ton of requests, then based on these metrics you get an option to duplicate and embed the recent 10 documents. Now you just have to fetch the user document once with no JOINs, its just fetching it from the SSD and done.
And you still want the comments stored in a separate collection, so that you can fetch older comments. Now this performance optimization leads to data duplication, and yes this can be quite challenging to manage, if you dont know what you are doing. But as I said, you can start designing your database in "normalized relational style" and evolve your schema according to how your performance bottlenecks. BTW I find designing relational database very straight forward. People seem to think its more difficult cuz you have to actually create the tables, set up the foreign key etc.
Also for usual applications, you have way more reads than data mutations(create, update, delete). Think about a blog post, that shows the number of comments, three recent comments, and the writer details. While you write the blog post once, maybe update it a few times, hundreds or millions of users could read that blog post. Imagine you have a relational database for this. You would either have to do all these duplicated JOINs, which will definitely kill your database or you would have to add another cache layer, just to solve the database issue. Now with this cache layer, which is very common with relational databases, even with a relational database you have duplicate data and have to manage cache invalidation(which is somewhat similar to duplication management) So even though data duplication can be additional work for the developer, its super beneficial once your app scales.
Also comparing PostgreSQL JSON column with MongoDB is really not fair. If you look at SQL, its "API" is optimized on reads and does all the data processing on every "Read", on the other hand MongoDB focuses on data mutation. You get to update the complex JSON data with a single update atomically.
In fact I think SQL(relational) databases are more useful for data engineers/scientists/analysts. For them its not about getting the data fast. Unlike the application, the set of "read queries" are not fixed and can change a lot. Well, in this case, it would be more like columnar databases such as BigQuery and Redshift rather than traditional database. Im talking more about the SQL in this case.
This all makes sense. Thankfully solutions like Vitess on MySQL can keep SQL read perf pretty nuts.
I'd put you in the category of the DynamoDB folks fwiw
@@t3dotgg Interesting, looks like it enables horizontal scaling with sharding to make it fast. I should take a look at that.
I was always very skeptic with horizontal scaling with relational databases.
I was skeptic, cuz one of the reason why RDBs' queries are fast, is because all the processing happens in a single machine. RDBs are very old technology, built for vertical scaling, which was more than enough at that time.
However, if you look at how sharded cluster works, unlike replica sets, each shard is a separate machine and has unique set of data. Basically, the rows in a single table is stored in different machines(shards). All the data processing(JOINs, GROUP BY, ORDER BY etc) can no more be done in a single machine, leading to a ton of network request just to process a query. So even though it could improve writes(more primaries), I was skeptic with faster queries
And as far as I know, another nightmare is that if you were following the convention to make primary keys with AUTO INCREMENT.., you would have to switch them all to something like uuid, cuz u have multiple machines that writes to ur database at the same time, meaning that unless the shards communicate each other to make sure that AUTO. INCREMENT is "incrementing" to a unique value, which degrades the benefit of horizontal scaling.
why bother dude, for me whenever someone repeat the famous misconception about mongo "not relational" it is end of discussion, they dont care, you will not convince them to invest a single unit of brain energy on the subject anymore. And for them it is like mongodb magically vanished and dynamodb is the only nosql left.
Just like they convinced the beginners to learn mongo at first in the couple of years ago without telling them about the big bottlenec of connection pooling, now they will again make a mistake of recommending against it while it is the right time to use mongo since now it is optimized for serverless and every use case.
MongoDb still the good choice it have lot of functions to relate the documents and we can use populate for get those relations it has lot more functions like that.
As a senior dev, I have to commend you man. Your chats are great. Even if I already 90% of the stuff I still listen to you from beginning to end.
You really didn't sit down and watch all 1 hour of this
@@windwardhive7363 a lot of people watch long form content, for a relevant example i watched all of theos 4 hour ama yesterday
I feel like I need to say something.. The idea that document dbs are completely a functional subset of sql dbs is not entirely informed. The functionality of Mongo aggregation queries _cannot_ be fully replicated in SQL - Take the “Users” and “Comments” example, you would either need to join Users to Comments and pull the same user record for each comment record (denormalized, and therefore larger result), or do one query for users and one query for comments (normalized, but multiple calls to the database). In mongo, if you had the same setup (a Users collection, and a Comments collection, just no relation enforcement) you could join the Users collection to the Comments collection with a $lookup aggregation and pull down a completely normalized result in a single query. Wether or not/where this is useful is up for debate, but it is certainly something a document db can do that a sql db cannot do.
Great comment!
With CTEs, jsonb functions, etc. Postgres could 100% do this "only a document DB can do this" query with a relational model. Is it as easy as pulling out of a document DB, no, unless…
…you store documents inside jsonb columns, in which case it's about the same complexity and speed as MongoDB.
I am not very familiar with CTEs or all of the josnb functions available to postgres, but perhaps my example is not a good one and the boundaries are being blurred for specific implementations of SQL and specific functionalities of mongo’s aggregation pipeline.
My main point is that the aggregation pipeline is not a functional subset of SQL; $lookup is but one of the many operators available, and I’m not convinced that SQL can do everything that a document db can do
@@the-avid-engineer Then the focus should be on finding a query pattern that document DBs can do-or do substantially quicker-but something like Postgres can't in a timely fashion. So far, I haven't seen such cases, but my expertise is in SQL engines, not MongoDB.
An example outside of this discussion, Graph databases, shows how optimizing for the connections rather than the nodes can provide substantial performance wins (though recursive CTEs are still able to do the same tasks, only orders of magnitude slower).
20:00 Jesus, if you design your data structure like that, congratulations on not forgetting how to breath, but don't blame mongo for not knowing what you are doing
yeah hes kinda dumb hahaha I already stop watching on that part..
ikr, like you define a normalized schema and you except to work. it's the whole point of using mongodb to have denormalized data as much as possible to ease scaling...
This is just a reminder to never blindly follow tech youtuber advice without a grain of salt. This is absolutely NOT how you would want to use mongo.
Thank you for this! The video made me pause/think more deeply about my choice to go w/ Mongo for an indie project I'm working on, and I started writing lots of code to handle relations, even after it started to become apparent not the best tool for the job (but was too lazy to want to do anything about it). Nothing against Mongo, I need it for certain applications, but this video helped motivate to bite the bullet and switch over to relational db, and no question it was the right choice after taking the change. Thanks again!
One of the advantages of document based databases like MongoDB is that you do not necessarily need migrations, instead you can use different schema versions of your document simultaneously and manage the migrations in the application layer without the need of running and syncing huge migrations at once. In relational databases on the other hand you need a migration for every single field you add rename, table you add etc. Also the absence of foreign keys (and migrations) is perfect for sharding and partitioning and therefore scalability. I think in relational databases migrations tend to be rather painful, since you are forced to migrate your schema and need to sync all your nodes. I really like the video, but the MongoDB bashing seems a bit too subjective to me, and I assume you did never design a sophisticated document based schema. E.g. in a real world document schema you would probably not add all comments to the user collection unless you expect not more than a few comments per user and also need the comments to be available everytime you load a user document. There is an old talk from one of the creators of reddit here on youtube in which he explains that they had to create one giant, flexible table for almost all of their models in order to ease migrations and scalability, which for me sounds like they tried to replicate a document database with a relational database. I think you should choose your database depending on the requirements and there are pros and cons for both concepts (many more than I've described).
He doesn't seem to know that Mongo has references.
Mongo literally can be a "relational" database. That second diagram he did for SQL is Mongo too, the User object can have an array of commentIds, then to get those comments you just pass those IDs to the Comments collection
And he made the worse model possible for his problem.
@@carlosemvqueiros yeah he doesnt know shit about mongo.
I agree with what you said about sharding / scaling out horizontally.
Maybe I'm just wrong, but wouldn't it be much easier to just do one huge migration rather than needing all these "what ifs"(which I assume you are persisting forever) inside the application layer to make sure your schema version is up to date? And then, when you start adding new features, you have to always take into account that some document might have an older schema version.
I think his main point was that relational databases are faster and more efficient in production, especially for big data. I do think it's silly that he says you shouldn't use MongoDB because of that, as not everybody is trying to be big tech/data (even if you were, it's probably not too hard to transform your data to fit into a relational model by the time there's any real benefit to swapping).
I almost exclusively use MongoDB because of how easy it is to get up and running. I don't want to sit here for an entire day+ every time I work on a new project to draft up some ER, etc, diagrams, just so I can start working on it (All so I can save some CPU cycles and queries? No thanks).
Neo4J is amazing for analysing data that can be conceptualised as graphs. I was doing some research-like work (investingating RNA-protein-disease-pathway-compound networks), and it was so nice for graph searches. Though it does require you to re-conceptualise some things (say, large enums, such as caregories of functionality) as abstract nodes - but when you get it right, you can zoom around the network SO fast! It's great. But yeah, I wouldn't use it for an app, unless I had a very good reason to
In terms of NoSQL, the main issue was you still have relational thoughts on document-based design. The NoSql design was not correct.
Shared this with my dev friends, I think this was the best hour of content on this channel so far 😎
It's looks like you've never done MongoDB property before. You can do relationships and use "aggregations" which is much more powerful then anything SQL can output.
Yeah, I think he was trying too hard to sell his points at that point lol.
Yeah, take a look at the relation structure he designed at 21:00 the man is a living joke trying to teach others.. fuck sake
yup hahaha hes a dum dum
Did he actually say you can't do relationships? In his MongoDB example(timestamp in description), he seems to understand that MongoDB can have relationships. I do remember reading, multiple times, that Mongo's strong suit is querying for a single document rather than multiple.
I think he puts way too much weight on speed though, as I highly doubt he recommends writing out a server in C++, or even lower-level languages.
I would love way more backend content from you, Theo. I find that I learn quicker from your explanations than I do from most other content creators and I am way weaker on backend than I want to be.
Theo you are dev youtuber we didn't know we needed, but we truly appreciate!! Def not for beginners but there are a bunch of other great channels for that need. Thank you for being constant at being you and putting yourself in a video! lol
also for beginners too that understand first principles & the presence of tradeoff
true
Your talks are perfect to listen to, I'm almost treating them like podcasts and I can learn some interesting things I wouldn't have thought about. Thanks for creating such a a great content.
Thank you a lot for your insides, been waiting for this a while. Please more on databases and why to choose them and the ease of use in comparison to the feature set. 👏🏾🙏🏾
Thank you for making these videos! I’m trying to learn web development and your videos have been very helpful!
One argument I'd like to make for mongo is that You can use mongoose ODM to map relationships between datatypes with the 'ref' type and the related object would have the same ID while remaining in different tables. I'm not that experienced but I'd like to know if you have actually worked with mongoose.
Yes this. I don't know if there's any downside
I totally agree wrt mongo, firebase, and fauna. But not so much with Hasura.
1. Hasura gives you a default GQL API for crud operations and basic permissions, making easy things easier, but this is JUST THE BEGINNING. It makes hard things easy too.
2. You can add your own custom GQL mutations/queries implemented as an http endpoint. You can do anything, including accessing your the SQL database directly with custom SQL queries.
3. In addition to #2, you get *** tRPC *** -like functionality with the "hasura actions codegen" command with the "typescript-express" template, so along with other GQL codegen tools, you can call a server side function on the client side as a direct function call.
4. Hasura allows you to extend your GQL schema with another remote schema. For example, you could extend the Hasura schema with an Apollo Server for your more complex cases.
5. Hasura has remote webhook triggers on CrUD operations, so you can run custom code on data changes.
Once I had my tools configured, Hasura made easy things easier, tedious things fun, and hard things easy (or at least no harder than other means).
💯
I agree 100%, I have been using it before 1.0.
It's flexible and can be integrated with actions and events to lambdas or anything.
I host my own instances, never had an issue.
Rock solid 👌 💪 🙌
true!
Thank you very much Theo for the video about DBs.
I wish someone had told me all this years ago (of course none of us knew).
It seems that almost all of us have suffered the same path of adoption, hype, disenchantment and technological debt.
It is sadder in cases like mine where despite having more than 10 years of experience with SQL DBs we jump to Mongo DB and other document-based databases as if they were a panacea.
Regardless of what you say, there is a MongoDB usage pattern that we use a lot in particular: A SQL database with a relational model with a centralized Back Office and Mongo Servers in the different regions in which we model the data exactly as the views (normally from ecommerce) require to paint it (as a "complex document cache of JSON", usually with the SLUG as the collection's primary key). In this way a getServerSideProps (to put it in terms of NextJS SSR) only has to do a findOne (by id) to a MongoDB Server located in the same region to fetch all the data required by the “view” (and some times throw query’s to overwrite some of the data with user’s personal pre-rendered information).
But I wouldn't recommend anyone to use Mongo's indexes and aggregations to do Joins and expect it to scale. Not to mention the architectural disaster of sharding and clustering.
I think the switch to document-based databases is due to the highly sped-up development time, originally I thought it was silly, however, it is much easier than having to draft up ER, etc, diagrams to even get started. I don't think most people are pushing their hardware to the limits anyways, and it seems like it should be easy enough to transform your document-based data into relational data if/when the time comes.
Why do you think sharding/clustering leads to "architectural disasters"?
@@adrycough computers are also so fast nowadays that even inefficient DB architectures are good enough for 90% of people
Why don’t you use mongo partitions? And caring about sharding so much?
Some of your sentences and paragraphs transverses years of education and experience. There are only so many people that are truly a gift for youtube viewers like this guy right here. Thanks man!
Well, ya gave me the confidence to not think I need to use Firebase whenever I come around to starting a side project. Also, the Railway website looked tight to get some stuff up and running quickly. I definitely struggle most with figuring out authentication and CI/CD.
I am just so thankfull for Theo existence, he explains things so well and shares best use cases for everything. Just thank you!
Dude I have been thinking about this for so long! I always liked SQL, it works great for every project and there's ton of official documentation and forums out there, way more than Firestore alone. Besides we get more compatibility with legacy systems and you can still control your APIs and make the necessary changes as they come. Wow It feels good to get that out of the chest hahaha
🔥Firebase sounds amazing, can't wait to try it out! Thanks!
Excellent video and excellent comment section. Thank you to the engineers in here adding their own experience & clarifications to the discussion.
Thanks for the vid I was already on the "I run my DBS on bare metal", but as dev u heard of lot of new tech everyday and you cleared some shit of my mind thanks for that !
Did you try Neo4j? I think it's awesome. But i guess you only see the full potential in apps needing graph search algorithms (like a social).
The cypher language is much easier and intuitive than it seems at a first glance
Yo, I have been watching this videos and on twitch for some time now, and I love it. You are so unhinged is super relatable. Keep up the good work man.
I've used Mongo fairly heavily. I wouldn't knock it that hard. You can band-aid a wide variety of its problems. It's better designed for more monolithic data, obviously, but there are a ton of things you can do with Mongo that feel simply magical. This kinda all goes down to a data storage level, though. It's a special use tool that people outgrow because they can prototype so quickly and end up stuck with a prototype their management doesn't understand wasn't designed for production.
watching this as I am helping to build an application on Firebase using Firestore (previous CTO decision). Very helpful, will keep the issues in mind, and try to isolate all of the database components so that it will be easy to move off later when we re-build the application. Will look out for costs and security.
Thanks! You just saved me from learning Mongo for an app when I already have a decent grasp on SQL!
From a Data 4 Analysis person the answer is always PostgreSQL. The way you can use Partition Windows is perfect, better than Oracle and M$. I know this video is not about representing any SQL DB, but you gave me so much ammunition against Document Stores that I have to thank you. I now understand that the message system that we were trying to build in MongoDB will be done better by simply using a K/V in the cloud or Redis on premise. Utilize technology without locking yourself into a platform. Robust yet nimble. I can get behind this concept of builders instead of endedness.
Just started watching this right after choosing firebase for my next project. Was able to switch to vercel etc once you pointed out the flaws in some of the things related to firestore -- thanks!
I've been a programmer a long time, and this is by far the best explination I've seen of KV/Document DB / SQL. Love the Venn diagram explanation. That's so true. Funny that I've never seen anyone else explain it that way.
MongoDB is legit my go-to for pretty much anything and now I'm feeling worried to hear why it shouldn't be...
Honestly for me too. I don´t think most of the points he had are a problem with mongo specifically rather than people treading mongo like any relational sql database and designing a mess. You can build horrible sql databases too. In fact when I started working where I do right now the database was mssql and it was slow af and it was a mess beyond "fixing it with migrations". To be clear not saying mssql is a mess, the database model they conjured up was bad as nobody had any idea what the program is actually REALLY gonna do at the time of making it up and leaving all the type decisions to an ORM is not ideal either. I do agree that too many people jump on hype trains for no reason. But imo mongo is legit a very good db, with very good sdks (although i mostly know the c# one, the c# one is great though) and it does not vendor lock you to a specific cloud.
@@shioli3927 Yes! Thank you. After watching the video I honestly thought his issues with MongoDB seemed a little offbase. Yeah, MongoDB is bad when you use it for things you shouldn't use it for. That's just a redundent statement.
Need a LOT of relational connectivity? Well SQL isn't your friend either, and a graph DB is what's appropriate. Good luck even getting your query right with its ridiculous joins, never mind the performance.
Again, "don't use a thing for something it's not very good at" is not compelling.
I get the point that, yeah, as a project grows often it turns into a case where SQL might be the best choice. I would just argue that you can re-frame a lot of use-case problems and make a document DB not just fit, but actually be really good choice over SQL. Where rapid iteration and flexibility are concerned especially, SQL just doesn't make sense.
As a last comment it feels like he framed his argument around the idea that document DBs have zero relational capacity. You actually don't have to dump everything into a single document type. You can certainly store ref ids so long as it's not an attempt to overly flatten your model. Yet again, don't use it for what it's not meant for.
Thanks for so thorough review!
learning so much. thank you!!!
Nice to see there is still a strong love for SQL in the nodeJS community, I am primarily backend JAVA developer who's thinking of switching to NodeJS on the backend, I am still on the fence. But this rant opened my eyes to a world of all sorts of crazy stuff! WOW!
For my first big project I ended up going with postgres on EC2 and i wish I went a more managed route.
One of the strangest ones i've used was a google sheet that i web scraped to manip data because i was too lazy to swap from a spreadsheet to something actually good and ended up just making more work for myself.
Haha there is atleast an api if you for some godforsaken reason wanna use google sheets as a database
@@abbe00abbe i remember using it and it being very janky to do what i wanted to do so just webscraped instead
def wouldn't recommend.
Ah your videos are honestly always so good
The best class I took for my MBA at Carlson was Relational Database Design in the late 90's. If business suits can learn third normal form, coders can. ;) Career-changing, and hugely influential in making decisions like this.
Thanks for this rant sir! This helps me a lot even if I need to do some more database homeworks yet again 🙄😅...
There is one issue for folks trying to build something quickly with SQL. It is offline first app. You'll need a DB on the device and another one on the server and figure out how to sync them. I hope a solution exists for quickly prototyping an offline first app. I believe MongoDB Atlas and FireStore are presenting themselves as a solution to that.
This so much. I tried to write the syncing code myself but oh god what a clusterfuck. Firebase solved it for me.
@@mekelius hopefully SurrealDB does it right.
Ive been using firestore as data transport between the backend and the frontend rather than a "database". The backend only writes to it and the frontend only reads. This also helps keep security rules pretty minimal. If I want to get rid of it cant I just choose a different transport layer?
You should take a look at databases that support the OpenCypher query language. Thinking in graphs comes just so much more naturally than any ORM or SQL.
Thanks for this video, I appreciate it!
This was great, I've used most flavors of SQL but never heard of half of these and always had some vague feeling like maybe I'm missing out. Now that feeling is gone :) that firestorm and fauna stuff seems quite hellish
Used Fauna at a few startups. It was cool at first. All the easy stuff worked fine. Right when I started getting into more complex logic, it became a nightmare to work with.
Hows it compare with SQL?
@@lutfiikbalmajid same experience... plus it's just so annoying to take up mental space with learning it's syntax, that does not translate to anywhere else
@@lutfiikbalmajid The syntax will break your brain lol. You're better off with SQL. I personally switched to Dgraph because I wanted to keep GraphQL but Theo hates that too lmao.
@@anyadatzaklatszjutub Lol FQL gets insane. There's a lot of cool stuff you can do with it but for what it is, I'd rather split apart my queries in a better designed language
@@Khari99 to be clear I really like GraphQL - I just like to use it for what it was built for. Not as an SQL replacement. I ranted about this on stream yesterday lol
Hasura is opensource.
You don't need a proprietary client for hasura, the datastore is just postgres, it's not Proprietary.
I have a decent experience of hasura in production, I can talk a bit about the not so good parts, they are not the one you talked about.
Except the authentication part, it was a bit restrictive to have to use JWT because hasura only works with JWT, where our application has no need for it.
Permissions of hasura has been not really an issue for us, the column level permission are very granular and expressive.
We selfhost everything, optimizing query is doing postgres indexes it's not that abstract, using hasura does not mean you can't use a server too 🤷♂
Bad points for us was:
- easy to write complex queries, it's harder to realize the cost of things
- the given console can be painfully slow, have not seen much love in the 3 years of using the db
- JWT only
- Subscription are NOT FAST and scale super bad, use with caution.
I would not re-pick hasura for my other projects today, but i had a good experience with the technology and it allowed to move fast when we needed it the most.
I still see it as technical debt
Hi GodOfMacro, thanks for your comment.
What alternatives to Hasura would you pick for other projects (if in need for complex queries and subscriptions)?
Thanks
@@jorgemacia9902 I do not have experience from a successful "generic subscription" tool, I would say, not firebase haha.
I tried pocketbase, it's not ready for production, the subscription is not reliable (updates sometimes skipped / missing).
If I had to do it again, I would do subscription myself directly.
@@GodOfMacro Great, thanks!
@@jorgemacia9902Postgraphile
Thanks Theo, this is super useful and timely content as I am thinking about the stack for a serverless sveltekit / graphQL project. What do you think about running Prisma in conjunction with Supabase as opposed to Supabase's built-in `pg_graphql`?
Mongo is great but you need to think how to dessign your collections. You can't just put everything into the arrays of the documents. There are some limitations to documents and developer should know those limitaitons and design everything properly. There are relationships in mongo but it's quite different than what we have in SQL dbs
Ive used Postgre for like 14 years in big expensive mission enterprise apps, whiches last minimum 15 years in production, and they have really stand the test of time. Rock solid in every aspects.
Btw, you can put AWS Aurora Postgre/MySql in the managed DB category,
firebase has an admin sdk for many languages which makes it really easy to work with if you have a service in python or go or whatever. also it can be very convenient for small projects, you just pay what you use (even supabase is 0 -> then 25$).
Thanks. I'm new to cloud, so it was helpful.
Quick question: For my project, I was considering using a remote file system (like AWS S3) and it seems it would fit your definition of database.
As it's mostly to store logs (a lot of writes, not a lot of reads), I felt like it was a valid option.
Am I too old school ? :)
For logs and "dumb writes", S3 is a totally fine option! There's a few other things like this (Snowflake and BigQuery come to mind).
Generally I only care this much about "active data", like the things that put my business under if they fail. I feel significantly less strongly when it comes to what you're describing
Interesting explanation about functionality subsets of DB, it is more clear now for me which solution to pick! Thank you for explanation! KV (small features) -> Document DB -> SQL (all features + standard = best)
Your video saved my time. I mean a lot. Thank you very much. I subbed u from now.
I used to have mongodb in my project, but later, i think mySQL is still best. And there are so much job in indonesia for SQL db rather than noSQL. For playground, fauna is good and easy to integrate. There are also CouchDB, database that can connect with rest api
Just for the sake of simplicity around 5 years ago I started using NameOfTableId to describe foreign keys with the lazy purpose of seeing the field and automatically recognize it as a foreign key, still use that pattern today on my sql databases, but I didn't remember seeing anyone using like that before.
Anyway happy to find a kindred soul
"Almost EVERY data problem has relations of some form" - THIS! Knowing this can save a lot of pain in projects.
This answer is always PostgreSQL (if you don't know the problem). Its easier to migrate that data/tables into other DBs. You can recruit at your local college or universities and find someone that can extend postgresql into some other popular Db.
I do agree that with mongoDB it is not as seamless as in sql to define relations between tables. However, I still use mongo, because it is very much possible and mongodb atlas comes with the ability to perform (potentially very complex) fuzzy search queries, which i need in almost every application I work on. Saves a lot of time not having to setup things like elasticSearch or using half-baked postgresql extensions
Your best video so far! When do we Get the Nestjs rant?
"they kinda suck, but they dont suck that bad" :D i love it. Thanks for another informative video theo!
I think it's pretty clear that MongoDB is not for all cases; well, nobody ever said that, not even the MongoDB creators. Another important point not mentioned is how well MongoDB's DX compares to SQL. MongoDB can be used out of the box, while no one will use PostgreSQL without adding additional libraries/layers, simply because pure SQL sucks.
Flexibility is another thing it is designed in mind. It doesn't not limit you or insist on certain conventions. But if one needs data validation or transaction support at some point, one can add that with relatively little effort, there are also a number of third party solutions for various use cases. I agree though that only certain projects will benefit from this, say, I'd use it for a medium sized project with a rather simple data structure and that needs to be prototyped fast.
I like arangodb a lot. It's super underrated and worth checking out. Open source multi model db (graph too) and very dev friendly. Though expensive if you use their managed hosting - much cheaper to host it yourself on aws or gcp or something.
Im slowly coming back around to ArangoDB
NOOOOOOOO, pls noooo
@@azdy68 😂😂😂
is sql still better for something like a chat app? or something with a lot of data being exchanged? I know discord uses something like casandraDB for example
Also You said at one point that the 10gb that planetscale gives you is a lot of data since you shouldn't be storing images or data intensive files like that in a database anyway, but what about text based content such as chat messages? that can add up right? I made a random calculation: if an average user sends 1500 characters a month and a character is 4 bytes, then it would take 1,790 MAU to send 1gb. after 10 months of back storage you will have reached your limit and then it will cost 0.14 cents per user per month for storage on planetscale which is pretty low i guess to be fair. But is it the optimal use-case?
I am a beginner to databases I would say
thank you very much, that was very informative.
Hey Theo, loved the video! What database would you recommend for a social media network if starting from scratch using the latest database choices on the market today?
Old is gold, I love MySQL.. and now comes SurrealDB, such a dream
I'm getting frustrated with supabase client and looking at moving off to prisma ATM.. looking tricky given auth and schema limitations in prisma, and hackyness with RLS. Kinda at a 'burn it all down' phase and thinking of next-auth, prisma and just hitting postgres directly within next APIs. I've quickly found I'm not using the supabase client a whole heap on frontend anyway, and leaning heavily on next API routes.. speaks to your comments around APIs being good.
The irony is that I'm very much a backend leaning dev and went for supabase (running locally) for batteries included to focus on learning frontend again... little did I know 😅
I am not a fan of third party clients like firebase and Supabase fwiw - I mostly recommend Supabase due to it’s Postgres underneath. Prisma’s a great bet 🙏
This channel is a gem for beginners!
great content was very interesting to watch and think about, thanks!
Seen this video twice, always learning some
I reject the assertion that DynamoDB users are smarter than you and probably aren't watching you anyway. I've been in a DynamoDB bubble for many years and watched this to see what the latest trends are, and I learned a ton. Thanks!
Hey Theo, just found your channel and I’m loving the videos. I actually got fired a few weeks ago for arguing with management about some database decisions they were making.
I still regret getting fired, but this was a nice sanity check that my concerns were not completely invalid.
You probably got fired for the way you communicated your disagreement
@@MysterCannabis lol. Probably some truth there. I was mad at them for a couple reasons, could have handled it better.
Next time management wants to drive off a cliff I’ll just keep my mouth shut. CMA moving forward
As a firebase/firestore user there are gooduse cases for the platform, some of them are not provided by supabase because supabase is not a 1 to 1 map to firebase but my Pain with the queries, relationships, even when they have an SDK that allow you to use it in your own server and write and API if the query you bid is complex and dynamic (like a sort in dynamic fields) your call fails in runtime because you have to buil the index to. and is a hard vendor lock. At the end if is for prototyping a quick idea is good but otherwise it will hurt you.
the standards in sql is really good
what sql needs is to become more serverless for resource but keeping all the standards.
Great video thanks! One questions though. I had used Hasura in a tutorial and enjoyed it. However I did a coding bootcamp and used raw SQL queries, which I also enjoyed. However I haven't had to do a ton of different joins and difficult queries. You can use Hasura with SQL databases and I thought it just translated your queries to SQL and didn't realize that it created proprietary language on the database server. You also don't use a Hasura database so not sure what you were referring too there. You can use any database server as far as I know. So I am not understanding something here. I will have to go back and look at the database I used for that project but pretty sure the data looked like any other PostgreSQL database and Hasura was just the middleman.
REDIS stack has document database, graph and more. Fully persisted, replicated and highly available. Everything stays in memory so it’s super fast.
This is one of the most insightful yet easygoing videos about database. I can't believe I've been listening for almost an hour.
Great content! Subbed and I'm squeling!!
I have to say, as a junior developer 4 months into my first job - listening to a senior developer talk about architectural concepts is a treat, it truly shows that you know your stuff, and the presentation is amazing. Thank you!
Question: What is your favorite SQL db?
Your go to for quick/smaller projects &
The db you would pick for an enterprise.
Quick/smaller: SQLite
Enterprise: Postgres
Thanks for this chat! Didn't realize the bottlenecks of firebase
Theo, do you export these drawings into a repo? It would be great to go back and look at it as once piece.
I was wondering the very same
Thanks guys you saved me a good 50mins of my time just reading the comments first! I thought his take on documentDB’s was a bit biased 😂
I learnt the hard way with Firebase... you cannot leave that platform unless you decide to start over
Firebase has Security Rules, pretty easy to check user.auth objects to restrict a user from getting someone else's friends.
I think if Firebase is still a good choice for proof of concepts, fast to market, or small projects with low bugets.
I don't understand the dunk on Hasura. There is no proprietary client SDK for Hasura, you simply use urql, apollo or whatever. It's easy to self-host, the permission system is very flexible and "nhost/hasura-auth" provides an easy solution to get auth and the JWTs running.
That relational database diagram burned my eyes. 3/3 relations are misrepresented. I still get your point, agree with you and understand that this wasn't meant to be a tutorial on how relations work but if I was a new viewer then such misleading information wouldn't convice me that you know what you are talking about. Otherwise I really appreciate your work and I definetly learned something.
This is the video that I have been waiting for🙂
if you use mongo with prisma, you can just skip schema migration to start off, but to have sql version before going live
this is the greatest video on youtube! thanks theo!
Hey theo ! Im new to SQL and Im learning postgres. Do you know how or where I can practice more of SQL/Postgres
really cool explanation.
I find my self on the DB line hah))