What is Database Sharding, Logical and Physical Shards, Dynamic vs Algorithmic Sharding | 2021
Vložit
- čas přidán 26. 07. 2024
- This is the twenty-first video in the series of System Design Primer Course. We talk about one more important component of System Design: Sharding.
We want software engineers and aspiring software engineers to develop basics and get ready for the world of interviews as well as excelling as a Software Engineer.
We talk about the following:
Horizontal vs Vertical DB Partioning
Logical vs Physical Shards
Algorithmic vs Physical Sharding
Pros and Cons of Sharding
------------------------------------------------------------------
Recommendations
------------------------------------------------------------------
Our full courses on youtube:
✒ System Design Primer Course: • System Design Primer C...
✒ REST APIs made easy: • REST APIs MADE EASY
Some paid courses that we recommend:
✒Educative.io: bit.ly/3qnW5ku
✒Interviewready.io: get.interviewready.io/ (Use coupon code SUDOCODE for extra discount)
------------------------------------------------------------------
About Us
------------------------------------------------------------------
Created and Instructed by:
Yogita Sharma
✒ LinkedIn - / yogita-sharma-83400b55
✒ Instagram - / sudo.code1
✒ Facebook - / sudo.code
✒ Medium - / yogita088
Post-production(editing, thumbnail etc) managed by:
CiKi
✒ Website: www.ciki.co.in
✒ LinkedIn: / 74735937
Colors and design by:
Naini Todi
✒ LinkedIn - / nainitodi
Both Arpit and Yogita are software engineers and want to help other software engineers become better by providing high quality and well researched content by adding their creativity and teaching twist.
------------------------------------------------------------------
Join Us
------------------------------------------------------------------
Hangout with sudoCode:
✒Discord Server: / discord
For business:
✒Email: sudocode.yogita@gmail.com
------------------------------------------------------------------------------------------------------------------------------------
Timestamps:
0:00 - Intro
0:59 - What is Sharding ?
2:15 - Horizontal vs Vertical shards
5:46 - Logical Sharding vs Physical Sharding
8:58 - Advantages of Sharding
11:07 - Algorithmic and Dynamic Sharding
13:33 - Disadvantages of Sharding
16:06 - Some more important details about Sharding
16:40 - Outro
The greatest drawback is the "complexity" it brings in your architecture. Sharding is the last resort as you said. Nice video with clarity.
a bright confident precise and concise lesson. you are a great teacher, kept my brain neurons constantly firing. what a class! keep all these good work.
Short, crisp and explained in the simplest possible way. Thanks for the hard work you put in. It is indeed very helpful.
Thanks Muskan
Yogita, just wanted to let you know I read database sharding from other popular sources also but your explanation with example is best I found. Keep Going on !!!
Great video, nicely explained.
A hands-on video to shard data on SQL and NoSQL databases will help a lot.
Good work.
Nice explanation yogita!! 👍 Hands-on demos would be really great!!
Sure 😊 Will do share soon.
Highly interesting and useful content to a beginner. I think the basics are covered very well. Good job and thanks for the video
One basic concept that can be added before getting into intermediate concepts like sharding - is Indexing. Often, before scaling up , it's better to index smartly to reduce the reading latency. Overall good video.
This series is awesome. Watching most every vid
The way Yogita teaches these concepts is too good. 😀
Superb explanation
Great ! A hands on video will make it complete. Thanks Yogita.
Most welcome 😊 Will try!
Great explanation, the way you explained is that the concept is concise and informative, Thank you and keep going!
Nice explanation.! I like the way you try to convey without unwanted gossip and excluding intro
Great work once again!!
Please keep it up!
Waiting for sharding strategy and shard keys video!!
Very soon!
Salute to your effort to explain the concepts, would love to see hands on for one relational and one non-relational db
Excellent & clear explanation. Kudos.
Nice explanation! Great work
Excellent explanation it helps a lot keep up good work…thank you so much it’s really helpful ❤
Very helpful, thank you!
Thanks a ton for this course. Really grateful
You're very welcome!
Yes, please a hands on video will be very useful.
One on relational(any one) and non-relational(any one) DB.
Noted! Coming soon!
Having the theortical knowledge is good but seeing the practical application will give the understanding how it is really done in the industry.
@@kumaraditya9986 You are right with your suggestion.
Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?
Nice Explanation of such a great and important topic in system desing. Please keep uploading more video, these are quite informational and it is very useful for cracking system design round for FANG! Thanks a lot for your time and effort.
Thank you, I will
this will help me to understand the logic but hands on will give clear idea how its actually work
Very Good Tutorial! Simplified the concept like anything. Sharding Key is a crucial here. Choose sharding if its really needed.. not every organization needed it. choosing wrong sharding key could bring lot of complexities and wrong data to the customer.
awesome video...it really helped
Yogita,your videos are really very helpful to understand the sharding concept. I have followed the whole system design series.thanks a lot for making this series..
Glad to hear that
I respect the way you explain
Being an architect myself, I can vouch this content is gold!
Thanks for the validation Soham. Feels good to know that you liked our content.
wow, well done!
Thank you so much, Mam.It was really Helpful😁
Thanks Yogita for such a great video..really helpful..we would like to have lecture on how to shard relational DB and NOSQL DB.
Excellent content and explanation.
Great job :)
Great content would like to see relational db sharding as a hands on first.
best sharding video
great explanation
Your explanation is very good but your rate of speech is very fast and maintain a positive tone when you are explaining so that the viewer feels easy to digest the data Thanks
Great video
Yes please 🥺 we want hands on videos as well
Good overview. My comment is: Sharding and partitioning are not the same though they both are breaking up a large data set into smaller subsets. Sharding implies the data is spread across multiple databases while partitioning is about grouping subsets of data within a single database instance.
Thank you very much
What an awesome intro
Good job
Good explanation
its Good explanation
Thank you Yogita for making this channel. I went through your primer course videos, and they explain all the concepts in a very detailed manner. However, I am not getting confidence how to give system design interview i.e how to connect all the concepts combined together to form a system. Could you upload a video in an interview manner like starting from a problem, discussion that happen between interviewer and interwee to proceed with the design?
Hi Yogita, Your explanations are awesome but I think you can deep dive once basics are done and also if you plan to have real implementation it would be of great help as most of the developers don't have access to work at such a scale.
Keep up the Great Work !!!!
Noted. I will share.
Hello Ashish, thank you for a point such a great point that will compliment her introduction. Great work Yogita.
Great video thank you !!! What are the usecases for vertical partitions?
Videos are very good and clear.If you do pratical implementations of some topics then it would be crystal clear.
Very nyc yogita concept is clear
Thanks :)
I'm happy this is free
Great video as always.
Hands on like sharding MYSQL would be great.
Any plans for that?
Hi Yogita, does adding more shards later on in the development phase also increase a challenge, what I mean is if we add another shard do I again have to redristribute the data from old shard ?
Nice explanation
Thanks
You are welcome
very nice videos, please keep it going ..........
Thank you, I will
awesome
Great Explanation. Can you please make video on Time series DB ?
Great explanation. Which mic are you using? You voice is very clear
Thank you :)
Awesome content on Sharding, Can you please do a hands on practical video on how to do replication and sharding
Will try to.
Hi Yogita, what is the use of logical sharding ? Thanks a lot for your videos ! :)
I have a doubt, I see that in vertical partitoning example from 3:49 you have put each column in a different database server. Can we do that for vertical partitining? I read that only horizontal partitioning is spread across database servers whereas vertical partitining is done withing a server. Can we do it for vertical too?
Before I say anything, I will admit I have no experience with sharding, but I do understand what you presented. Scenario. With 1million users, would you say that user info maybe be small but related user information, like images or videos can take up a lot of space. So it's not so much the amount of users as opposed to the space user supported data can take up.
Thanks for the video, this is really awesome. Though after watching and learning abt this I am more confused with the other various terms. Please make a video to clarify those. Database Partition Vs Table Partition Vs Distributed Database Vs Replication.
What I understand is Replication is a Read-Copy with a distributed system. Distributed Database is having multiple database at different servers however in Sharding we are doing the same, if so what is the difference. And how the Database Partition is different than Table Partition, as in table partition we do not have to worry abt accessing it or no downtime is required however for database partition we would need some downtime when adding new partition.
Please make practical sharding video
18000 views 800 like... ☹️ you need to remind people to like the video in the middle of it... This was very helpful
Helpful video .We would like to have hands-on on how to shard relational DB and NOSQL DB.
Reply
Is sharding and partitioning the data same? As per my understanding, partitioned the data based on column which will create separate file for each partition value, so that it will improve the read performance when we query the data based on partition column
Great explanation. One video if possible on Zookeeper please with use case.
Will add to the list.
nice
So the strategy to scale in most use cases: Vertically scale -> Separate Read and Write -> Archive and keep it under control -> Try breaking the application into smaller services -> Shard the last resort
add cache, replicas* in between
can we do replication as well as sharding for the same DB architecture, if so how. Could you please explain
Yogita!!! your presentation is just amazing and out of the box.
IMO you are the best teacher
please please make videos on DSA too. I am 100% sure you will get overwhelming response
Thanks a lot Akshay. 🤗
Are their any other use case of logical partition other than query optimization?
You teach a lot better than Gaurav Sen. Why don't you you launch your course on System Design. It would be a great help for all of us.
Should I design and code my own database engine?
i would like to see video comparing time series dbs for low latency large data sets.
I had one question to ask : let's say that I have sharded the date in to mulitiple machines for desinging facebook, then in this case how the "FEED / TIMELINE" of a particular user be generated? One user can have multiple friends and he can follow many pages as well. So do we need to go to each of the shard where the friend of the user is residing to retureive the post information ?? It will make the system bit slow. How should we overcome this issue??
Hi, when a particular user posts something then there is something called fanout services which push the same data into the timeline of the friends' user which is also maintained using some caching strategy and it happens in the background so that when friend user looks at their timeline can see the post and also one more thing to keep in mind is the data is eventually consistent in most of the social networking platform so it may take some time to update the timeline.
Can you make a tutorial on Spring Boot internal working?
Please make video on sharding in mongodb.
Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?
Can't we create index instead of sharding like search query will be faster rather than doing sharding?
demos for both relational and non-relation would be really helpful...
noted!
Did you mention problem with transactions after sharading?
Hi Yogita, Thanks for such a vivid explanation. I just have one doubt. You gave an example of Tinder while talking about Sharding sometime around 10:22. We can shard on the basis of cities definitely. I was just wondering what would happen to a particular person's data (in db) when he moves from city X to city Y ? Will that data be lost ?
Insert the record in new shard and soft delete from old shard
Thought: Would it be better to mindfully split the tables in different databases instead?
You should add link to the next and previous video in your description section
good idea!
Hands-on videos would be of great help
will do try for sure.
Thanks ! City based partion will come under column ( Vertical partion ) , right ?
no, it comes under horizontal data partitioning only, with city name as the shard key. all rows related to same city will be stored in same shard
reading data from a table having a index vs sharding , which gives more performance? index helps to look up isnt?
Problems change with scale. Indexing might not help when datasets are huge.
The best way anyone could have explained it. Been working in IT for the past 2 years I just feel lost in the CS concepts and never found any structured content. Since I did my bachelors in Electronics. Thank you.
DB Shards and DB Partitions are two different terms if we take distributed database into the picture. In fact, sharding and partitioning have different meanings in distributed system.
Could you please link the source to your claim here?
@@sudocode FYI -> czcams.com/video/wXvljefXyEo/video.html&ab_channel=AsliEngineeringbyArpitBhayani
Please make videos on Data structures
Will try.
What is the point of having logical shards if we can have multiple physical shards?
Can you please cover how bloom filter can help here?
Will cover in a separate video!
Good one. How sharing of db relate to cap theorem ?
It's a long answer!
@@sudocode may be I have used wrong world...it is sharding and not sharing..
Query across the shards is not a disadvantage I mean situation itself will not arise as in shards( horizontal partitioning ) we are going to put whole schema in any particular shard not just a specific table..
Horizontal sharding implies splitting a table into multiple shards. In some cases all those shards have to be queried.
Hi please come up with hands on using mongodb will help a lot 🙏
Will add it to the list.
Hello, I really liked your video and your explanation is very clear but I don't agree with that partitioning and sharing is the same. For my knowledge partitioning is when we are splitting a table in database and sharing is when we are splitting data to different databases. So partitioning is done in the same database, we will just split/ partitioning the tables and sharing is done having multiple database, we will sharing data between database.
I am sorry to say but that’s not accurate. Please check your source from where you read this. Check description for sharding links that I shared.
Can you please make hands-on videos please. That will help in understanding things practically.