Back-Of-The-Envelope Estimation / Capacity Planning
Vložit
- čas přidán 12. 09. 2022
- Weekly system design newsletter: bit.ly/3tfAlYD
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
Other things we made:
Digital version of System Design Interview books: bit.ly/3mlDSk9
Twitter: bit.ly/3HqEz5G
LinkedIn: bit.ly/39h22JK
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.
![Eminem - Houdini [Official Music Video]](http://i.ytimg.com/vi/22tVWwmTie8/mqdefault.jpg)
One of the first things I learned in physics was Fermi estimation. It’s an incredibly useful way to think about problems, extrapolation, and what levers you have to influence an outcome.
Hi, can you share details about Fermi estimation, what is special about it?
O😮 and i started with newtons laws.. What was i thinking
Another really useful tool that ties in is Dimensional Analysis (keeping track of units when performing your conversions)
I believe that this is the first time someone explained this on system design videos. Usually they just mention the numbers of read times or size in kb... Well done!
These videos are making me a more well rounded programmer, amazing work!
'Thank you' for taking effort to put these up! I have had opportunities to do these in the past but the way it is presented, says a lot about how one should think about the steps! That is a fantastic thing: clarity of thoughts.
You make everything very easy to understand. Bravo 👏🏻👏🏻👏🏻👏🏻👏🏻
The videos just keep getting better and better 👏👏
This is exactly what I've been looking for. Thanks!
I am so glad I found your channel. Keep up the high quality content
Thank you, if I'm going to do data analysis for a web application, this video will be very useful to me.
Thank you for considering the request about including animation in the videos. It's elegant and easy to understand. Great work as usual
Awesome knowledge, thanks for your amazing content, first time I see someone sharing about this on CZcams with clear examples.
This is highly useful. Thanks for uploading this!
Well, thanks for the videos! Your newsletter was the first I intentionally subscribed to. Also got the green book 👍
_intentionally_ subscribed
haha nice one
Amazing thank you so much for sharing
Omg your channel is worth resource for me ever i'm Full stack developer who interested in System design. and I would like to say it great to have this channel in CZcams :)
Great video!
This is awesome! Please also talk about estimating the number of machines needed.
Amazing video, thanks for share the knowledge.
Thanks for the video. It was super easy to follow. I bought both the system design interviews book now :D
These are very useful videos.
I bought the second edition and have been reading it. It's fantastic by the way, but I wanted maybe a section just like this video that was explicit about the estimation process. However I will say that picking it up through practical examples really drives it home! Also, I was literally just thinking about this and this video popped up. Get out of my head!
Do you by any chance mean the second volume? There is a "Back-of-the-envelope Estimation" chapter in the first volume. Or even better, the chapter is available for free on his website.
@@andyl.5998 Yes, my apologies, I have Volume 2.
Awesome video. I was hoping you would cover machine count in terms of processing speed
Thank you a ton!
Awesome video!
Can you guys do a follow up video that summarizes good benchmarks for when you should shard or start scaling beyond single node?
E.g., 1 request per second => single box + backup is sufficient. 1E12 requests per second, => need distributed fleet. Can you help provide insight on the values in between?
nice video, keep it up 👍
thanks! really useful
fantastic
Awesome
Thank you for the videos, the animations are entertaining and i'm learning something for design interviews.
Great
I can't imagine how complex the calculation when the users request is not categorized or limited by system.
Love your animation and visual, can you share how to do it?
Thanks for the awesome content. Also, how many queries per second does generally a single modern web server handle if we also want to estimate the number of web servers required. A video explaining limiting factors and general number of QPS handled by different web servers of different capacities would be a great addition to this and also provide a practical insight.
I think you missed the point of this video
Hi, ByteByteGo. How did you make this video? What tools did you use for video edit and animation? Thank you very much.
nice video. could you incorporate the estimation for resources, like what do those number means. how many manchine cores needed for QPS or storage. and why
The first problem can be simplified by ignoring the x0.5 and x2.0 as they cancel each other out, at which point the problem is basically just 150e6/1e5 ~ 150e1 = 1500
Hi, thanks so much for the video
i don't quite understand this part on minute 03:00. (disclaimer, I'm not quite good at math)
300 million MAU;
50% use daily;
you said, DAU = 300 million * 50% = 150 million
how does MAU gets converted into DAU here ?
is that how the calculation supposed to work ?
shouldn't 150 million supposed to be "MAU that uses daily" ?
so, if we want to look for DAU, shouldn't we divide it further by 30 ? (assuming 1 month = 30 days)
that gives us, 150 mil / 30 = 5 million DAU ?
Sure I agree that this is useful in most cases, but at the same time, it can be very misleading if we don't understand what our accuracy is.
Once you start multiplying a couple of these estimates, that have large error bars, the number you get in the end shouldn't be strongly relied on. I find this realization even more important when you start to use these estimates to make a business plan, especially if it influences profit margins. You end up getting a false sense of security imo, and it's what I like to call "playing with numbers".
What I prefer to do is give min/max estimate for each metric, and then calculate a min, average and max final estimates for the total system. It's a bit more work, but it can help get a better understand where the risks are and how fragile our design is.
I'd still like to know a better way of doing that (fuzzy numbers?) in Google Sheets/Excel as having 3x the mumbers makes the whole thing harder to understand at a glance.
I was wondering the same thing about error margins. With such large numbers in multiplication, it can have huge error margins. It is good to see someone with more knowledge validating the thought.
I also like the idea of mix/avg/max instead of having a single number for a metric.
does anyone know with which animation tool this video is created?
No doubt, videos are simple and awesome. Still I do have one doubt:- When we calculate storage, why do we need to multiply time duration with storage capacity? Means if we need 1 GB of storage for 5 years, then its 1 GB till the end of 5 years, right, why we need to multiply 1 GB * 400 days something?
For media attachments, most companies don't keep them forever. There is usually a retention policy on these types of data. In our example, we assumed that the retention policy was 5 years. To calculate the storage requirement for holding the data for 5 years, we multiply the yearly storage requirement by 5.
Hope that helps...
So the estimates are a per day estimation(as we began with no of tweets per day) and if we retain it for certain duration like 5 years, then we need to multiply the estimates per day with total number of days.
How come this channel has so less videos but has so many subscribers
We published two best-sellers on system design interview.
We have a 100K+ newsletter on system design.
We also have large social followings.
We try our best to make quality videos with high signal-to-noise ratio.
All these factors help.
If we have 300m MAU and 50% use it daily, the DAU should be higher than 150M because it will also include users who visit service once a week and once a month for example, so 50% that are DAU + some portion of other half
If you understand DAU correctly, DAU already consists of those visit service once a week or once a month. It is a general average daily user, yet not every user accesses the service daily. You do not need to add more users on top of DAU. The calculation is meant for general estimation, not precise estimation.
Does Anyone know how to make video animation like this channel?
How do you estimate if you are designing a new system and have no idea how much requests are you going to get?
In a real workplace, system design tends to look very different.
I work in tech and rarely do I see numbers like this. It's generally adapt/upgrade as you go and on existing services, the expectations are already known.
TL;DR: System Design interviews are just as realistic of the actual job as Leetcode questions are in the workplace.
You have to learn to play the game. It's pretty pathetic tbh but that's how the job market works.
@@2RosarioVampire
1.So you mean to say that estimations like these in real projects are made on certain assumptions and then scaled on the go?
2. Do these or such estimations have anything to do when choosing DB?
Say if you get 10K query requests and 100K query requests on a DB, is the database going to change for that 90K difference.
DB (Database here I mean)
@@ziakhan-tk7rk
1. yes.
2. Most mature firms already have databases everyone else is using. Most projects aren't special and those would be obvious anyways. For instance, Google uses Spanner for db in many of its projects internally.
@@ziakhan-tk7rk Just assume System Design interviews would be horrible in real life practice.
As real as algorithm questions revealing how good a worker is as a developer. Most apps are similar enough.
For coding, it's mostly CRUD or ETL.
For designs, it's mostly based on designs from other projects. And mature firms have a good infra team so these shouldn't be an issue.
For instance, say you need to update X when Y happens. Unless you particularly need X in sync, you will just need X async. So you would just use a distributed message queue everyone else uses from the fired event.
And that distributed message queue will work at scale (you can have that assumption reasonably). And it's the infra team's job to do it properly.
Of course you also look at the metrics to then determine whether to scale more or not. But really, I don't see numbers in many actual designs. At least not the type of System Design Interviews.
Questions like these and Leetcode are fine for startups and they make sense. Startups operate at faster speeds and tend to have less money to spend on large teams, so they have a few people with a wider breadth of knowledge.
First time hear this name, for me it was always volumetrics...
boomerang
π = 3
Damn... I feel like a total newbie.
Amazing video as always! Just one thing: at 5:49 you say that one Kilobyte is 1024B, but that is wrong. 1 KB is 1000B. 1 KiB (Kibibyte) is 1024. This was adopted in the 90s to avoid "kilo" meaning 2 different things. Kilo means 1000 in Greek and that's why it is used in the SI. Also, 1 Billion is 10¹², not 10⁹. That is one thousand million, or a milliard (Americans use the short scale, the rest of the world doesn't 😉). Other than that, great job!
Please forgive my ignorance, but I don't see the point of all the calculations in the head. Instead of giving a quick number, I'd rather write down all the factors and calculation steps, so that not just me, but others can also review later.
Be carefull about "Billion=10^9" and "Trillion = 10^12" overseas thouse words change meaning: "billion = 10^12" & "trilllion = 10^18"
I feel like you focus so much in the animation but does not go into the details of anything