Why Google and Meta Put Billion Lines of Code In 1 Repository?
Vložit
- čas přidán 16. 06. 2024
- Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bytebytego.ck.page/subscribe
Animation tools: Adobe Illustrator and After Effects.
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.
Google had to make changes to their mono repo over time. "Accidental" or "unpermitted" code re-use led to too many dependencies. Teams would find that they would check in a simple change, and cause test failures in some other project that they didn't even know about. To solve this, Google introduce visibility controls. Before creating a dependency on a new package, you have to be added to it's visibility file.
I have seen monorepo's go wrong. Repo's where many unrelated project were put together. It was a big ball of mud and how easy it was to add more coupling didn't exactly help.
I have also seen microrepo's go wrong where it's basically one big project due to coupling but spread out over multiple repo's.
Main important thing is to put things together that are related and separate things that are not. Usually those will also change together. Apply vertical slices but at a higher level. Also don't just reuse everything just because you can, there's always a cost.
Things that change together should live together :)
@@oscarljimenez5717 exactly
Monorepo does not have to be mono. If you have three different products, with three different stacks, it does not make sense to have them in the same repo, unless they do share the same pipeline, they have the same dependencies and they do use the same stack. If you have a product with 20, 50, 100 services, then having 100 repos is a nonsense. Different products can have different repos. They will be fully managed by different people, they will use a different stack. If there is no common ground, then use different repos, but do not separate services of the same product in multiple repos, that is stupid.
If I have 30 different repos, my CI/CD team essentially needs to support 30 different pipelines. This is not scalable.
Monorepos don't seek to group projects by relatedness, but to reduce platform level toil. Coupling unrelated projects and bad hirarchical design are not problems with a monorepo approach, they're just bad engineering.
I agree so much with this. What I do nowadays in my team is to start a new feature as a monorepo, and group all apps and libraries together. If I need functionality that is common between these services, that goes somewhere else.
We found that this approach keeps related projects together and reduces repository surface area of the team. New team member started? Great, here are the 4 monorepos that map exactly to the features we have. Each one may have its intricacies, but at least it’s clear where to look in when things break
I found your channel a few days ago and can't stop watching. Thanks!
These videos are great. The narration is clear and easy to follow along with, and the animations are very helpful.
sure, i also love this channel
For microrepos, the most powerful dependency management approach is SemVer. It trivializes bumps and rollbacks, at the cost of tracking compatibility (which requires tools or discipline).
No
This is very well put, and adding to the perspective is how the release management is structured around a project. And what timelines are projects chasing. At times I have seen teams working with strict timelines are struggling to put proper documentation and face trouble with monorepos.
importing things from random places in monorepo is not an advantage. And on the microrepos side, you don't need to communicate the changes between the repositories, you define your dependency and target specific version of another project, which is almost impossible to do with mono-repos. Excess communication, hard dependency management, release trains are all possible disadvantages of the monorepo.
That said, you shouldn't be choosing the strategy based on what google or uber does. Pick that suits the best for your team. And remember that dependency management is a very important aspect of a long-living application no matter which strategy you use.
Using monorepo tooling like bazel it's trivial to declare visibility for your packages. This ensures you don't need to worry about random imports, as each package can declare the scope of responsibility, and service specific code is not reachable elsewhere.
At scale, versioning of dependencies is a really bad idea. Versions get updated for a reason, and although backwards compatibility is important, if we're fixing a security vulnerability, we need it live on all repos immediately; this is incredibly hard to do across repos but near trivial in a monorepo.
@@muizzy i disagree. Why versioning is a bad idea? All dependencies are risks for your application. Not in a bad way, but in a way that you need to be aware of them and evaluate your pros and cons of brining a new dependency to your project. Updating all of your dependencies willy-nilly is a risk which can be mitigated by substential automatic and manual test coverage, but sometimes it is not enough. For certain projects you will need code and security evaluation which you cannot afford when you randomly update. Of course not all projects need that, but not being aware about your dependecies is a ticking time bomb
I can't imagine putting a shared library used by 10 different teams in a mono repo. The process of introducing change in such an environment would be super slow. If that code resides in an independent repository, publishing and maintaining different library versions simultaneously would be much easier. Downstream projects can consume different versions of the shared library and move to the newest version at their own pace.
That assumes the different applications are building the shared library every time. Technically, you could still use a single repo and build with the correct packaged version of the shared library. Everything is a cost/benefit decision.
Almost 99% of cases, if a mono-repo is managed by multiple teams, it'll be a mess.
Could you please provide any concrete scenario? as these code submission process go through several code reviews.
And by definition, it will, since ALL of the code of ALL services will lie into it. So...still bad throughout large orgs. I think a monorepo is fit in a startup entity where there's a couple of engineers and that's it.
Me who worked in Meta, I can say there is no problem with a single repo.
All this pros are good only on paper
Monorepos are definitely way better. Try to work at a place like Amazon with versionsets is a hell-scape of chasing teams down for months for version updates. Monorepos have their own problems, but definitely way more efficient.
Thanks. I just joined Google as a L5 engineer and was overwhelmed with this 80 terabyte repo.
Never seen something so big. Your video clearly highlights the pros of having a monorepo and also discusses the trade-offs my company made.
I want to let the readers know that at Google, we don't have a local development environment. We use Cloudtop, a remote server, and a browser embedded IDE named Cider to code as it is next to impossible to clone 80 TB repo locally.
Our macbooks are just a window to look into our Cloudtop machines.
Very good content! And What tool do you use for video rappresentanti on? 😊 thanks for sharing this!
Always love seeing the animations for these videos, would love to see a video on the BTS of making and using them
Ditto. What tools do you use to create the animations?
I just started a new job managing a new team which is doing micro repos and the dependency management is killing us. It’s a nightmare, I was not used to it before, but the downside of having micro repos, especially if you have tons of homegrown libs, seem to massively surpass the upsides.
Congrats on your new job!
Multiple repos suck.
I had seen a few OSS services reverting to monorepo in GitHub. They may have suffered from the same reason.
The dependency management doesn't go away with a mono repo, it is just more complex and requires meticulous tagging. With micro repos there is technically no need for tagging since the dependencies are managed with release artifacts, not source code.
You haven't mentioned how many repos are you handling. Unless it is a large number - like greater than 20, any competent DevOps team should be able handle it. It could also be a competence issue, which the team is covering up as complexity issue ! We had nearly 35 repos, and only 3 dedicated DevOps personnel. With proper governance and the right team, we were able to roll-out to prod every two weeks
You are missing the whole point of the title and the real facts. Google has a main mono repo due to legacy constraints (development started when CVS was round and was mono repo in design and the best tool at the time) and they built tools, before git etc., to overcome those limitations. Their Chrome and Android code bases are separate repos... mmmm I wonder why. Ultimately choose the best option for the development and development lifecycle of the product and not "because Google etc. does it this way".
Both have their benefits, as you said, and both can be used in the same company.
We design our repository management strategies around their release CI/CD pipelines. If a software project needs its own pipeline, it has its own repo. Our front end only needs a single release pipeline so we have that as a monorepo with a bunch of plugins etc. within it. Our backend has multiple CI/CD pipelines whether we're using the code for internal tools and the website or our business-to-business platforms and databases, so those all have their own pipelines where appropriate and thus their own repos.
I would try to stick to a monrepo for as long as possible (perhaps until unicorn status), but for big enterprises I think the benefits of microrepos eventually outweigh the costs.
Thanks for this ... btw if you created a video on how to use the Adobe tools to create those diagrams ... it would be a big hit
Hey I like the presentation you do on all of your tutorials and i know you use Adobe Illustrator and After Effects but can please make one video how you make this animations?
Great explanation.
Just one thing, at 6:40, under Microrepo Dependency, shouldn't the conclusion be "Service do not share dependencies" (because you have given explanation of A=1.0 for scv A, A=1.1 for svc B and A=1.2 for svc C).
Great video! A topic suggestion for another video: how to the big techs deploy the services from those monorepos? Since there is no package versioning. I'm assuming they version the builds/deployments. But how do they ensure compatibility between them? How can a breaking change be safely deployed? How to avoid deploying services that haven't changed since last CI/CD build, etc. Thank you!
ByteByteGo - please please share the tools and softwares you use to create these wonderful videos. It will be extremely helpful to learn them and use it for work and share knowledge in general. Thanks in advance!
A great summary. As always ...
Google didn't really create Blaze/Bazel to deal with a monorepo. They were created mainly for (a) Hermetic Builds, and (b) To manage a compute farm of 1000+ worker nodes. Google needed a massively parallel system to compile and run unit tests for every check-in. Some projects like the web page fetching system might need to compile & run 230,000 files (2018 numbers) to verify that an interface change didn't break anything. Thus, a compute farm of 1000+ cores needed to be harnessed. This was not because of the monorepo but more because of the huge number of unit tests against other systems. On the other hand, Google had to REWRITE Perforce (p4) to scale 1000x bigger, so they rewrote it as g4. G4 is literally 1000x faster than perforce, and can check out a codebase with 400M files in 1 second, using file system tricks and on-demand file fetching.
coordinating in both cases requires a lot of communication, especially in monorepos when there are conflicting changes.
Hi, Sahn! Great video, thanks for sharing! I wonder, maybe you know, what is a right way to handle a rollback after deploy of monorepo? Nowadays we have a great tooling, like Nx Workspace, which can build/test/deploy only affected apps. But what happens if you changed 5 apps and 2 libs and you release your monorepo changes to prod. And then on prod you see that 1 app works badly with some issues. Is there a strategy how you can rollback just 1 app? Or what is the right way to handle such scenario when developing in monorepo? Thank you in advance
very good point!
great content as always!
In my experience, the greatest challenge for the microrepos comes when the teams did in fact create a dedicated library, for shared utility or business logic. When multiple teams start updating code in the shared library, that all the other micro repos are using, it becomes a huge mess 😅
For that case a simple versioning of the library would solve
Good practices for that case:
- version the library (semver)
- change log with version changes
- a small team of experienced dev mergin the Pull Request
- clear guidelines of how work on the library (code styling, test, etc..)
Shared repos also start getting confusing when you have lots of micro services. For example, if you had 10 microservices, and there's a small library that's used by only 2 of the microservices. Does this small library get put into the "shared library"? Or does it get put into it's own smaller separate library?
What happens if future microservices do start using this? Do you merge it into the large shared library?
Or what happens when you have lots of small libraries that are shared between multiple different repos, but is not used by a majority fo the repos?
Ever project is different, so there's no one size fits all answer. But I've seen cases like this where the code for each of the 10 small repos is very small, but the shared library is large. At this point, the project has basically turned back into a mono repo again.
@@vhmolinar that in itself solves the issue of "static stability", but does not resolve the overall mess which happens when different teams have their own needs and start to drag the "reusable" lib in the direction each of them needs, and those directions are different and conflicting more often than not. That's hard to tackle sometimes.
@@kostiantynkolesnichenko1504 yeah, I agree. That is why DRY must not be a default. Even more, that is why I would not pick a mono repo, because those situations tends to happen more easily
I have a quick question.
Since I have only worked with micro repos till date, I always imagined testing a monorepo would be super hard right?
Amazing video as always ❤🎉
on the contrary ,it pretty easy with tools like Buck or bazel
@@alexanderrichburg7934 Ahh I see.
I missed buck and bazel being in the picture. Thank you.
Can you make a video explain how monorepo manage their code？
Can’t imagine the git graph of a large team using monorepo
It's much easier than you might think. In Meta we always rebase our code changes, not merge. So the graph is just v1->v2->v3->...->vx. As the video says, to finally achieve this goal, a huge amount of work needs to be done by the team maintaining the repo. For us normal users it's straightforward rebasing.
There is no one size fit all solution. Which repo to use depends purely on the specific project. For large scale enterprise projects, which runs into multiple years and in this era of outsourcing, handed down t multiple vendors, Microrepos are the preferred choice. Microrepos also lends itself well to Microservices architecture and each team choosing their own Programming Language. However, if the project is a small one - let us say a Mobile App catering to specific need - I would prefer monorepo. This is because, Microrepo is an overkill.
There must be tons of instructions and restrictions to use mono repositories. How to deploy, build and manage, and who is responsible for each part of the domain? It's impossible to keep in one head all 'domains'.
But if the project is new, this may help simplify the process and speed up development.
Is there a bazel like tool for microrepos? A tool that provides build and deploy methods for multiple projects?
How you created this type of video animation, I want to use it for my college project?
aren't blaze/bazel and buck build tools as opposed to version control repos ? surely these are two distinct things.
What if your organization manages some private and some public projects? Especially packages on platforms like NPM, the convention is to use a dedicated repo for that.
Often times when you're maintaining a public project, your internal version of that project is more specialized. It's not uncommon to have an internal version, with periodic updates happening to the external version that copy some features.
Public versions of projects are always in a dedicated repo, though.
@@muizzy thanks! Makes sense
curious what SW to create the animation of the diagrams?
thank you uncle for sharing your knowledge
I disagree with the assertion that a monorepo helps promote code reuse.
Having seen first hand code bloat from copy pasted code in a massive monorepo.
Monorepo , my project depends on mostly reusability of the code from other modules.
Azure uses micro repo too :)
i'm so dumb man but i try to learn something every day and ur vides are so cool
I agree that micro repo is very challenging to manage the feature changes that across multiple systems. That does require a lot of coordination, planning and management. I'm not super convinced that the coding style is an issue, or at least not important one. People who does not know generic coding guideline will commit bad style code regardless how the repo is being managed. Another big con for micro repo is on change management. Changes will need to be done in a way that's backwards compatible. Since your dependency team will not be making the update at the same time when your team doing the deployment. Coding against "time" added another dimension that makes even easy change hard. The logic to retain backwards compatible will need to be removed and it is often omitted since you don't know when all your clients will be migrated to the new code path. This processing making the repository harder and harder to understand and maintain.
If you are using REST, you can introduce different URLs based on the versions to ensure backward compatibility
I have accumulated experience with both strategies, for different projects, for 2 of the FAANG, and for others.
If I consider all pros and cons, I remain firmly a believer of product mono repo, but company micro repo.
Mono doesn't sound like a good idea for any company... It can lead a huge amounts of merge conflicts, poor build times and bloating ci pipelines
I feel that even though dependency management with micro repos is troublesome, monorepos have tight coupling and thus, require a lot more collaboration and reduce the development speed, as everything affects everybody.
Let's say we're changing the functionality of a utility and restricting it's scope for performance purposes, all it's usages would have to be updated, which might not be supported. Thus, there would be multiple versions of code created and used in the same repo, rather than in different versions of repos. The point being that monorepos require more guidelines which isn't always the case and it becomes the wild west.
That's a terrible example, mate. In your example, you would have different repos depending on different versions of the utility in a micro repos setup. That's a delay or bury the inconsistent issues for future, and it's harder to consolidate across different repos, how is that better?
I feel like monorepos are a strange normalization of bad practices around dependency management - and if we squint a little, it's really about project management. If there's a lot of shared code across projects (typically a good thing), there's nothing wrong with carving out chunks of that code into logically cohesive libraries with thoughtful interfaces and their own versioned repos. The applications and services depend on those interfaces, and if they're decently architected and documented the average dev won't need to look at the code to understand how to consume them.
Monorepos seem like a great way to consolidate tech debt into a giant congealed mass of doom.
Discipline and order is probably better encouraged with microrepos, agreed. But monorepos are about removing the enormous overhead of managing 50 "independent logically cohesive libraries with thoughtful interfaces", thus saving many man-hours spent on "bureaucracy" figuratively speaking. It comes down to the priorities of those folks working on it.
I think a mixed approach is best.
Make one repo per Customer for Example. Or one repo per technology stack. One repo just for libraries / frameworks.
Imagine google with their thousands of failed / trashed projects all living in that mono repo?
how to create these videos, tools?
we chose the middleground, multiple repos, one per team, lets see how it holds up :)
Durability, reliability, and security become super important in a giant monorepo like Google's or Meta's. Wonder how they manage that? I mean it's basically a single point of failure. Maybe they have many backups stored in a super reliable storage with lots of security layers?
All big tech companies will have highly durable storage (not just for the code, but for everything). In terms of security it tends to be a combination of best practices: Hardware certificates, single signon, MFA, expiring tokens, etc.
One Repository, does not mean one code base, or one application, or one CI/CD pipeline. Just saying.
Why would you put everything together if they don't share code, pipelines or infrastructure though? Just making your tools slower for no reason.
@@truevelvett You wouldn't. I didn't say they don't share code, pipelines, or infrastructure, I said there could be multiple apps and pipelines in a single repo.
I heard Google stopped doing that years ago.
Surely you meant polyrepo not microrepo?
I appreciate the presentation. However I don't agree that mono repos are better for code quality or dependency management. No matter what kind of repo strategy, management of code quality and dependencies are necessary. Mono repos just make it humongously complex unless there is a dedicated team that builds all the tools that you get for free with micro repos.
I dare say mono repos are an unresolved (and by now unresolvable) technical debt owed to a drive for "simplicity" at the start of those companies when having multiple repos seemed too much of an effort. That made sense back in the early ~2000 years when setting up a new repo essentially meant to buy a new server and spend a couple of weeks configuring it. Luckily those days are gone, setting up a new repo is literally two or three clicks & commands, and above all it is virtually free.😅
My advise is to use micro repos and not lose another thought to this question. Save your energy for more important decisions.
Micromono repos are the new hotness, only the truly anointed can grok them.
What about Norepos?
Google drive gang
Call me old school, but I’m struggling to see the advantages of a monorepo. Less tooling and flexibility is a good thing, and microrepos scale better, IMO. For most projects, shared code, like utility modules, can simply be referenced via npm, or whatever package manager you’re using. Regardless of approach, good documentation, management, and communication cannot be optional. I feel like we devs are drowning in configuration and tooling these days, and it’s wise to avoid any complexity that you can.
Amazing explanations!
As always neither is the definitive answer and you pick what works the best for the given situation...
i'm not a fan of mono, especially with large teams.
the PR's...
microrepo
It's just an opinion. Do what works best for you :)
It's maybe because they don't want anyone to read that code.
For DevOps, microrepos make more sense.
When was the last time Google came out with a brand-new, amazing product? Google has billions of dollars to burn - which, I am sure, is what they are doing to maintain that monorepo. I would not base my decision to adopt a monorepo solely on the fact that "Google did it." I believe you might be greatly disappointed in the future.
People have a hard time understanding that the tooling that a company like google uses is created out of necessity for the insane scale at which they operate. Although some translate well to small scale (golang), most just add complexity where it's often unnecessary (bazel)
please improve your audio quality...video is better but not clear and loudly audible!!! don't give me chance to say this next time
😂
I'm guessing you caught Bazel → "blaze"
Thanks for sharing your great videos!
Blaze is the name of the internal tool. A version of it is open sourced as Bazel
Got confused by this as well