I come from the school of Joel Spolsky and his heydey as a writer and blogger (before he retired from writing and went on to found SO) so I am not one bit surprised at the efficiency and practicality to solve the problem at hand. I have seen multiple engineering teams and piles of tech thrown at problems that would be easily solved by a tried-and-true setup like this (albeit with a fraction of the DB server memory... in one case arguably a properly configured lone Raspberry Pi could have handled the traffic...). I love your videos. Thanks for sharing something that reminds us all to consider the alternative and see the forest for the trees :)
1.5 tb of Ram for the DB instance is the game changer 😎 why to create an external cache when the DB engine is capable of but vertical scaling has its own limit .
I entirely agree with this, but one could argue that microservices are simpler than monoliths. The real issue is the folks at Stack Overflow are trying to keep the lights on and a rewrite for microservices would be a massive undertaking. It's more manageable for them to continue optimizing their monolith. Both are viable solutions. But you don't just throw the newest trend on a decade+ old tech stack. Just like you wouldn't try converting a stable microservices based solution into a monolithic one.
@@jacobmills3076 Thats not the only reason. Microservices does not automatically make things faster or more scalable, a well designed monolith can be both faster and more scalable depending on your use case. So before going for microservices, you should know why MS is the preferred solution for you, is it team management. If you have hundreds of developers, merging and testing all different changes can prove to be a source of bugs and delays and require massive amount of extra testing in a monolith. If you have different parts that have vastly different scaling requirements that could be a good reason to separate them into separate services where some might run 1-3 instances while others run a hundred spread out over multiple datacenters and physical computers. SO is running with a small stable team and their application scales uniformly, so they can just run more instances of the same monolith. And its mostly reading, not updating or complex relations with changing type of queries. That makes a monolith easier to administer.
@davidsmartensson yes, they are only a view people who already build a working monolith. Also Stack Overflow is a rather simple service where view provide text for many people. There is not much to you could abstact in a micro service.
This is only "unconventional" because most software developers are really bad software designers. Stack Overflow is showing a mastery of software design that most of the rest of the industry should learn from. Don't sold problems by adding more layers of complexity. Solve problems by finding the simplest solution that works. Such a rare skill in this industry.
I push back against distributed microservice architectures whenever I can. It's a great tool, but it's not a silver bullet and has its tradeoffs. I've seen too many teams go for microservice or distributed monolith systems when a better first solution is just.. monolithic. Monoliths aren't inherently bad.
I remember pushing back during an interview about going microservices and big data when the problem set was in a few gigs and relatively small data. I was looked like as if I was a moron. If you can't build a monolith, you're going to build a distributed ball of mud. It's gonna happen.
i strongly agree with you. Idk why we like trends so much and then it sucks because people want others to know the technology when in all honestly probably not even the best solution and its gain so much tech debt they dont care😭
like i turn my head and everyone wants to use the next big think. That why trend will never help you learning just fundamentals and innovation and deep understanding of problem solving is a goldmind
absolutely love this episode. a real world example (even at scale) showcasing the importance of thinking outside the box and being practical. thx for making and sharing this video!
I can understand "no cloud" because Cloud is very expensive when used for a big always-on service. On-prem is still relevant today because you can maintain absolute control, you just have to make sure your Infrastructure guy has built you a resilient environment to live in. Modern business keep reading the internet buzz and making technical decisions they know nothing about, or as I like to put it "an Exec and his budget are soon parted". I've been in this business for nearly 40 years and I've seen a few great innovations, but most change has simply been re-inventing the wheel that was working perfectly well before (ah, the cycles I have seen repeat). Clouds et al works very well for some situations, but it isn't the panacea for all problems.
I've been in the industry for 20 years (another 20 to go i guess? 🙂) and i've seen everything you've just described. Personally, I hate cloud as much as i like it. I can see the appeal, and how it can be really useful, but it's just made my life as a network engineer 5 times more complicated than it ever needed to be. Everything my employers are doing in cloud they could do just as easily on-prem for a fraction of the cost. They just move it in cloud because they think it's cool and looks good on their resume. I'm just waiting for the next fad when they'll want it all back on-prem again.
Lol I love this. I even have a feeling the stack overflow codebase is being maintained by 2 people once every month. I have never met/heard of an engineer working at Stack Overflow 😂
I’m working for a small company where I’m head-everything for a software project. I only got one parttime coworker. The software should be handling several honderds of users per month. The initial idea is to use micro services but monoliths with a few “external dependencies” feel better to design/create/maintain/extend. Might sound weird, but we’re crossing away reasons to create a micro service architecture.
Thanks for sharing. I always thought the big tech's stacks are over engineered. I always believed in stacks like SO's. But you wouldn't pass the interview if you gave the SO architecture. Sad.
One of the recruiters from big tech companies told me that I need microservice experience to talk about during the interview. I told him the companies I worked for didn't need microservices. Now what? Should I convince my current company to switch to microservices?
OUTSTANDING talk. Reality is shocking isn’t it? How many engineers follow trends vs creating a well-thought-out architecture with a fully open mind? I would wager a high percentage of the former.
Stack Overflow has also existed for nearly two decades now. It was originally designed as a monolith and rewriting it to use microservices would be a tremendous undertaking. I'm sure there is an immense amount of technical debt in the backend. Yes, they are a success story for how to operate a monolith and optimize it. They also need 1.5 Terabytes of RAM on the database - a resource that is surely expensive and not an option for the majority of organizations creating a new web service. The takeaway should be that monoliths can work and sometimes it isn't worth making everything a microservice. The takeaway absolutely is not "microservices are bad."
As an on-prem system, the 1.5TB cost is a one-time thing. They buy it; they run it. That could very easily be a cheaper scenario than paying cloud fees month after month, especially if your whole infrastructure is just 9 servers. Nowhere in this video was the message "microservices are bad." Instead, the message was "stop thinking monoliths are bad." There are plenty of ways to break out a monolith in an efficient and safe way (strangle the monolith). As they grew, they could have done this. Why didn't they? Because they're proving that they don't need to. It doesn't add any value to them. Dev cost. HW cost. Maintenance cost. Expected benefits. When all factored in, my guess is the monolith wins. And in my experience, the monolith will often win over microservices. I've seen very few projects that truly need or truly benefit from going microservices. And the trend in the industry to revert back from microservices to (distributed) monoliths proves the point.
Another excellent video - thank you. I find the premise both important and amusing. The idea that using a tried-and-true approach, which might be thought of as "thinking inside the box", results in "a true testament to the power of thinking outside the box" (3:45). It's important to know what box we're in (our default mindset) when approaching any problem.
It's also important to remember they were thinking well within the trends when SO was designed. They simply failed to replace it (making it more resilient and efficient)
Monolith, together with other patterns, is only one of the options. Together with a fantastic stack overflow case, I saw tons of corporate monoliths that "killed" their team and companies, gathering all the negative consequences of the one codebase, terrible migrations, hundreds of bugs in internal cross dependencies, and lack of possibility to support their companies growth. So your possible next choice "Let's do like in stack overflow, I saw it in the blog" could be fatal for your project in the perspective.
What you describe isn't a problem of a monolith, it's a problem of a bad team. I saw multiple teams struggling implementing microservices because it is "cool". If those people can't build a monolith they will fail with microservices for sure.
And so could "lets do it like Netflix". I'd suggest starting like StackOverflow, when you're big enough to need to give services room to scale you'll know. But if you want fast, every extra network operation is a curse.
I have seen many microservices that kills the whole app. Dozens of microservices are unmanageable. Corrupted data from one microservice kills the whole app. Hundreds of bugs and terrible migrations are everyday headaches. And there is "that's not my problem" problem in addition as well. Data inconsistency leads to failed UI and angry users, the situation when some part of UI shows 3 orders, and another part shows only 1 order, is terrible. Distributed logs make it difficult to find the origin of the problem. Microservices interconnections kill performance
It seems like the architecture is designed around the data -- mostly static content. Articles and comments are stored in SQL, so there's no serving up of "files". Makes sense. I've always like the idea of microservices but having separate databases underpinning them with some kind of asynchronous bus keeping them all updated seems like a nightmare.
Thank you for publishing a thought-provoking video! Stackoverflow is an enormously useful resource. My websearches frequently direct me to it, but I might still be unaware of some of its features. From what I see, the web site serves static content in the absolute majority of cases, given that most visits are by anonymous users and even when they are logged in, I surmise they comment or vote infrequently. Clearly, it's always hard to design a system at scale. But in this particular case, I don't see a need for more than a few off-the-shelf web/file servers for articles and a moderate-size SQL DB for votes and info on registered users. Maybe they have some anti-spam logic, but I'm not sure. What am I missing? BTW, just tried their built-in site search. It kind of sucks. I deliberately mistyped Dijkstra's algorithm, switching i and j in the name. The search didn't offer me a correction, but pointed me to a number of articles with this mistyped name in their title. Bing did a much better job auto-correcting.
Regarding that search... if there are many articles with the misspelling, the system might genuinely think that's a different term and that's why it's returning those specific results. Though I don't really know.
SO is just a glorified forum, forums have been around for decades and are are mostly just CRUD operations. There is no real-time'iness about it, no huge amounts of data (think IoT, streaming, personalization/suggestions, etc). There doesn't seem to be any integrations with other large systems (CRMs, data warehouses, BI, ML, etc). They don't protentially lose large amounts of money if a number of users can't login/create a post. It's extremely read heavy, that's the easiest to design for. They are not dealing with financials/transactions/money, if something is out of date or data is slightly wrong it doesnt matter. SO still has downtime/maintenance basicslly every week or so, i wonder why...it's a sinple architecture because its a simple system.
I think Twitter is far more feature rich than StackOverflow despite me hating it. So having microservices makes sense. Reddit having a lot of microservices doesn't really make sense as that would be like 10 for every dev there :P
This is an example of a technology built for a specific business objective, rather than one built to follow latest technical trends or to achieve perfection of engineering.
@@fxfighter ye then you need to rack it, maintain hardware backups, redundant locations (different countries at least - currencies, tax, processes, laws, etc), pay someone to manage and maintain all of this security/vulnerabilities, networking, storage, patching, etc...infra engi easily 50k+ a year.
This teaches us it’s not what we do, it’s how we do it. Most of what we use is way underutilized. Before we can even scratch the service, we move on to the next trendy thing in the dev world
i think it's worth remembering that while it has a lot of traffic, it's not a complicated application. It's about as simple as they get, it's not much more than a guestbook app with bells on. And all their other sites are cookie-cutter carbon copies too. Much of the 'modern' designs are to enable rollouts by large teams working complicated apps, but stack overflow just isn't one.
Taylored ot their unique constraints and specificities. Juste great work. Micro services pile up on standardization, but we see times again that once you get bug enough, you just develop your custom stack, which in this case is quite simple and efficient. Props
You could guess that from the fact that they have regular maintenance during the night and even unplanned outages. Also remember this is initially a hobbyist Microsoft hack, turned into a two-man business plan.
I loled a bit calling a straightforward architecture unconventional. I'll go as far as saying if you put the effort to understand your system you can have this.
Great evidence of how many apps and websites are way over engineered. I’ve never felt comfortable with the modern state of development, especially JavaScript… kind of feels like a hype marketing train targeting kids/younger devs with shiny object syndrome.
it's actually targetting the older managers (Chief xx Officers, replace xx with whatever), who don't know jack about jack, but want something that looks good on their resume.
If i remember correctly because they have low cpu usage on their servers (5%-ish) they compiled their code in those machines (or at least they used to do so).
Same can be said ab any other sql instance on the planet: the buffer pool is definitely a cache. Makes you wonder why people default to drop a cache in front of the db instead of relaying on the buffer pool.
Like anything else, this video too should be taken with a grain of salt. SO will not receive traffic spike similar to that E-commerce site. Their traffic is predictable, hence on prem is sufficient for them and it's cheaper. Here's the question. What happened if all of a sudden this site received high surge of traffic similar to ticketmaster selling Taylor swift ticket?? This is the selling point of cloud computing.😊😊
A lot of modern architecture is designed to enable large teams to work on a system. SO was basically written by Joel Spolsky and Jeff Atwood - two seriously expert programmers with a lot of ability and experience (they were both prominent in the community before they got together for the project). It doesn't need dozens of programmers.
Looking back, stackoverfw was born in late 2000s. At that time, microservice best practices was not well established… after ten of years, if it works, we may not change it and leave it as legacy app. U don’t expect to make huge change anymore. If u are asked to redo stack overflow today and include many woo features like chatgpt etc, u may pick micro-service as the approach.
Yeah any startup now wants to fully go microservices style, and end up paying alot, when they could have have done proper estimates of the number of users, and the solution is a simple monolithic app...
Great reminder that engineers are gonna over-engineer. Even senior professionals are often too quick too chase the shiny new hotness, and do what’s trendy. Monoliths are fine for far more than they get credit for, and usually a good way to start a project, at the very least. Keep design and tooling simple, and minimalistic, with the idea that you might break the monolith up, later on. We’re drowning in trends, tooling, configuration, overly-complex architecture, and general over-engineering. Very few of us are building anything that requires all this stuff. We’re not all Google and Amazon!
@ I haven't seen SO being down a lot. But while SO availability certainly is a good metric. I'd more interested to know SO engineering feel stifled/restricted by the infrastructure. But as others have pointed out in the comments, SO's forum-like read-heavy pattern might fundamentally be a good match for a monolithic architecture.
I'm very curious too. Theoretically, if they have 9 servers, even if they're very beefy (think of maxed out Epyc CPUs), that's still, I think, very cheap compared to -most- all cloud solutions of the same size. And something tells me that they don't have hundreds of programmers and devops either.
Corporate web dev seems heavy on strong, under-thought opinions. Start-up culture mistakes enthusiasm for intelligence. It'll be interesting to see if that changes.
Use the right tool for the right job. I see overbuilt systems that made more issues than they solve due to complexity because they think they will be google in a year lol. Sometimes it's because some managers read some book or heard a seminar or had a consultant state that was their best solution. They end up making things worse forcing a design that does not fit the needs and bend things to fit causing issues.
The only thing simple about the monolith is the diagram with 1 block. Inside you will find multithreading multiprocessing queues and mutex...all intertwined in a complex web.
Change Data Capture (CDC) is a software process that identifies and tracks changes to data in a database. CDC provides real-time or near-real-time movement of data by moving and processing data continuously as new database events occur. You could read more on it online.
It's the basics that shine. I hear people talk big about all the cloud funky stuff but doesn't know how does even a web request travels through different OSI layers and reaches to the other end. Irony.
Keep it simple, from now on for every sys design interview I'll always go for a monolith, the ms archi is hyped up in educational circles and most tech conferences, but in reality only a few corpos are adopting it, most large scale software products in the industry are just monoliths
Only a few? Literally 1000's of companies follow microservices (or at least try to). For large corps, sure legacy stuff might be monolithic, that's not because monolithic is better, it's just that changing an enterprise system once it's been in prod for many years to anything else is a massive amount of work. Anything in the last 5 years or so (of a decent scale) will more than likely be microservices (again, at least try to).
@@M3t4lstorm That's the problem, most large orgs are not running systems that were built in the last 5 years, and let's not underestimate the skill levels it requires to design and run them successfuly, that's not evenly distributed.
I know this video is 4 weeks old. But 1.5 TB of RAM on a server is nothing special these days, especially considering you have multiple CPUs on a motherboard
I hate microservices and ive never understood why engineers chose to jump on that bandwagon. It's difficult to maintain and notoriously pron to odd hiccups. Also, lack of database integrity because you have to split your tables and tie them to different services. Smh
There is absolutely nothing "Shocking" about this architecture, it's the obvious choice for 99. and a quite a few 9s percent of the businesses out there, but that would require most companies to have people with actual skills making the technical decisions and not Twitter-driven, hype-addicted tech bros.
Rolling deployments with load balancer node draining/readying (HAProxy sits in front of the web servers). Google "nick craver how we do deployment" for all the detail on Stack Overflow's deployment.
Monoliths have little to do with downtime deployments. I've worked with many monoliths with 0 downtime deploys over the years. It's almost always a form of rolling deployment, often with primary/secondaries where you switch a secondary to the primary while you stop routing traffic to the old primary during upgrades.
seeing the monolithic diagram fills me with calmness
I come from the school of Joel Spolsky and his heydey as a writer and blogger (before he retired from writing and went on to found SO) so I am not one bit surprised at the efficiency and practicality to solve the problem at hand. I have seen multiple engineering teams and piles of tech thrown at problems that would be easily solved by a tried-and-true setup like this (albeit with a fraction of the DB server memory... in one case arguably a properly configured lone Raspberry Pi could have handled the traffic...). I love your videos. Thanks for sharing something that reminds us all to consider the alternative and see the forest for the trees :)
1.5 tb of Ram for the DB instance is the game changer 😎 why to create an external cache when the DB engine is capable of but vertical scaling has its own limit .
It's a meme. Efficiency my ass.
I know of instances with 4TB ram and orders of magnitude less traffic struggling. 1.5Tb could have been wasted easily, but they used it well.
@@CatalinAdler What is the industry and company size of teh compaines you are talking about
@@skyhappy Anything that matters has RDBs with terabytes of RAM: governments, finance, medical, insurance...
@@themrubios You mean all important but un-optimized shit?
"An idiot admires complexity, a genius admires simplicity"
I entirely agree with this, but one could argue that microservices are simpler than monoliths. The real issue is the folks at Stack Overflow are trying to keep the lights on and a rewrite for microservices would be a massive undertaking. It's more manageable for them to continue optimizing their monolith. Both are viable solutions. But you don't just throw the newest trend on a decade+ old tech stack. Just like you wouldn't try converting a stable microservices based solution into a monolithic one.
@@jacobmills3076 Thats not the only reason.
Microservices does not automatically make things faster or more scalable, a well designed monolith can be both faster and more scalable depending on your use case.
So before going for microservices, you should know why MS is the preferred solution for you, is it team management. If you have hundreds of developers, merging and testing all different changes can prove to be a source of bugs and delays and require massive amount of extra testing in a monolith.
If you have different parts that have vastly different scaling requirements that could be a good reason to separate them into separate services where some might run 1-3 instances while others run a hundred spread out over multiple datacenters and physical computers.
SO is running with a small stable team and their application scales uniformly, so they can just run more instances of the same monolith.
And its mostly reading, not updating or complex relations with changing type of queries.
That makes a monolith easier to administer.
@davidsmartensson yes, they are only a view people who already build a working monolith.
Also Stack Overflow is a rather simple service where view provide text for many people. There is not much to you could abstact in a micro service.
This is only "unconventional" because most software developers are really bad software designers. Stack Overflow is showing a mastery of software design that most of the rest of the industry should learn from. Don't sold problems by adding more layers of complexity. Solve problems by finding the simplest solution that works. Such a rare skill in this industry.
" by finding the simplest solution that works",hey, that's micro-services
I push back against distributed microservice architectures whenever I can. It's a great tool, but it's not a silver bullet and has its tradeoffs. I've seen too many teams go for microservice or distributed monolith systems when a better first solution is just.. monolithic. Monoliths aren't inherently bad.
I remember pushing back during an interview about going microservices and big data when the problem set was in a few gigs and relatively small data.
I was looked like as if I was a moron.
If you can't build a monolith, you're going to build a distributed ball of mud. It's gonna happen.
@@robertluong3024what was the job position u we’re interviewing for if u don’t mind me asking
Buh my resume!!!
i strongly agree with you. Idk why we like trends so much and then it sucks because people want others to know the technology when in all honestly probably not even the best solution and its gain so much tech debt they dont care😭
like i turn my head and everyone wants to use the next big think. That why trend will never help you learning just fundamentals and innovation and deep understanding of problem solving is a goldmind
absolutely love this episode. a real world example (even at scale) showcasing the importance of thinking outside the box and being practical.
thx for making and sharing this video!
I can understand "no cloud" because Cloud is very expensive when used for a big always-on service. On-prem is still relevant today because you can maintain absolute control, you just have to make sure your Infrastructure guy has built you a resilient environment to live in. Modern business keep reading the internet buzz and making technical decisions they know nothing about, or as I like to put it "an Exec and his budget are soon parted". I've been in this business for nearly 40 years and I've seen a few great innovations, but most change has simply been re-inventing the wheel that was working perfectly well before (ah, the cycles I have seen repeat). Clouds et al works very well for some situations, but it isn't the panacea for all problems.
I've been in the industry for 20 years (another 20 to go i guess? 🙂) and i've seen everything you've just described.
Personally, I hate cloud as much as i like it. I can see the appeal, and how it can be really useful, but it's just made my life as a network engineer 5 times more complicated than it ever needed to be. Everything my employers are doing in cloud they could do just as easily on-prem for a fraction of the cost. They just move it in cloud because they think it's cool and looks good on their resume. I'm just waiting for the next fad when they'll want it all back on-prem again.
Lol I love this. I even have a feeling the stack overflow codebase is being maintained by 2 people once every month. I have never met/heard of an engineer working at Stack Overflow 😂
I’m working for a small company where I’m head-everything for a software project. I only got one parttime coworker.
The software should be handling several honderds of users per month. The initial idea is to use micro services but monoliths with a few “external dependencies” feel better to design/create/maintain/extend.
Might sound weird, but we’re crossing away reasons to create a micro service architecture.
Thanks for sharing. I always thought the big tech's stacks are over engineered. I always believed in stacks like SO's. But you wouldn't pass the interview if you gave the SO architecture. Sad.
True, I find some part of the IT industry is based on hype & trends .... and some lethargy. I am no saint though :)
One of the recruiters from big tech companies told me that I need microservice experience to talk about during the interview. I told him the companies I worked for didn't need microservices. Now what? Should I convince my current company to switch to microservices?
Now you’d need to integrate AI too.
OUTSTANDING talk.
Reality is shocking isn’t it? How many engineers follow trends vs creating a well-thought-out architecture with a fully open mind?
I would wager a high percentage of the former.
Stack Overflow has also existed for nearly two decades now. It was originally designed as a monolith and rewriting it to use microservices would be a tremendous undertaking. I'm sure there is an immense amount of technical debt in the backend. Yes, they are a success story for how to operate a monolith and optimize it. They also need 1.5 Terabytes of RAM on the database - a resource that is surely expensive and not an option for the majority of organizations creating a new web service. The takeaway should be that monoliths can work and sometimes it isn't worth making everything a microservice. The takeaway absolutely is not "microservices are bad."
As an on-prem system, the 1.5TB cost is a one-time thing. They buy it; they run it. That could very easily be a cheaper scenario than paying cloud fees month after month, especially if your whole infrastructure is just 9 servers.
Nowhere in this video was the message "microservices are bad." Instead, the message was "stop thinking monoliths are bad." There are plenty of ways to break out a monolith in an efficient and safe way (strangle the monolith). As they grew, they could have done this. Why didn't they? Because they're proving that they don't need to. It doesn't add any value to them. Dev cost. HW cost. Maintenance cost. Expected benefits. When all factored in, my guess is the monolith wins. And in my experience, the monolith will often win over microservices. I've seen very few projects that truly need or truly benefit from going microservices. And the trend in the industry to revert back from microservices to (distributed) monoliths proves the point.
Nah, that's definitely the takeaway
@@CasparHarmer Then you missed the entire point
Another excellent video - thank you. I find the premise both important and amusing. The idea that using a tried-and-true approach, which might be thought of as "thinking inside the box", results in "a true testament to the power of thinking outside the box" (3:45). It's important to know what box we're in (our default mindset) when approaching any problem.
It's also important to remember they were thinking well within the trends when SO was designed. They simply failed to replace it (making it more resilient and efficient)
Monolith, together with other patterns, is only one of the options. Together with a fantastic stack overflow case, I saw tons of corporate monoliths that "killed" their team and companies, gathering all the negative consequences of the one codebase, terrible migrations, hundreds of bugs in internal cross dependencies, and lack of possibility to support their companies growth. So your possible next choice "Let's do like in stack overflow, I saw it in the blog" could be fatal for your project in the perspective.
What you describe isn't a problem of a monolith, it's a problem of a bad team. I saw multiple teams struggling implementing microservices because it is "cool". If those people can't build a monolith they will fail with microservices for sure.
And so could "lets do it like Netflix". I'd suggest starting like StackOverflow, when you're big enough to need to give services room to scale you'll know. But if you want fast, every extra network operation is a curse.
When you say you saw corporate monoliths is this when you worked there
I have seen many microservices that kills the whole app. Dozens of microservices are unmanageable. Corrupted data from one microservice kills the whole app. Hundreds of bugs and terrible migrations are everyday headaches. And there is "that's not my problem" problem in addition as well. Data inconsistency leads to failed UI and angry users, the situation when some part of UI shows 3 orders, and another part shows only 1 order, is terrible. Distributed logs make it difficult to find the origin of the problem. Microservices interconnections kill performance
@@ihateidiots9484 Sounds more like a bad Dev team/system design.
It seems like the architecture is designed around the data -- mostly static content. Articles and comments are stored in SQL, so there's no serving up of "files". Makes sense. I've always like the idea of microservices but having separate databases underpinning them with some kind of asynchronous bus keeping them all updated seems like a nightmare.
For small scale businesses microservices is overeengineering and suicidal. Every business is not Facebook/Netflix.
Thank you for publishing a thought-provoking video!
Stackoverflow is an enormously useful resource. My websearches frequently direct me to it, but I might still be unaware of some of its features. From what I see, the web site serves static content in the absolute majority of cases, given that most visits are by anonymous users and even when they are logged in, I surmise they comment or vote infrequently.
Clearly, it's always hard to design a system at scale. But in this particular case, I don't see a need for more than a few off-the-shelf web/file servers for articles and a moderate-size SQL DB for votes and info on registered users. Maybe they have some anti-spam logic, but I'm not sure. What am I missing?
BTW, just tried their built-in site search. It kind of sucks. I deliberately mistyped Dijkstra's algorithm, switching i and j in the name. The search didn't offer me a correction, but pointed me to a number of articles with this mistyped name in their title. Bing did a much better job auto-correcting.
Regarding that search... if there are many articles with the misspelling, the system might genuinely think that's a different term and that's why it's returning those specific results. Though I don't really know.
SO is just a glorified forum, forums have been around for decades and are are mostly just CRUD operations. There is no real-time'iness about it, no huge amounts of data (think IoT, streaming, personalization/suggestions, etc). There doesn't seem to be any integrations with other large systems (CRMs, data warehouses, BI, ML, etc). They don't protentially lose large amounts of money if a number of users can't login/create a post. It's extremely read heavy, that's the easiest to design for. They are not dealing with financials/transactions/money, if something is out of date or data is slightly wrong it doesnt matter. SO still has downtime/maintenance basicslly every week or so, i wonder why...it's a sinple architecture because its a simple system.
Well a lot of crud web apps are still managed by a team pushing for microservice architecture... about 90% of the time lol
Reddit and Twitter also experience downtimes all the time and I bet they use tons of micro services.
I think Twitter is far more feature rich than StackOverflow despite me hating it. So having microservices makes sense.
Reddit having a lot of microservices doesn't really make sense as that would be like 10 for every dev there :P
"Just CRUD"??? Oh man. EVERYTHING falls under CRUD.
In the end, everything is just CRUD. Other fall under system design and infrastructure
Ok this knocked my socks off
This is an example of a technology built for a specific business objective, rather than one built to follow latest technical trends or to achieve perfection of engineering.
It's a big f**k y*u to all the folks onboard the tech hype train. Loved it.
Yes, a big F U to the trend of not using a host with 1.5 TB RAM
@@thedragonrises6882 It's pretty cheap to make your own server with this amount of ram though, cheaper than running a bunch of cloud servers.
@@fxfighter ye then you need to rack it, maintain hardware backups, redundant locations (different countries at least - currencies, tax, processes, laws, etc), pay someone to manage and maintain all of this security/vulnerabilities, networking, storage, patching, etc...infra engi easily 50k+ a year.
This teaches us it’s not what we do, it’s how we do it. Most of what we use is way underutilized. Before we can even scratch the service, we move on to the next trendy thing in the dev world
We are going back to where we come from like now going from client side rendering to server side rendering and also from microservice to monolith
i think it's worth remembering that while it has a lot of traffic, it's not a complicated application. It's about as simple as they get, it's not much more than a guestbook app with bells on. And all their other sites are cookie-cutter carbon copies too. Much of the 'modern' designs are to enable rollouts by large teams working complicated apps, but stack overflow just isn't one.
Simplicity is always the best.
Taylored ot their unique constraints and specificities. Juste great work.
Micro services pile up on standardization, but we see times again that once you get bug enough, you just develop your custom stack, which in this case is quite simple and efficient.
Props
You could guess that from the fact that they have regular maintenance during the night and even unplanned outages.
Also remember this is initially a hobbyist Microsoft hack, turned into a two-man business plan.
I loled a bit calling a straightforward architecture unconventional. I'll go as far as saying if you put the effort to understand your system you can have this.
Great evidence of how many apps and websites are way over engineered. I’ve never felt comfortable with the modern state of development, especially JavaScript… kind of feels like a hype marketing train targeting kids/younger devs with shiny object syndrome.
it's actually targetting the older managers (Chief xx Officers, replace xx with whatever), who don't know jack about jack, but want something that looks good on their resume.
I would argue that because it hasn't fallen over, simplicity sometimes does work!
They have plenty of outages. Planned and less planned.
Love the videos, but is there a source (blog post) for the architecture?
If i remember correctly because they have low cpu usage on their servers (5%-ish) they compiled their code in those machines (or at least they used to do so).
Congratulations on your channel, I really enjoyed your content. How do you make the illustrations? What software do you use to animate?
With 1.5 terabytes of ram in the sql database, isn't that pretty much like caching? Reading from Ram is going to be faster than reading from disk.
Same can be said ab any other sql instance on the planet: the buffer pool is definitely a cache. Makes you wonder why people default to drop a cache in front of the db instead of relaying on the buffer pool.
Like anything else, this video too should be taken with a grain of salt. SO will not receive traffic spike similar to that E-commerce site. Their traffic is predictable, hence on prem is sufficient for them and it's cheaper.
Here's the question. What happened if all of a sudden this site received high surge of traffic similar to ticketmaster selling Taylor swift ticket??
This is the selling point of cloud computing.😊😊
A lot of modern architecture is designed to enable large teams to work on a system. SO was basically written by Joel Spolsky and Jeff Atwood - two seriously expert programmers with a lot of ability and experience (they were both prominent in the community before they got together for the project). It doesn't need dozens of programmers.
If you think microserivces are the solution to code conflicts then I don't want to work where you do
What software you use to prepare such creative videos?
Looking back, stackoverfw was born in late 2000s. At that time, microservice best practices was not well established… after ten of years, if it works, we may not change it and leave it as legacy app. U don’t expect to make huge change anymore. If u are asked to redo stack overflow today and include many woo features like chatgpt etc, u may pick micro-service as the approach.
For what reason? What it would improve?
It feels that many people choose micro services just to impress other people.
Good point. The architecture may have been the best solution at the time, with no incentive to change as new patterns emerged.
Your videos are very informative. Thanks!
Was it really "thinking outside of the box" when it was first designed?
Yeah any startup now wants to fully go microservices style, and end up paying alot, when they could have have done proper estimates of the number of users, and the solution is a simple monolithic app...
I've seen rather well-payed people design micro-service architectures for systems with fewer than 100 transactions/requests per second. Madness.
You know performance isn't the number 1 priority of a microservice architecture right? Nor is a deciding factor if you should use MS or not
Deeper understanding >> Trends
In tech nothing is Outdated. It's all about your requirements & understanding!!!
Great reminder that engineers are gonna over-engineer. Even senior professionals are often too quick too chase the shiny new hotness, and do what’s trendy. Monoliths are fine for far more than they get credit for, and usually a good way to start a project, at the very least. Keep design and tooling simple, and minimalistic, with the idea that you might break the monolith up, later on.
We’re drowning in trends, tooling, configuration, overly-complex architecture, and general over-engineering. Very few of us are building anything that requires all this stuff. We’re not all Google and Amazon!
I would love to hear from stackoverflow devs and devops about how they feel about their monolith.
Probably fine? How often has SO been down?
@ I haven't seen SO being down a lot. But while SO availability certainly is a good metric. I'd more interested to know SO engineering feel stifled/restricted by the infrastructure. But as others have pointed out in the comments, SO's forum-like read-heavy pattern might fundamentally be a good match for a monolithic architecture.
Okay but let’s talk about costs?
Have you seen cloud costs lately?
@mattstyles yes, lambda are a no-go 😢 on-prem is cheaper at SO scale, for their use-case.
I’d love a cost breakdown, or an estimation per DAU
I'm very curious too. Theoretically, if they have 9 servers, even if they're very beefy (think of maxed out Epyc CPUs), that's still, I think, very cheap compared to -most- all cloud solutions of the same size. And something tells me that they don't have hundreds of programmers and devops either.
Cheaper than cloud costs, they've reviewed this many times over the years and their setup has always been cheaper.
Just Amazing 😍
Corporate web dev seems heavy on strong, under-thought opinions. Start-up culture mistakes enthusiasm for intelligence. It'll be interesting to see if that changes.
Aren’t most applications built around the same time stackoverflow started using the same architecture? What’s the surprise ?
I like all your videos thank you.
another amazing video
Small typo on the right side of the thumbnail "Wha it actually is" instead of "What it actually is"
Use the right tool for the right job. I see overbuilt systems that made more issues than they solve due to complexity because they think they will be google in a year lol. Sometimes it's because some managers read some book or heard a seminar or had a consultant state that was their best solution. They end up making things worse forcing a design that does not fit the needs and bend things to fit causing issues.
It makes sense though because they only really provide one service: a forum
Very interesting and insightful
gigachad stackoverflow
I’m doing similar thing in our company. We even batched all business logic in procedures. Hell that thing performs 🤟
Thanks again ❤
Thanks
Yes what's the cost here? Would it be cheaper if we build it differently?
The only thing simple about the monolith is the diagram with 1 block. Inside you will find multithreading multiprocessing queues and mutex...all intertwined in a complex web.
Hello,
May I ask you which softwares do you use to make these animations in the videos. It’s quite interesting.
Stack Overflow will never overflow
#Monolith #Rocks 💪 with its simplest design.
What is CDC at 1:03??
I think it's Change Data Capture. Basically a mechanism to detect changes in data. Like an SQL-Trigger for example.
Change Data Capture (CDC) is a software process that identifies and tracks changes to data in a database. CDC provides real-time or near-real-time movement of data by moving and processing data continuously as new database events occur.
You could read more on it online.
Maybe stackOverflow has a reason for doing it this way? Would be interesting to get reasons behind their decision making.
It's the basics that shine. I hear people talk big about all the cloud funky stuff but doesn't know how does even a web request travels through different OSI layers and reaches to the other end. Irony.
curious about its availability metrics. how do they maintain that? hot backup?
That means most to of the distributed microservices might just be over-kill
Please make video on substack system design
That's actually insane. Gives you a different perspective, huh..
Thats interesting. If I propose the same architecture in a design interview today, I will never make progress to the next rounds :)
All technology have tradeoff.
That's awsome
Facinating
I don't have to worry about new trends, I take too long to consider them and by the time I do, they are old news 😂
What does StackExchange look like underneath the hood?
What is the source for this?
I think not collecting and track8ng users giving them edge becuz without login they don't need to write user data to data bases that leads to this
wait.. 1.5tb ram for each instance?? that are 9 super instances instead of normal instance like 2c/4g!!!
awesome
So stack overflow architecture is a hand cannon 😂
The only thing I know they use is C#.NeT on the backend
monliths are best, microservices are just a decoy of spending more designed by cloud service providers.
I think I’m respondisble for about 1 billion of those pageviews 😝
now, ont the efficinecy of the stackoverflow community... well, that's a completely different story
Bro might have found this design in a stackoverflow answer
if vertical scaling is not an issue, then all is well
Keep it simple, from now on for every sys design interview I'll always go for a monolith, the ms archi is hyped up in educational circles and most tech conferences, but in reality only a few corpos are adopting it, most large scale software products in the industry are just monoliths
Only a few? Literally 1000's of companies follow microservices (or at least try to). For large corps, sure legacy stuff might be monolithic, that's not because monolithic is better, it's just that changing an enterprise system once it's been in prod for many years to anything else is a massive amount of work. Anything in the last 5 years or so (of a decent scale) will more than likely be microservices (again, at least try to).
@@M3t4lstorm That's the problem, most large orgs are not running systems that were built in the last 5 years, and let's not underestimate the skill levels it requires to design and run them successfuly, that's not evenly distributed.
thanks grt vid!
Which database is used? Must be some kind of NoSql
Stack Overflow's Architecture: 🗿
I know this video is 4 weeks old. But 1.5 TB of RAM on a server is nothing special these days, especially considering you have multiple CPUs on a motherboard
To a man with a hammer, everything looks like a nail . Shouldn't be the case ...
I hate microservices and ive never understood why engineers chose to jump on that bandwagon. It's difficult to maintain and notoriously pron to odd hiccups. Also, lack of database integrity because you have to split your tables and tie them to different services. Smh
if SurrealDB becomes a norm i think we won't even nee message brokers and redis as much
There is absolutely nothing "Shocking" about this architecture, it's the obvious choice for 99. and a quite a few 9s percent of the businesses out there, but that would require most companies to have people with actual skills making the technical decisions and not Twitter-driven, hype-addicted tech bros.
KISS ❤
Ohh ok so all we need is 1.5 TB of RAM....
But how they deploy updates without downtime?
Rolling deployments with load balancer node draining/readying (HAProxy sits in front of the web servers). Google "nick craver how we do deployment" for all the detail on Stack Overflow's deployment.
They do have downtime sometimes
Monoliths have little to do with downtime deployments. I've worked with many monoliths with 0 downtime deploys over the years.
It's almost always a form of rolling deployment, often with primary/secondaries where you switch a secondary to the primary while you stop routing traffic to the old primary during upgrades.