MircroServices (and a story about Netflix) | Prime React
Vložit
- čas přidán 10. 03. 2023
- Recorded live on twitch, GET IN
Perhaps the greatest meme master: • Microservices
Author: / @krazam
/ theprimeagen
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord - Věda a technologie
all happy developers are alike... every unhappy developer is unhappy in their own way
pppppppft you speak as if you know... you weren't there man... _you weren't there!!!_
The “Anna Karenina” dev principle
This is the best crossover episode I could've witnessed ❤
What is this thing you call a happy developer?
@@godowskygodowsky1155 A retired developer
So much work for something where everyone I know was like "Oh weird it starts with season 4, better switch to S1E1 manually"
Hard agree, everyone I know thought it was a bug and manually changed to S01E01
This so much, wasted effort imho
This is a text book example of something "cool" that someone in management, marketing, UX, etc comes with
That makes engineering a suffers for a while for little or nothing.
In these situations I always remind myself "errybody gettin paid"
loool, is that what they ended on? Just pointing you to a different season? I assumed they were switching the metadata to make S1E1 a different video, like they did with Love Death and Robots
The fact that this problem arised serving up Black Mirror contains irony that is not lost on me.
“arose”
@@gregoryshields4258 "penis"
“arisen”
"aroseth"
"arizethed"
Watching you draw S's like a psycho feels like Black Mirror
That's how left handed people write S's ;)
What's wrong with the way he writes the 'S'?
@@bdlowery Nah, I'm left handed and I don't write my "s"es like that. Don't put that on just us 😂
@@bdlowery he's not left handed tho :)
holy shit, now I can't unsee it, the way he writes the four is also weird for me
Somewhere around 11 mins, I just started laughing as it was so clear he was completely re-enacting the video
its... literally the netflix architecture when i did the change... its not a joke, which hurts.
@@ThePrimeTimeagen I'm about to start a Mastercard internship using Java Springboot... so I'm looking forward to understanding the pain :D
@@linusdannull1247I'm one month into my internship at AMEX and the pain is growing every single day.😞
4 months and multiple services for something most people considered to be a bug. I now look at my boring job in a whole different light, thanks for that. 😀
Yes, but you don't make tens of thousands of dollars a month like in big tech lmao
This skit raised my blood pressure, but the PM's reaction at the end brought it back down.
Then your diagram raised it again.
Yes but the middle name thing is like "we'll be going through this again very soon"
Wow I remember trying to introduce someone to Black Mirror after S4 but i had already watched it, and i remember how S4 would play first and it was really confusing. Can't believe that was you lmao
Gotem
@@ThePrimeTimeagen galactus at it again, predicted time and space for the episodes that must be watched before the viewer knows what they want to watch
Lol, my friends and I just switched to S1E1 manually because it started from S4 🤣😂🤣
I feel bad for you Prime 😅
I was showing my younger brother this video and before you got around to your story. Black Mirror was mentioned and he said 'Oh, the show where they screwed up the order of the seasons.'
The fact that he can just rebuild this whole system from memory shows how much he had to deal with all this every day
LoL I initially thought you meant the K video and not actual primagen😂
People got paid like $300k a year to come up with this solution... and it took 4 months...
Yep. Starting to think big tech is just a giant ponzi scheme
And all the end users thought it was a bug and manually changed it to S01E01😂 Brilliant.
@@TheMsksk Must be one of those more expensive bugs
@@TheMsksk It was a bug, it just happened to be written by a product manager at the spec level, instead of by a programmer at the code level.
@@ricardoamendoeira3800 I like this way of looking at it
Learned a lot today love Content Metadata Service uh, but we'll talk next week about adding a new reverse button player. Awesome sounds good, take care. 😂😂👌💪
Lmao 💀
*Roto-levitates out of room*
I love the Netflix story here. Perfect example of microservices in practice.
it's a mess. microservice given time doesn't really makes sense for the use case it's intended for. usage and complexity grows.
Nope, MS should never be overcomplicated, they should be simple individual jobs of.execution. this.type.of overcomplification is just showing "rush.to.complite the job", not ever take a second to think and design proper solution.
Basically is.showing difference between Developer and Engineer. Prime is a developer, very good.and experienced developer, but Engineer would solve this quite differently
When you have 3600 hammers connected with unique cog sizes in a 4D map with multiple space time continuum flaws, everything starts to look like a nail to you.
Prime... I would've just hardcoded the Black Mirror ID and sort that out hardcoded on the front-end.
Yes, do I that crap when they want things for yesterday, I do it in a 1~2 days and then I would take the 4~6 months to do it properly and remove the Hardcoded stuff, I call this: "Tech Debt Quantitative Easing" as Manager most likely would never want me to pay that debt.
I hate that I can relate to this. We have feature flags, but there's no backend bandwidth so let's just hardcode org IDs in the UI application and we'll clean it up never.
@@TheBswan Yes the universal word for "later" and "another point in time" in tech is "never".
@@dealloc make sure to put a TODO comment in code so that you can read it in 5 years and see how useful it was.
@@rand0mtv660 LMAO. Those TODO comments are like troll posts from the distant past all around in codebase 🤣😂
@@rand0mtv660 "FIXME: This is a hack that should be fixed later"
OMG, I was like "Man, what does Black Mirror specifically have to do with microservices?".
Hold onto your monoliths tight peeps. As long as you can, never let go.
I'm currently working at a place that does a hybrid microservice /monolith style of architecture. The decision to structure it this way came about from specific needs (think content creation vs content moderation) and many code refractors (services being deprecated quickly so it's hard to rely on one). The monolith single responsibility app consistently performs better than the microservices based apps.
I think we'll see the industry start to realize sometimes it's better to have a tiny monolith single org responsibility style repo vs microservices hell.
I right now work on a fe that support multiple themes ( think of same app for different country or slightly different ui ). Project manager who is also main be dev made a decision to split one int o a separate repo ( which dev ops need to set up and they don't seem to have enough time as it is ). I could not figure it out how it will make anything easier for us since you don't need to share code if it is the same repo what difference is there if a directory is in the same repo or not?
Then I got a little story how they have 20 repos on the back end and ever since then I don't bother to even try sway anything anymore. Few months back we got 2 user tokens so 1 user could change and see data of the other user. But you have to send the token from the correct user to see what is going on.
If I remember correctly, Netflix used to have a monolithic architecture. They eventually switched to a microservice architecture because they realized the cost of developing and maintaining a microservice architecture is lower than the loss of revenue caused by more frequent crashes from a monolithic architecture.
Are you sure the service you couldn't remember why called Galactus?
Irony is that the way he tells it, it feels like those 4 months were a black mirror episode in itself
Leseon learned: Dont use "smart" names for your services. Racoon, bingo, etc. It gets easier to communicate later to other devs if every service has a boring name related to what it is supposed to do.
Bonus points if you can avoid acronyms
@@zyzlol Yep, exactly. I've worked with both. I find the "smart" names, or random names, easier. Cause then you're saying real words rather than just spewing letters that are ultimately meaningless.
The real thing to avoid is making services with illogical responsibilities. If something is a bit of logic, put it in a library. If something stores some data, make it a service. And if you're going to make a service, THINK about it. But in large organizations people are looking for something to do to prove themselves, because the organization over-hired. Then everyone wants to make a service, and you get nightmare fuel.
I disagree, every service must be named according to a matching Warhammer 40k unit, obviously.
Except for Video Metadata Service... which didn't store any video metadata.
I thought the lesson was don't fall for the microservice scam
Some years ago, I watched that video and did not feel much because I was a student. Now as someone who has recently done a new feature with many microservices involved, I totally feel the pain. Sometimes I wonder, are microservices actually a better way than monoliths to do things.
Spoiler: Fucked up how everybody abandoned him. He took one for the team and they just left him to hang...
The problem with software engineering is that there is no such thing as a senior or we got to point where because there are so few senior software engineers that the blind started leading the blind.
Back in the day, we were building this beautiful machine that is sending us bunch of data from space. Me being a young engineer that is full of ideas brought up the idea of using micro services to process the data and it sounds like this micro service idea help a lot with scalability.
A engineer on the team that having been coding since the 80s. Went home and did a bunch of research on micro services.
What he found out was shocking.
It turns out that micro service are services with the world micro attach to it.
After studying SOAs in depth. I came to the same conclusion also..
I always wondered why the episode numbering for Black Mirror was broken. Figured someone dropped a bollock in the database, never occurred to me it might be deliberate... 😆
So much energy and network bandwidth used for something so simple. Feels like rather than writing code for business logic, we're calling services and composing their results to build the logic up.
Then again, I'm just a student and have no idea about the kinds of compromises that happen in production code.
I am also a student but I guess this much complexity is because of Scalability. Simple things are complex to do but large scale stuff become easier I guess.
@@reralt I feel this is because they're using the same components that are used to solve bigger issues, to solve small issues. It seems like you send a condition to a service that evaluates it to true and false returns the result to you - kinda thing.
@@VivekYadav-ds8oz I agree. Basically all they do is build abstractions based on some assumptions that are invalidated by a business requirement later on. So they have to do all of this patching and jumping around just so that their existing abstractions can be reused. Modifying your existing abstractions is a no-go because it can break more things than you can imagine. I find it funny to be honest.
@@TheMsksk Yeah this! You put it into words better than I could. It is funny tbh yeah XD
back in the days, when you're not a "student" microservice isn't a thing. and suddenly gets trendy and becomes the "norm" as with any innovation that comes in the industry. and then goes to your curriculum in school. until we learn something in the industry.
This is your best video by far, loved it.
I have so many feelings from this because currently this is my life at work. I nearly wrote a giant paragraph about how this made me feel then needed to delete it sigh.
Oh man. I’ve been on the internet long enough I basically delete anything past a certain number of lines. All but guaranteed to be a toxic vent on unsuspecting strangers if you go on that long, and you feel like shit afterward. I feel you.
For what its worth, I went back and rewatched the PF video after this.
Positive side effect to microservice architecture.
Convinced that if we made web devs make big online video games everything would take a year to make, would never be patched and would run at 2 fps
Dead right apart from the patching. Everyday it'd be patches to patch the patch's patch.
But as some defense to my tribe, for all the complexity of game dev, we have one problem they don't. Let's forget multiplayer for a second coz I promise their backend is just as webby tangled as ours'. Single player is the core and there the task is just making the best game that can get the most out of one computer.
We server side engineers are trying to make a massive brain out of thousands of networked computers with the assumption that they're never all working. It has to be able to take some wrench strikes to the head on a bad day and keep limping on.
That means your system has to be architected a lot less like a single expert airplane pilot and more like a committee of various flaky halfwits who each know how to operate one control or are middle management.
I mean that's the microservices way. You could do it the Google monorepo way but you need their insanely advanced internal development platform for it to stand a chance.
@@Sonsequence Is it really that much harder than what Fortnite does? or better yet an MMO? and if so is it just self imposed difficulty?
True. But convinced also of the opposite. Ask game devs to realize an enterprise web platform, and you'll endup with 10s responses and unmaintainable stuff.
Just how much data is replicated across all these services?
Yes
Was the craziness of all this explained to the person(s) who wanted that feature?
Maybe after seeing the cost of developing all this would have made them change their mind
The original seasons were better than the Netflix ones as well anyway lmao.
I've tried explaining to the affected that the solution they were asking for, even if it seems simple, would complicate our current architecture because it was essentially a departure from the initial spec of the project, but it's hard to make someone who only thinks in features instead of maintainability actually see any value in forgoing a feature for a healthier stack.
This is so incredibly accurate.
recently finished helping modernise a monolith. Treasure your microservice architecture. you think that takes a while to get stuff done. 25 years of code piled on top. took me months to migrate just the db out
Lol and I thought the system I’m working on right now is a hot mess. Thank you for making me feel a little better about my job.
That guy could tell you some stories though. I heard he was later teaching Javascript to a small pack of primates.
I LOLd at VMS. Maybe I'm slowly getting the microservice scepticism. I'm definitely doubling down on 1.) a minimum number of layers in an architecture and 2.) well defined layers and terminology.
It relatable to me as well. Business guys don't care what you working on they just want a result.
Oh wow, thanks for the background. The opposite season sorting on black mirror always just looked like a bug 😅
LMAO i now remember s4 thing, and I remember also how somewhat satisfied I felt “you’ve a bug there Netflix, gotcha!” 😂
it was both cathartic and gave me extreme existential dread.
I'm still not quite sure why you need to do all of this to reverse the season number. Why can't the website team put some special cases in JS code for Black Mirror? Is it just for the sake of decoupling frontend and backend?
As the team leader for the BE team on the project I'm working on, Primes story made feel better on two fronts. First of all I feel better about our microservice architecture after seeing that diagram that prime drew. But the best part is when he said how the front end guys complained about the naming of the flag... like yesss thanks for sharing this, now i feel like this is something that is just happening to me!
P.S.
Love you Prime but the way you write the letter s should be illegal xD
well, i feel that the architecture could be better. Microservices are fine but what do you think about just using a distributed database like yugabytedb where each microservice can read the data it needs. I mean the goal of these services is to provide data, isn't it. So why not let the database handle the scalability issues when you want to provide data to all the users. Even if you call other microservices to access data in the end you will create a request to a database, you added just a lot of network requests
@@hfspace That sounds super cool to me, i would be down to try something like that.
But sometimes you are constrained with the tech you can use. For example if because of budget constraints you can only use the cheap AWS serverless services like lambdas with DynamoDB and do as best as you can with that.
But what you said sounds super interesting to me :)
@@blackfrog1534 well, at my client, we also don't do it like that. I would also like to try it out, but i did not have the opportunity, yet. In my mental model micro services are the ones handling the application logic, but the data access would be managed by such a distributed database, which makes sense imo as you want to protect data at the source.
@@hfspace What happens if the data is one thing when one microservice reads it but changed before another microservice reads it, etc? Then they have different data from the same place. How is this handled? Who knows what is the correct data and sync it?
@@dealloc one thing: you define responsibilities. Only the microservice who owns it, writes to the table. That would at least be my suggestion. The other microservices can only read which will be needed for aggregation of data.And you can set up roles for that in the database. For reading before writing, i think that problem can happen all the time even in the architecture as described here in the video
I’m just now watching this… PF is legendary 😂
Krazam is the best! Thank you for doing this. 😂😂
I knew there was something fucked up when I went to talk about the episodes with friends and I realized the order of the episodes everyone watched was different from mine.
I fainted 3 times while watching the Netflix microservices graph grows.
Do I even want to ask what changes needed to be made to support the varied episode ordering of Kaleidoscope?
🤯🤯🤯🤯
I think this is true with any service that has been around long enough and has accumulated a lot of dependency…
now I'm just an itty bitty teeny tiny fullstack web dev, so my understanding is limited, but would you say that backend was over-engineered? or was every services' existence justifiable?
This is why everything is so buggy and slow despite our blazing fast hardware
when you're dealing with the scale of Netflix, things like this happen
It's over engineered. The system starts simple but as you add more features, adding another service becomes the fastest way to solve business problems.
The problem is simplifying the system is hard to justify and even harder to quantify. Deprecating and migrating services can take a looong time. All these factors make deprecation projects risky. And so... things just tend to get increasingly complicated over time.
@@videoguy640i feel the problem here is mostly the restrictive access to data for a service when you have multiple disconnected datasources which are all associated to their micro service.
@@videoguy640 Just throw more money and bodies at the problem.
It is funny to me looking back - micro-services done well are identical to what we already had established and working in the 1990's (even 80's) with SOA. :)
Ngl got an ad for jimmy johns and thought it was a micro service. Good job you got me.
this one is awesome
Don't come into this chat with your VS code vibes😂😂
This is crazzy. BECAUSE OF ONE SHOW
this was very entertaining 😂
I bet Netflix performance and ease of development would be improved with monolith. Lol
krazam's content is gold
I remember when I first got into black mirror it started playing s4e1 and I was like wait, why is it playing from the start of season 4, and I remember wondering why they did that. It's cool that I now know why that was. lol
12:25 - what I'm hearing is that it should be far easier to hack my way into watching temporarily blocked episodes/videos than I expected!
I still dont understand why they couldnt order the list on FE for when it is necessary , yes it is messy but so is the pipeline that is being shown for it.Would appreciate if anyone can help me understand
Same
Probably something along the lines of: Netflix is supported on a wide range of devices, some of which may be eol and not receiving updates. Solid chance trying to do this on the front end would result in patches for hundreds or thousands of different types of devices.
@@lvlinty No, but they did something with the hidden flag names, and even had to switch them, so they had to do updates.
Maybe the FE team didn't want to bloat the bundle size.
IMO, the FE doesn't look good in this picture, maybe it's their fault. The issue with the flag names hints at an obsession with procedure. "This is not the right way" kind of mentality.
@@DF-ss5ep Doesn't make sense, since most of the lolomo things are server-side rendered. Only things like interaction, streaming and loading additional content after first load is fetched on client. I'm sure they have a sort of BFF that could do the mapping.
the idealistic argument is "it's should be done in backend"
I was feeling every second of that video skit. It all was too relatable. About The PF episode, I watched the show from the first season. I love the show and listened to the explaination, but it sounds like several services and rewiring with a v2 for what was a single use case. If it is broadly used for more purposes now, it is fine, but if it is just for PF, I'd say some hardcoding and to put to bed those PFs nightmares and never forbid speaking of it again 🙊.
What a glorious story.
I just wanted to watch season 1 episode 1.
The fact that this somehow works is truly astonishing
The pain will always be present. Sure we'd love to have simple straightforward services that do exactly what we think they should, but we also want performant systems that will last a long time as it turns out product doesn't want the user to get hit with a 5 second query at request time so we build caches and indexes to solve that problem. Turns out you from 5 years ago did not possess a third eye that can see into the future and anticipate what product wants today so you don't have direct access to the data you want that is guaranteed to be up to date and valid for a transaction and now you have to make elaborate changes to accommodate that feature. Ultimately this is why any sufficiently long lived codebase starts to look like a hoarder's house as you build seemingly redundant features that meet varying use cases.
Couldn't a popup to start from S4E1 ("recommended") vs S1E1, at the frontend solved the problem?
if y'all only knew how the product page worked on your last amzn purchase.
Truly microservices hell
Pure genius to migrate the “Ooops, something went wrong.” microservice to the Netflix “Whoops, something went wrong.” microservice. On par with Under Construction animated GIF.
This was the closest reenactment of that It's Always Sunny In Philadelphia meme
I shared this with junior level developers a year ago... nobody felt my pain. 😭
Man I must be extremely dumb because I came up with the same question as in the end and I thought wait I need to re-watch, and now I feel like the interior of my skull is a gigantic ocean of nothing because I'm still wondering why you needed all those microservices to reverse a fucking list of videos
And that's why we are so slow, Omega Star don't support ISO timestamps like they said they will A MONTH AGO!
We are constantly blocked!
My story is the same but with old IBM ecommerce software.
I'm crying tears of laughter. Laughter and regret.
Using (and setting) a compositor like picom fixed my screen tearing issues!
It is so infuriating starting a new show, only to realize I was starting somewhere in the middle 😕.
Did i just watch a video and then a recreation of that video in the same video? xd
when i watched about neflix services - i wondered, why there are so many of them. Now it kinda makes sense
this engineering failure vulnerability here is giving me signlas prime
audibly saying "oh nooo" at every new service in the netflix story lol
lol, I now understand why i tried watching black mirror and it started from S06E01
13:54 “I told you, it’s the design of our back end!”
vscode vibes holy shit I'm crying
I am a RF hardware designer who sometimes codes too. If the system so inflexible to make it “impossible” to add feature and so difficult, isn’t that a sign of bad software design to begin with? I apologize in advance for potentially asking stupid question.
Absolutely
Have a look at Uncle Bob vs Casey Muratori. Bob (a Clean Code book selling guru) argues for these kinds of highly separated architecture so it's easier to change. Casey (a game dev) argues for just telling the computer what you want it to do, because either way it's not going to be easy to change and you might as well write less code and when you inevitably have to rewrite it later, have less code to rip out.
TLDR: all these architectures are supposed to bring flexibility benefits but they don't really
Yeah this is a documentary of our lives, huh?
I died when I saw this the first time. 😂
I remember watching black mirror and being very confused that the first episode wasnt S01E01
Highlight metadata on season list. Order by hightlight then season number.
Wow, S4. That's really putting the user first.
I still don't get why you couldn't get all seasons as a list of lists, and then move the fourth list (a fourth season) to the beginning of the list without reordering anything else. Feature flag it and make the front-end reorder them.
My theory is they wanted it to be done all in the backend because then it would work across every device Netflix is on without having to ship new versions for all of them.
You can use picom, I had the same screen tearing issue and then I used picom with i3 and it suppresed the tearing.
I currently am using picom
@@Ultr4noob He literally said in this video that he hasn't switched to Wayland yet, so no, he doesn't use Sway.
LOL! Screen tearing was on your side. I thought it's on my side. Nvidia drivers on my laptop goes crazy on linux. So I have tearing half of a time.
You should watch the “senior engineer” from krazam.
even today, when i start a new TV Series, I have to go to S1E1 manually, because Netflix cant detect that i have not watched previous episodes ??
we can all feel the pain in your laugh at 8:59
This feels like of each show was an objecr u could just edit the blackniror object
😂 the huge brush size
my brain went for a smoke and I found it hiding under the chair outside
Reverse season order is such garbage. When it first came out, it was easy to accidentally start watching from a later season rather than S01E01. And it's such horrible UX to scroll through the episodes and them not be in order. You want to promote a certain season? Sure, fine. But when I'm viewing the full episode list, show them in gogdarn order!
This is incredible
You can tear my monolith from my cold, dead hands
I'm surprised that you didn't have a way to weight items in the sorter.