Hey tech newsers, it's Jakob from the ad read. I hope your enjoyment of this video didn't distract you from learning that you can go to saily.com/techlinked and use the code TECHLINKED to get an exclusive 15% off your first purchase. No? Ok cool...what are you up to this weekend?
Working nights. Reading dune for the first time (I left my steam deck at home for this two week trip because my one month bio 1 class is kicking my butt so.. only VR fit games this month)
@techlinked The link says error 404 not found, on saily /techl.... but... may be temporary. And you still have Home button. Checked this from eastern EU :-P
My laptop was literally unaffected in every possible way. I was thinking that this was a pre-august fools joke until this expanded to other people talking about it.
"Let's give remote kernel level access to our critical systems to one proprietary non-transparent company as our security" - the entire world apparently 🤦🏻♂️
I was at tractor supply and the computer ringing me up blue screened, there were no employees nearby so I just hit restart then watched it happen again. Then I googled and realised the world was on fire lmaoo
But that's the best time to push an update. If everything goes tits up then the techies have the weekend to clean up the mess, and as we don't do business over the weekend my figures and therefore my bonus remain relatively unaffected.
Not even close to 90% - ClownStrike has about 20% share of the EDR market. This was only a fraction of what could have happened if all 'doze systems were affected, for instance
@@MapsaiLiv Much easier for you than for every company on Earth that is forced to be compatible with the status quo, whether it be to ensure that they can keep their hardware running, or to coordinate with other companies.
People often use “amen” out of its original religious contexts, simply as a word of agreement, it’s similar in vibe to “Yeah! I agree”.This usage is particularly common in the American southeast.
I don't necessarily disagree with those particular words, but relevance? I am slightly confused as to what you mean by "centralizing" in this context. A bunch of people bought the same piece of software, and the makers of that software had bad IT and QA practices, which caused the people who bought the software to have problems. What exactly is centralized about that scenario?
I flew across the US today. Singlehandedly the most dystopian thing I have ever seen. Hundreds of people sleeping in the airport all day, waiting for their flight that takes off in the next days. I am very grateful to be home only 6 hours late.
@@mycosys Have you heard of hyperbole? There are obviously more dystopian sights all around us. I would say it's the most dystopian thing I've seen in aviation.
I work in finance IT. I had customers asking if we were affected by the M365/Azure outage that happened in real time. Turns out its all Crowdstrike related as even Microsoft uses them. It scares me more what a bad software update can do because all the chaos can lead to bad actors getting in while the world is most vulnerable. It goes to show that Crowdstrike is too big for our own good and too many companies rely on them. This outage will cause many companies to change threat prevention companies. Will Crowdstrike survive this? Not sure, considering they caused the world to almost pause for a day.
@@mycosys I've seen news outlets like Sky News say up to 24% market share for Crowdstrike. The problem here was that they serve service providers as well so companies were indirectly effected by this or companies didn't know this was the issue because it was one of their vendors or partners etc that uses Crowdstrike that was the issue.
I think the problem is more so that Microsoft is so big, and they happen to use Crowdstrike. Half the world uses Azure somewhere in their processes, and M365 is used almost everywhere (although M365 doesn't usually affect critical processes, if it does, the infrastructure is quite janky). Due to the easiness to duplicate in IT, there isn't a lot of competition, and therefore surprisingly few companies own basically all the critical infrastructure.
I really never heard of Crowdstrike as a big figure in IT sec until around 3 weeks ago. By then I thought that using a falcon as symbol might render the world prey and prone to some misdeed on their part. Now...
I wish the headlines would put proper blame on Crowdstrike, almost all of tehm paint it as a "Microsoft/Windows did this" and not "Crowdstrike brought down Windows Machines through incompetence"
I work overnights in a Data Center in Downtown LA. I came in last night and everything was on fire. Even our NOC Laptops and Compute Sticks that run the TVs with the Network Monitoring were Bluescreening. Fun night 😅
Unfortunately most people don't realize how fundamentally unstable pretty much all modern tech is, and so they put way too much trust in it. People like to point and make fun of critical infrastructure in places still using computers from the 80's that use floppy disks, but quite frankly I'd trust those a lot more than I'd trust any modern computer with software that's feeling more and more like a Jenga tower.
@@Locked_outt"Glory" 🤣 it's his company bro, it's like saying Dave Thomas only wanted glory for being in all the old Wendy's commercials. Please grow up.
i was working at woolies during this IT outage, took out all our systems and the store was nearly shut down. Very interesting experience to say the least.
@@nekdo_kavc how TF would someone working in a store of a megacorp know why TF the CTO made the decisions they did? BTW all linux machines with CS got a kernel panic issue a few weeks earlier. Was CS not 'doze.
@@nekdo_kavc when you worked retail could you provide reasoning why people choose the operating system that was used? Further to that could you explain why the operating system mattered when it was *third party software* that failed? Literally nothing whatsoever preventing this happening on Linux from software from the same vendor.
Love how we have different choices to resolve the problem 1. You can get to the recovery screen, entire the BitLocker key, boot into safe mode, and delete the file 2. If that is too complicated, just reboot your computer 15 times.
Heh, now that MS has *forced* many people into changing their computers to EFI boot + GPT partition problems of this kind are suddenly much harder to fix.
Three times is the norm to get a Windows system that cannot boot normally into recovery/safe mode. No idea where the number 15 came from or how they got it.
So many questions... Why can you not just roll back your windows installation before the update by a click of a button?? How can a company working in security roll out an untested update like this??? Why are important companies rely so much on certain technologies, without a backup safe plan?
1. It kind of exists already but doesn't usually work, and this is a program problem not an OS problem anyway; 2. They're morons; 3. Tremendous expense.
From what the remediation instructions look like, Crowdstrike pushed a bad component or configuration update. Since it is a cloud-based endpoint protection (fancy AV) service, every machine with a subscription downloaded their update files for the day and promptly crashed
Yeah good luck getting in the safe mode since windows 8, somebody thought it was a good idea to get rid of the F8 menu to make it boot faster before the existence of SSDs.
Modern systems, especially with the advent of UEFI (Unified Extensible Firmware Interface) and SSDs, boot much faster than older BIOS-based systems. The F8 menu was designed for slower boot processes, and its removal allowed the boot sequence to be optimized for speed since input delay was no longer needed.
If it was intended to prevent an exploit discovered on Thursday, it makes sense. Afterall, many, if not most, of their clients do not shut down for the weekend.
In sure if you opt in to their pushed update feature there’s an asterisk in the contract that says that you take on some risk. Also, a global lawsuit would just cause them to file bankruptcy, so any liability coverage is mute
Windows update policy is too aggressive and guarantees to corrupt billions of PCs simultaneously- all over again - in case of a different update going wrong
Apparently you arent one of the linux users with empathy. Been admining 'nix since systemV and a still feel for my brothers and sisters in h3ll today. They dont get to make OS or platform decisions. & Same thing happened to CS' linux users a few weeks earlier too.
@@gagagero I heard a bunch of Crowdstrike-managed Linux servers went down, but even if no Linux systems were downed it's only because it would have been a Windows-specific file that bit the dust At the end of the day you had to have a Crowdstrike contract active and have their software actively managing a machine to see the effect They shipped an empty file in the place of an AV scan binary, nothing you can do to save that on any OS I hope Linux is more popular in the future so we can go "Wasn't Linux, this was trusting one company to run the majority of the IT safety on the planet"
@@gagagero This particular instance did not affect Linux. But there was a situation last year when updating Debian and Crowdstrike out of order corrupted the whole system requiring a wipe and reinstall. This is poorly tested software running at the driver level, not a Windows problem. A buggy Nvidia driver could cause the same problem - and often does on Linux.
I cant overemphasize how much this wasnt a small mistake from a software management perspective: - How, literally how, was this not discovered by Quality Assurance? Are you telling me, CrowdStrike doesnt test installing software with Kernel-level access? - Why the hell was this rolled out to everyone at once? Are you telling me CrowdStrike doesnt do rolling releases / canary releases for software with Kernel-level access? - How did so many companies do the same thing internally with the update they got pushed? Are you telling me, all these companies are just blindly installing updates on all their devices without checking whether it breaks things. Edit: as pointed out below, the point of crowdstrike is to automatically get security updates to get ahead of zerodays etc. and tbh I get that... Seriously, this isnt a "dont push to prod" issue, this is CrowdStrike and their customer-base not adhering to incredibly fundamental software release processes. I will go out on a limb here and suggest that these are the predictable results of cost-saving measures.
The whole point in crowdstrike is their very fast updates, new zero day? Boom crowdstrike has it covered and your system is already protected before you even know about it. Which in theory is great, unless... Well unless this
@@Lonewolf_121 But can't you take a few minutes to install it on a machine, check it doesn't crash, reboot the machine, checks it works and is stable for a couple mins again. Not a huge delay.
Not really sure how that mantra applies at all in this case. The failure here was caused by security software, not some singular critical infrastructure failing. Are you implying that companies should be somehow distributing random security software to different machines in their network?
@@LimitedWard What I mean is, first Crowstrike should have done more testing and did the update in stages and not to everyone at once and two, companies should have had backup systems in place so that if something took down their main systems like what Crowdstike did, they can switch over to their backup systems. Point is that Crowdstrike was a single point of failure. I'm not saying that companies should be using multiple security softwares in their network but that companies should have contingencies in place to minimize any downtime caused by a piece of software or hardware taking their main systems offline. What happened with Croudstrike highlights how vulnerable we are. When I say single point of failure I'm referring to both hardware and software.
russia wasn't striked cause our large companies, airlines, etc had to switch to linux astra in last 2 years i don't count it as any win, but it is cool to know there are some benefites from de_globalization
Rolling out updates instantly is Crowdstrike's main selling point. For zero day exploits, they leverage this to instantly patch computers before the vulnerability gets a chance to spread across network. In this case, it turned out to be their demise.
Not everything rolls out in waves. It really shouldn't have to either. Maybe even better, considering their line of business. There were two versions of the same sys file distributed in the update. One of them was corrupted while the other wasn't. The corrupted one contained pointers with null addresses which caused the crash. The real kicker is that there is no rollback capability built in, yet the program enforces itself to be a requisite for OS boot. That's the real problem.
@@paulelderson934tbf most people would probably desire that behaviour in any case other than this. Given those failures would generally be the result of a bad actor. People would rather a boot loop than having some malware run successfully (generally). We both know if you gave users the option to "override and roll back" they would press it in a second to let the virus run.
@@paulelderson934 I understand the context, but I don't believe anyone can build a rollback capability before your OS even boots up. It's like the "Chicken and Egg" puzzle.
This has been a terrible weekend. Our IT teams have been directed to take calls for internal employees, while the more experienced service desk staff are calling out on backlogs of internal employees issue tickets. Our environment has 10K employees globally so it’s been a nightmare of support this entire weekend. For the first time we had mandated weekend work on Saturday and Sunday
Just because something runs public code, doesnt mean it would prevent a bad update being pushed while everyone was asleep and waking up to find there was a large problem. If the code was published a week before it was rolled out maybe it would of been spotted. But you could argue the same thing had they tested this update before rolling it out globally.
@@robinbegley1077 While it does not prevent a bad rollout but lowers the chance significantly by multiple factors. And also keep in mind that multiple authorities (public infra -> has to be done) can review prior rollout and not only some shitty company makes the decision for millions of ppl with underpaid engineers. while everyone was asleep? Check the globe, its not flat not everyone sleeps at the same time.
@@MrBoboka12 You do know that security companies that have gov contracts have to get source code safety audited (they have to show their source code under a non disclosure agreement with parties involved)
Considering they just banned Kaspersky as one of the businesses solution for security software. It's kinda frightening to know that this kind of things can definitely happened and the security solutions is now having fewer choices.
Unfortunately, no, not unless this outage literally threatens to send us back to the dark ages. It’s gotta last for weeks if not months, something on the level of SKYNET becoming self aware. Some of these affected PCs were fixed the very same day, so while some people may switch to Linux just on the bad taste left in their mouth over this, there won’t be a massive night and day type of transition.
I work at a deparment store and our emergency alert system was activated over this. The same system we'd use if the store was on fire. Every employee got a text that all our computers were down. This included our POS systems, which are all windows machines. We have some backend functions that can run entirely off Android handhelds, assuming the servers aren't Windows at least.
It's all fine and silly to say "I'm a linux user so this doesn't bother me", but fact is, it does bother us, because chances are, if you gotta go to the hospital, their computers are affected. If you need to get somewhere in a hurry, the airport is affected. ONE Faulty driver is all it takes to bring down all of these services. Hackers need not apply when captain butterfingers is at the helm.
This isn't a one-person problem. The fact that this release was able to get through engineering checks at all is a failure of Cloudstrife itself. I'd bet money that a slough of budget cuts and layoffs preceded this update.
To my untrained eyes, it looks like "BEVERLY HILLS TRIAL ATTORNEYS, P.C." saw a bunch of people with more money than sense and decided it was a cash cow worth milking by charging them some kind of fee. They actually included in the lawsuit that they wanted to apply an article so that the defense would have to pay for the "reasonable attorney's fees" of the plaintiffs, should they lose. So it seems like the law firm is getting money no matter what, even if it's a lot more if they win :D They may have some point but overall this looks like a straight lose for the plaintiffs, their most valid point is probably that the car cannot actually be fueled in 5 minutes as advertised because the fueling system is just not good. One of the points of the trial was also "should the plaintiffs have known the claims were obviously false", so the fuel thing is insanely shaky. Their ads currently say "$15000 or 3 years" and then in fine print "whichever comes first", unless they had some sleazy sales people (which I can definitely believe), I doubt they were every actually promised the stated 5 years.
how is that even possible that a software update was pushed in production everywhere at the same time? Are all this companies just put their computer on auto update or something???
@@LadyRenira I can understand and trust Windows auto update, as MS already handle such thing by not pushing it everywhere at once for example. But a piece of software running on this machin, even if its a security one, how the hell does it was not tested properly in preproduction? I mean, it does not look like a very hard to find bug I really dont get not having a proper preproduction environment, even with a short delay, especially for critical one
Even if you had windows it wouldn’t have affected you since Crowdstrike is a B2B security company, therefore it’s only installed in… you know… computers critical to infrastructure like airlines and banks and stuff, no big deal
Thank you Crowdstrike, I had to drive ten hours to get home yesterday. Thank you Crowdstrike, I didn’t have to pay for the rental car because their system voided my rental agreement after I left the lot. I will take the wins when I can get them.
The BSOD didn't require a reboot. Since the file that broke it wasn't the driver itself, it BSOD'd the minute the update finished downloading when it was dynamically loaded in the running system.
I've heard people mention that a similar issue affected Debian running CrowdStrike a couple of months ago, it just didn't make the news because it wasn't that widespread... Or the people who said it made it up, idk, too lazy to check. Either way, this can theoretically affect Linux as well.
@@the-answer-is-42yes, the real solution is not to install kernel rootkits, no matter what OS you are on. That being said, installing CrowdStrike on Linux is like installing anti-virus on Linux: you can do it, but why would you? Which is the reason no one has heard of the Debian crash (not even the CrowdStrike people, because it didn't happen) or the actual RedHat 9 CrowdStrike crash (that did happen last month) - because while Linux is the most used operating system on servers - almost no one is using CrowdStrike on it, only IT that also manages CrowdStrike on Windows and doesn't know any better.
It's wild how critical infrastructure can be rendered useless by something like this. Really goes to show how dependent we are on digital technology to function in today's world.
I have no idea why people are claiming this is Microsoft's fault - for once, this has nothing to do with them. Heck, the faulty software is also available for Linux systems.
I'm assuming people are confusing the Azure outages and the fact that (I think?) the update is distributed via Windows Update. But yeah, unless CS was relying on some sort of file audit from MS to detect corrupted files in the update, they probably had nothing to do with the actual issue other than effectively being the messenger carrying a faulty message.
@@icantcomeupwithnames469 It's their responsibility to vet the companies they work with, and maybe to check update files for malicious content, but testing is on CrowdStrike.
Yeah, and why not? MS has been running businesses for decades with relatively few issues and this isn't even Microsoft's fault! Are you going to tell me that Linux/UNIX systems have NEVER had issues?
"Hotter threads than TJ Max" That is a really good one. I'm a Linux user and no, I don't feel empathy for Windows users! I'm enough of a masochist already.
Eeh, I feel bad for the people who didnt get the surgery they waited months for... CrowdStrike isnt really installed on personal-use devices, so theres really no individual at fault here, more the failure of an entire industry deciding to create a perfect break-point by buying into quasi-monopolies like Windows and CrowdStrike...
Hey tech newsers, it's Jakob from the ad read. I hope your enjoyment of this video didn't distract you from learning that you can go to saily.com/techlinked and use the code TECHLINKED to get an exclusive 15% off your first purchase. No? Ok cool...what are you up to this weekend?
Working nights. Reading dune for the first time (I left my steam deck at home for this two week trip because my one month bio 1 class is kicking my butt so.. only VR fit games this month)
Just got a floatplane subscription. Binging the exclusives for the next few days.
"you're enjoyment"... nice :V
@techlinked The link says error 404 not found, on saily /techl.... but... may be temporary. And you still have Home button. Checked this from eastern EU :-P
Playing Helldives, Siege, switching work vans, cleaning my apartment, etc... I should probably start cleaning
Crowdstrike: "Don't worry, wasn't a Cyber Attack, just incompetence, you can still trust us with your security"... Great, where do I sign up?
Hanlon's Razor basically.
Wrong company name Bro, it's Clownstrike.
Ahem, I mean Crowdstrike.
cloudstrike probably thought that no hacker could block their customers computers if they block them first
@@rafaelroma1657Crowdstrike really striking the entire crowd with this outage
@@rafaelroma1657 Clownstrike sounds a like a wacky military supplier. Crowdstrike sounds a terrorist supplier.
Seeing Linus doing TechLinked is like hearing "We interrupt this broadcast"
Sh*t got so real they had to bring back the founder
My laptop was literally unaffected in every possible way. I was thinking that this was a pre-august fools joke until this expanded to other people talking
about it.
@@user-zd2xj9rl2zI think the only ones affected are business laptops
@@user-zd2xj9rl2z only enterprises were affected, its an enterprise EDR package.
@@user-zd2xj9rl2zno shit, most computers weren’t affected, this was for enterprise PCs using crowdstrike’s services, did you not watch the video?
Cybercriminals: *attempt to hijack and take down computers around the world*
CrowdStrike: hold my virtual beer
Cybercriminals as half the biggest companies start disabling their security to get back online: $~$
"Hold my security update"
When the cure is worse than the illness!
truly, hold my update
They passed the just hold my beer level... they have achieved "hold my baby" status 😮😅😂
Crowdstrike: Global Offensive
Crowd Strike: Global Outage
@@VTCuongDevcan’t wait for Crowdstrike 2
😂😂
@@VTCuongDev lol good one😂😂
CSGO irl 😂😂
"Let's give remote kernel level access to our critical systems to one proprietary non-transparent company as our security"
- the entire world apparently 🤦🏻♂️
Don't forget the part about potentially needing said company to get physical permission to fix god only knows how many computers.
Equifax moment
Combined with windows aggressive update policy which guarantees to take out billions of systems AT ONCE
Doesnt need that company to do the fix @gorkskoal9315
@@addydiesel6627 huh?? it wasn't windows fault bruv..
I was at tractor supply and the computer ringing me up blue screened, there were no employees nearby so I just hit restart then watched it happen again. Then I googled and realised the world was on fire lmaoo
That’s awesome! 🤣🤣🤣
it really isn’t lmaoo
And then you walked out with the goods?
looks like y2k happened a few years late
My worlds on fire how about yours. That's the way we like it and we never get bored.
Don't push code to production on Fridays, RIP techies needing to manually fix each and every computer.
But that's the best time to push an update. If everything goes tits up then the techies have the weekend to clean up the mess, and as we don't do business over the weekend my figures and therefore my bonus remain relatively unaffected.
It was out on Thursday, but update propagation takes its time. They should hold for Mondays, but maybe it was critical?
@@romapires happened during midnight or so. Maybe super tired or deadline.
It is actually better to it friday
unless they used netboot :)
This is what happens when corporations become too large and control 90% of their markets.
And this is why I (sort of) disconnected from corporations, so these things can't happen to me. And it's very successful so far.
All the gas stations around me couldn't take cards last night
Not even close to 90% - ClownStrike has about 20% share of the EDR market.
This was only a fraction of what could have happened if all 'doze systems were affected, for instance
@@MapsaiLiv Much easier for you than for every company on Earth that is forced to be compatible with the status quo, whether it be to ensure that they can keep their hardware running, or to coordinate with other companies.
@@MapsaiLivcongrats. Most of us have to deal with a massive corporation tho.
Crowdstrike sure lived up to their name.
How ironic, no?
Like naming company ocean gate
@@reviewchan9806good point nearly forgot about the crackling graphite sub
Malware hackers are jealous of Crowdstrike’s official channels.
Watch the new hacker meta be to create legitimate cybersecurity apps, only to come out with a malicious update years down the line.
@@leonro They'd never go to all that trouble when they can just hack an existing one like SolarWinds.
This is practically what people in the year 1999 thought Y2K was
telling people not to worry just cuz it wasn't a cyber attack is peak corporate bros undermine
That's 100% legit though. This is bad, but if it were an attack, it would *much* worse.
@@hellterminator unless it was an attack and they are just saying this to save face not incite panic and other reasons
Who let Linus replace Riley =(
Seriously, Riley was ours... Bring him back >:(
who’s linus
Riley has been on vacation. He'll be back next week!
He is a robot
Riley is an AI powered by Microsoft and CrowdStrike. It’s no coincidence that this video is when he’s missing
Centralizing everything is convenient until something goes wrong
Amen. No, literally. This gets forgotten like *so much* and often...
@@IngwiePhoenix why did you say "amen"?
Decentralized all the way
People often use “amen” out of its original religious contexts, simply as a word of agreement, it’s similar in vibe to “Yeah! I agree”.This usage is particularly common in the American southeast.
I don't necessarily disagree with those particular words, but relevance? I am slightly confused as to what you mean by "centralizing" in this context. A bunch of people bought the same piece of software, and the makers of that software had bad IT and QA practices, which caused the people who bought the software to have problems. What exactly is centralized about that scenario?
I flew across the US today. Singlehandedly the most dystopian thing I have ever seen. Hundreds of people sleeping in the airport all day, waiting for their flight that takes off in the next days. I am very grateful to be home only 6 hours late.
If thats the most dystopian thing you have seen, you need to look outside your bubble little more.
@@mycosys Have you heard of hyperbole?
There are obviously more dystopian sights all around us. I would say it's the most dystopian thing I've seen in aviation.
@@strangegamercTwo Boeing whistleblowers died under suspicious circumstances.
@@kyle_mk17, I don't think OP has witnessed them in any way.
Honestly, IT issues in airports grounding flights is a surprisingly common occurrence
I died at "Linux users feeling empathy for Windows users." 😂
Right! Like they'd ever do that!
I was really hoping someone would enjoy that joke ❤
@@JakobRush as someone who has been using 'nix since systemV and 'doze since 2.0, i sure did. We need more empathy.
@@mrgw98a lot of Linux users are Windows expats. We know Windows users pain but decided to stop suffering.
@@sergeykish I still have to use Windows for work on occasion, and it is excruciating every time.
I work in finance IT. I had customers asking if we were affected by the M365/Azure outage that happened in real time. Turns out its all Crowdstrike related as even Microsoft uses them. It scares me more what a bad software update can do because all the chaos can lead to bad actors getting in while the world is most vulnerable. It goes to show that Crowdstrike is too big for our own good and too many companies rely on them. This outage will cause many companies to change threat prevention companies. Will Crowdstrike survive this? Not sure, considering they caused the world to almost pause for a day.
20% market share.......... did all this
@@mycosys I've seen news outlets like Sky News say up to 24% market share for Crowdstrike. The problem here was that they serve service providers as well so companies were indirectly effected by this or companies didn't know this was the issue because it was one of their vendors or partners etc that uses Crowdstrike that was the issue.
I think the problem is more so that Microsoft is so big, and they happen to use Crowdstrike. Half the world uses Azure somewhere in their processes, and M365 is used almost everywhere (although M365 doesn't usually affect critical processes, if it does, the infrastructure is quite janky). Due to the easiness to duplicate in IT, there isn't a lot of competition, and therefore surprisingly few companies own basically all the critical infrastructure.
@@matthewjbauer1990 sorry but if you watch sky news i cant trust a thing you believe.
I really never heard of Crowdstrike as a big figure in IT sec until around 3 weeks ago.
By then I thought that using a falcon as symbol might render the world prey and prone to some misdeed on their part.
Now...
9 years ago I made an oopsie that resulted in 800 people twiddling their thumbs for a whole morning. I still dream about that.
I wish the headlines would put proper blame on Crowdstrike, almost all of tehm paint it as a "Microsoft/Windows did this" and not "Crowdstrike brought down Windows Machines through incompetence"
I work overnights in a Data Center in Downtown LA. I came in last night and everything was on fire. Even our NOC Laptops and Compute Sticks that run the TVs with the Network Monitoring were Bluescreening.
Fun night 😅
Good golly Christ, why are those running Windows?!!
@crash.override Some Clients want Windows :(
It's crazy how advanced technology is and how much we rely on it... one small mistake and things go down in flames.
I mean, the whole world of software is also less stable; progress... or rather, the degradation of progress.
@@Atsumari so what is the focus then you would like to see?
Whats really crazy is theres not a single person on earth who knows how the whole network operates.
Or even the entire code on one windows computer.
A software tester walks into a bar...
Unfortunately most people don't realize how fundamentally unstable pretty much all modern tech is, and so they put way too much trust in it. People like to point and make fun of critical infrastructure in places still using computers from the 80's that use floppy disks, but quite frankly I'd trust those a lot more than I'd trust any modern computer with software that's feeling more and more like a Jenga tower.
Holy crap, I forgot Linus used to do techlinked. I don't think we've seen him here in nearly 19 months?
He did some GameLinkeds in the meantime
@@Locked_outtno Riley is just on vacation, unlucky timing
@@Locked_outt it's his fucking company and channel!! Jeez!!
@@Locked_outt And that's a bad thing? If you're goiing to 'report' on a very important subject, you'd want your A-team to do it.
@@Locked_outt"Glory" 🤣 it's his company bro, it's like saying Dave Thomas only wanted glory for being in all the old Wendy's commercials. Please grow up.
i was working at woolies during this IT outage, took out all our systems and the store was nearly shut down. Very interesting experience to say the least.
Can you then provide reasoning for using Windows in the first place?
@@nekdo_kavc Corporate HQ. Its nearly always Corporate HQ
@@nekdo_kavc how TF would someone working in a store of a megacorp know why TF the CTO made the decisions they did?
BTW all linux machines with CS got a kernel panic issue a few weeks earlier. Was CS not 'doze.
Woolies? You're gonna need to specify if that's Woolworths UK, South Africa or Australia 😜
@@nekdo_kavc when you worked retail could you provide reasoning why people choose the operating system that was used? Further to that could you explain why the operating system mattered when it was *third party software* that failed? Literally nothing whatsoever preventing this happening on Linux from software from the same vendor.
One the my favorite comments I've read was "This is what happens when critical software isn't treated as critical."
“Cyber hackers and business execs HATE this 1 trick!” - Ad on a CrowdStrike dev’s laptop a day before pushing the bugged code
I gave up rebooting after 10 or 12 times. This made me laugh that Microsoft said to reboot it 15 times. I was so close....
I seriously have no idea where people are getting the 'interrupt booting process 15 times to get into safe/recovery mode' information. ㄟ( ▔, ▔ )ㄏ
Love how we have different choices to resolve the problem
1. You can get to the recovery screen, entire the BitLocker key, boot into safe mode, and delete the file
2. If that is too complicated, just reboot your computer 15 times.
Heh, now that MS has *forced* many people into changing their computers to EFI boot + GPT partition problems of this kind are suddenly much harder to fix.
Why 15 times? Could Bill 'Crowdfix' Gate make it more complicated? By design Windows is made to make people give up and call support
@@addydiesel6627 More like 100 times, too bad F8 doesn't work :(
Three times is the norm to get a Windows system that cannot boot normally into recovery/safe mode.
No idea where the number 15 came from or how they got it.
@@Carcinogenic2 'security is bad, mmmkay' oof
So many questions... Why can you not just roll back your windows installation before the update by a click of a button??
How can a company working in security roll out an untested update like this???
Why are important companies rely so much on certain technologies, without a backup safe plan?
1. It kind of exists already but doesn't usually work, and this is a program problem not an OS problem anyway; 2. They're morons; 3. Tremendous expense.
Also the mindset of "this thing exists, may as well just use that thing we know works, they'll handle everything; convenient for us"
From what the remediation instructions look like, Crowdstrike pushed a bad component or configuration update. Since it is a cloud-based endpoint protection (fancy AV) service, every machine with a subscription downloaded their update files for the day and promptly crashed
Those dudes had already multiple times f*d up non-windows stuff but it was not that widely publicized
Sorry, have yet to experience any fuck ups. Must be a statistical thing.
They really did *STRIKE* the *CROWD*
*Struck
@@luimu WRONG! You can only have one past tense verb in a sentence.
@@_GhostMiner oh yeah you are correct, completely read over the "did". Without it I would've been right and the meaning would be the same.
That was a fun day at work! So many pissed off customers 😅
Yeah good luck getting in the safe mode since windows 8, somebody thought it was a good idea to get rid of the F8 menu to make it boot faster before the existence of SSDs.
Modern systems, especially with the advent of UEFI (Unified Extensible Firmware Interface) and SSDs, boot much faster than older BIOS-based systems. The F8 menu was designed for slower boot processes, and its removal allowed the boot sequence to be optimized for speed since input delay was no longer needed.
We had a server outage for an hour till the backup kicked into action, dont even know if its related because everything else worked fine
"Hotter [cpu] threads than the real TJ Maxx" is a certified bar. Take my upvote.
105% agreed
Crowdstrike is cooked by the biggest blue screen in the entire history of humankind, suffered by only one bug, and Microsoft joined the fray
Bri their rich, unlikely to go away but maybe overtime will waver off.
Mans really pulled out the Ideal Gas law joke
Kinda hilarious that Linus didn't get it. His T must be too high.
".. rolled out an update .."
that's the problem, they didn't roll it out, they pushed it to EVERYONE at once
Crowdstrike: "Hey, when Cloud Strife took down just ONE evil corporation, everyone called him a hero!"
Pushing to production on a Friday? What could go wrong?!
If it was intended to prevent an exploit discovered on Thursday, it makes sense. Afterall, many, if not most, of their clients do not shut down for the weekend.
@@rightwingsafetysquad9872 Depends on how important the discovery was, how long it's been in there and imminent risk of attack tbh.
crazy how a single "security" company messed up everything across the world.
I wonder if the 911 system being down killed or injured anyone. If so, would Crowdstrike be held accountable?
In sure if you opt in to their pushed update feature there’s an asterisk in the contract that says that you take on some risk. Also, a global lawsuit would just cause them to file bankruptcy, so any liability coverage is mute
Windows update policy is too aggressive and guarantees to corrupt billions of PCs simultaneously- all over again - in case of a different update going wrong
My workplace that uses Linux: 😂
Apparently you arent one of the linux users with empathy. Been admining 'nix since systemV and a still feel for my brothers and sisters in h3ll today. They dont get to make OS or platform decisions.
& Same thing happened to CS' linux users a few weeks earlier too.
Crwdstrike also do endpoint management for Linux sooo this ain't it penguin bro
@@ThePlayerOfGamesAnd yet, it's unaffected.
@@gagagero I heard a bunch of Crowdstrike-managed Linux servers went down, but even if no Linux systems were downed it's only because it would have been a Windows-specific file that bit the dust
At the end of the day you had to have a Crowdstrike contract active and have their software actively managing a machine to see the effect
They shipped an empty file in the place of an AV scan binary, nothing you can do to save that on any OS
I hope Linux is more popular in the future so we can go "Wasn't Linux, this was trusting one company to run the majority of the IT safety on the planet"
@@gagagero This particular instance did not affect Linux. But there was a situation last year when updating Debian and Crowdstrike out of order corrupted the whole system requiring a wipe and reinstall.
This is poorly tested software running at the driver level, not a Windows problem. A buggy Nvidia driver could cause the same problem - and often does on Linux.
There’s no coming back from this one for crowdstrike😂
Y2K hit 24 years later.
so this is why my workers keep complaining that their pcs wont leave boot lol
I cant overemphasize how much this wasnt a small mistake from a software management perspective:
- How, literally how, was this not discovered by Quality Assurance? Are you telling me, CrowdStrike doesnt test installing software with Kernel-level access?
- Why the hell was this rolled out to everyone at once? Are you telling me CrowdStrike doesnt do rolling releases / canary releases for software with Kernel-level access?
- How did so many companies do the same thing internally with the update they got pushed? Are you telling me, all these companies are just blindly installing updates on all their devices without checking whether it breaks things. Edit: as pointed out below, the point of crowdstrike is to automatically get security updates to get ahead of zerodays etc. and tbh I get that...
Seriously, this isnt a "dont push to prod" issue, this is CrowdStrike and their customer-base not adhering to incredibly fundamental software release processes. I will go out on a limb here and suggest that these are the predictable results of cost-saving measures.
CrowdStrike is one of those special pieces of software that silently updates. I'm not sure why companies use it.
The whole point in crowdstrike is their very fast updates, new zero day? Boom crowdstrike has it covered and your system is already protected before you even know about it. Which in theory is great, unless... Well unless this
@@Lonewolf_121 thats actually a fair point... hm, okay, gotta reconsider the customers role in this!
@@Lonewolf_121 But can't you take a few minutes to install it on a machine, check it doesn't crash, reboot the machine, checks it works and is stable for a couple mins again. Not a huge delay.
Heck you Linus, I actually nerd squealed a little over 'Get PVnRT'd' 🤣
You should dedicate that nerd squeal to Adam Sondergard lol he wrote that
A warm happy feeling filled my chest at that joke yet it got so little love in the comments! Thanks Adam!!
There's a reason why crowdstrike is called crowd strike
I work customer service and it has been a day
💀my sister and I were next in line when this happened ☕️
They comped our beverages for bearing with them 😊
Yeah I got to say as an IT professional, let's not do this day again. Please, please.... Anyone.... I got called in off of PTO 😂😵💫
BAHAHAHA.I still have PTSD from working at compcast, and the fucking updates that were delayed to fridays was "fun".
These companies seem to keep breaking a fundamental rule of online services: "Don't have a single point of failure"
Not really sure how that mantra applies at all in this case. The failure here was caused by security software, not some singular critical infrastructure failing. Are you implying that companies should be somehow distributing random security software to different machines in their network?
@@LimitedWard What I mean is, first Crowstrike should have done more testing and did the update in stages and not to everyone at once and two, companies should have had backup systems in place so that if something took down their main systems like what Crowdstike did, they can switch over to their backup systems. Point is that Crowdstrike was a single point of failure. I'm not saying that companies should be using multiple security softwares in their network but that companies should have contingencies in place to minimize any downtime caused by a piece of software or hardware taking their main systems offline. What happened with Croudstrike highlights how vulnerable we are. When I say single point of failure I'm referring to both hardware and software.
Russia got this too. As far as I know, a local chocolate MARS factory suffered from outage as well.
It's a lot better than hospitals and airports
one more reason to never ever buy anything from nestle. those abhorrent people still operate inside russia.
Well MARS is American
NOT THE MARS BARS!?!
russia wasn't striked cause our large companies, airlines, etc had to switch to linux astra in last 2 years
i don't count it as any win, but it is cool to know there are some benefites from de_globalization
as a physics dropout i was not expecting an ideal gas law reference
"this level of chaos was caused without any malicious actors whatsoever" - LOL 🤣🤣🤣
Oh man, as a native San Diegan, that "La Jolla" pronunciation was a direct hit psychic attack 😭
“What do you mean we can’t do swimming at “Children’s pool””
“It’s full of what?”
La "HOY-a"
I don't blame him, you can never predict the local pronunciation of spanish town names in california. Vallejo uses an english L and a spanish J.
How tf did they not roll out in waves? Wtf is wrong with these people? I never heard of that company before.
Rolling out updates instantly is Crowdstrike's main selling point. For zero day exploits, they leverage this to instantly patch computers before the vulnerability gets a chance to spread across network. In this case, it turned out to be their demise.
Not everything rolls out in waves. It really shouldn't have to either. Maybe even better, considering their line of business.
There were two versions of the same sys file distributed in the update. One of them was corrupted while the other wasn't. The corrupted one contained pointers with null addresses which caused the crash.
The real kicker is that there is no rollback capability built in, yet the program enforces itself to be a requisite for OS boot. That's the real problem.
@@paulelderson934tbf most people would probably desire that behaviour in any case other than this. Given those failures would generally be the result of a bad actor.
People would rather a boot loop than having some malware run successfully (generally).
We both know if you gave users the option to "override and roll back" they would press it in a second to let the virus run.
The entire world runs on software and support companies you've never heard of.
@@paulelderson934 I understand the context, but I don't believe anyone can build a rollback capability before your OS even boots up. It's like the "Chicken and Egg" puzzle.
Had my flight cancelled yesterday, had to drive 8hr to reach my destination on time 😢thx crowdstrike
I am waiting for a lawsuit
This has been a terrible weekend. Our IT teams have been directed to take calls for internal employees, while the more experienced service desk staff are calling out on backlogs of internal employees issue tickets. Our environment has 10K employees globally so it’s been a nightmare of support this entire weekend. For the first time we had mandated weekend work on Saturday and Sunday
Public Infrastructure = Public Code. Period.
Public Infrastructure = whatever is fault tolerant and reliable code
@@seansingh4421 NO. This is supposed to be fault tolerant. No one can verify their shitty code.
Just because something runs public code, doesnt mean it would prevent a bad update being pushed while everyone was asleep and waking up to find there was a large problem. If the code was published a week before it was rolled out maybe it would of been spotted. But you could argue the same thing had they tested this update before rolling it out globally.
@@robinbegley1077 While it does not prevent a bad rollout but lowers the chance significantly by multiple factors. And also keep in mind that multiple authorities (public infra -> has to be done) can review prior rollout and not only some shitty company makes the decision for millions of ppl with underpaid engineers.
while everyone was asleep? Check the globe, its not flat not everyone sleeps at the same time.
@@MrBoboka12 You do know that security companies that have gov contracts have to get source code safety audited (they have to show their source code under a non disclosure agreement with parties involved)
Considering they just banned Kaspersky as one of the businesses solution for security software. It's kinda frightening to know that this kind of things can definitely happened and the security solutions is now having fewer choices.
Considering Kaspersky may be controlled by Moscow - one less security threat
I know I've been huffing the copium for years, but I really hope this outage makes Linux go mainstream
Or at least learn that they should turn off auto updates on production.
Nope. It will not
linux used to be mainstream in 2000-ish in some countries offices. i don't really know why they aren't now
Unfortunately, no, not unless this outage literally threatens to send us back to the dark ages. It’s gotta last for weeks if not months, something on the level of SKYNET becoming self aware. Some of these affected PCs were fixed the very same day, so while some people may switch to Linux just on the bad taste left in their mouth over this, there won’t be a massive night and day type of transition.
@@IslamistSocialist371prob because linux is free but then yeah even more reason why they should use it
MacOS was not affected either, nor were PCs fortunate enough not to have crowdstrike installed.
Linus: Everything is fine.
Linus says everything fine!
Everyone panic!
It is a continual miracle that anything works And I say this in the most positive way possible
Just imagine the post mortem meeting 😢
I work at a deparment store and our emergency alert system was activated over this. The same system we'd use if the store was on fire. Every employee got a text that all our computers were down. This included our POS systems, which are all windows machines.
We have some backend functions that can run entirely off Android handhelds, assuming the servers aren't Windows at least.
Crowdstrike hired Colton?
It's all fine and silly to say "I'm a linux user so this doesn't bother me", but fact is, it does bother us, because chances are, if you gotta go to the hospital, their computers are affected. If you need to get somewhere in a hurry, the airport is affected. ONE Faulty driver is all it takes to bring down all of these services. Hackers need not apply when captain butterfingers is at the helm.
This isn't a one-person problem. The fact that this release was able to get through engineering checks at all is a failure of Cloudstrife itself. I'd bet money that a slough of budget cuts and layoffs preceded this update.
Is it toyota's job to tell the customers where to get the fuel and what they cost?
To my untrained eyes, it looks like "BEVERLY HILLS TRIAL ATTORNEYS, P.C." saw a bunch of people with more money than sense and decided it was a cash cow worth milking by charging them some kind of fee. They actually included in the lawsuit that they wanted to apply an article so that the defense would have to pay for the "reasonable attorney's fees" of the plaintiffs, should they lose. So it seems like the law firm is getting money no matter what, even if it's a lot more if they win :D
They may have some point but overall this looks like a straight lose for the plaintiffs, their most valid point is probably that the car cannot actually be fueled in 5 minutes as advertised because the fueling system is just not good.
One of the points of the trial was also "should the plaintiffs have known the claims were obviously false", so the fuel thing is insanely shaky. Their ads currently say "$15000 or 3 years" and then in fine print "whichever comes first", unless they had some sleazy sales people (which I can definitely believe), I doubt they were every actually promised the stated 5 years.
how is that even possible that a software update was pushed in production everywhere at the same time?
Are all this companies just put their computer on auto update or something???
After the hack fiasco a while back that hit non-updated Windows machines (many in hospitals), yeah, pretty much.
@@LadyRenira I can understand and trust Windows auto update, as MS already handle such thing by not pushing it everywhere at once for example.
But a piece of software running on this machin, even if its a security one, how the hell does it was not tested properly in preproduction?
I mean, it does not look like a very hard to find bug
I really dont get not having a proper preproduction environment, even with a short delay, especially for critical one
Crowdstrike has kernel access, the update rolls out without the need for user interaction.
Its Best to Test New Code in Production 🙂🤷♂️
I, as a Linux user, didn't even realize that happened until I saw a YT short about it.
Even if you had windows it wouldn’t have affected you since Crowdstrike is a B2B security company, therefore it’s only installed in… you know… computers critical to infrastructure like airlines and banks and stuff, no big deal
Its been confessed to be a PEBKAC and ID-10-T errors
as a linux user i only feel mepathy for the hospitals
and laugh my ass off with the rest
Laugh as hard as you want, your OS will still be an also-ran in the desktop world.
Thank you Crowdstrike, I had to drive ten hours to get home yesterday. Thank you Crowdstrike, I didn’t have to pay for the rental car because their system voided my rental agreement after I left the lot. I will take the wins when I can get them.
Crowdstrike: Lets outsource to the lowest bidder and not test the code and force a reboot what could go wrong??
The BSOD didn't require a reboot. Since the file that broke it wasn't the driver itself, it BSOD'd the minute the update finished downloading when it was dynamically loaded in the running system.
Cold November rain got me...
And America banned kaspersky deeming it a threat, lmao
The real threat was the friends we made along the way
They banned Kaspersky because Kaspersky refused to stop tattling about the stuff they are doing.
Kas was actually wrong and sus for different reasons, granted, they're not exactly Angels either
To all my IT brothers and sisters, good luck!
As a Mac and Linux user, I have never felt more superior than in this moment!
I feel sorry for you, you have a very empty life
I've heard people mention that a similar issue affected Debian running CrowdStrike a couple of months ago, it just didn't make the news because it wasn't that widespread... Or the people who said it made it up, idk, too lazy to check. Either way, this can theoretically affect Linux as well.
@@the-answer-is-42yes, the real solution is not to install kernel rootkits, no matter what OS you are on. That being said, installing CrowdStrike on Linux is like installing anti-virus on Linux: you can do it, but why would you? Which is the reason no one has heard of the Debian crash (not even the CrowdStrike people, because it didn't happen) or the actual RedHat 9 CrowdStrike crash (that did happen last month) - because while Linux is the most used operating system on servers - almost no one is using CrowdStrike on it, only IT that also manages CrowdStrike on Windows and doesn't know any better.
Bitlocker, TPM, Other Security Measures miiiight mess up the safe boot method
It's wild how critical infrastructure can be rendered useless by something like this. Really goes to show how dependent we are on digital technology to function in today's world.
You wanna go back to pen and paper? no 3d diagnostics at hospitals?
@@mycosys how about disconnecting your doorbell and fridge from the internet? has no one thought of this?
What do doorbells and fridges have to do with critical infrastructure
@@johndoe1274 congrats on the stupidest most reconstruction take today i guess?
What part of a hospital CT scanner is the doorbell?
@@mycosys CT scanners don't require wifi smart one. Do you need an antivirus for a CT scanner? 😂 God I wish there were mandatory IQ tests.
this is why corporations hate open source and love close source proprietary software: it remove all blame from them.
Who wants to bet that crowdstrike CEO is still gonna get thatcsweet bonus?
the incompetence of people at the highest levels is incredible. welcome to idiocracy.
I have no idea why people are claiming this is Microsoft's fault - for once, this has nothing to do with them. Heck, the faulty software is also available for Linux systems.
I'm assuming people are confusing the Azure outages and the fact that (I think?) the update is distributed via Windows Update. But yeah, unless CS was relying on some sort of file audit from MS to detect corrupted files in the update, they probably had nothing to do with the actual issue other than effectively being the messenger carrying a faulty message.
This must be why many banks still use COBOL😂
its not a windows problem, its a crowdstrike problem.
Microsoft signed a driver with zero testing.
@@icantcomeupwithnames469 Uhh no
@@icantcomeupwithnames469 It's their responsibility to vet the companies they work with, and maybe to check update files for malicious content, but testing is on CrowdStrike.
This is what happens with greed and relying on a single platform with no competition
CrowdStrife? 🤣 @0:11
I think you missed the joke there.
@@alexv4907 lol no, forgot the the emoji
They must have hired productivity specialist from a major video game producer. Well at least the patch shipped on time.
riley and the other guy were affected by the outage too
Next time some asks why are software developers paid so much , this is why , as a doctor or a lawyer you cant do this much damage
Jesus Christ. Are emergency systems running ####### WINDOWS?!
Yeah, and why not? MS has been running businesses for decades with relatively few issues and this isn't even Microsoft's fault! Are you going to tell me that Linux/UNIX systems have NEVER had issues?
Yes, the NSA and the other Alphabet agencies need to have a convenient backdoor to those systems.
Sounds like a major headache for Windows users. Shocking to see the ripple effect of such a simple bug on such a global scale.
"Hotter threads than TJ Max" That is a really good one.
I'm a Linux user and no, I don't feel empathy for Windows users! I'm enough of a masochist already.
People buying third party security software had it coming anyway.
Aint you lot lovely
Eeh, I feel bad for the people who didnt get the surgery they waited months for... CrowdStrike isnt really installed on personal-use devices, so theres really no individual at fault here, more the failure of an entire industry deciding to create a perfect break-point by buying into quasi-monopolies like Windows and CrowdStrike...