The Worst Hack Ever Almost Just Happened
Vložit
- čas přidán 5. 06. 2024
- Sponsored: Discover the new @Bitdefender Cryptomining Protection. It’s available for Bitdefender Total Security, Premium Security, and Ultimate Security protection plans at no additional cost for new and existing customers. For more information visit: www.bitdefender.com/solutions...
▼ Time Stamps: ▼
0:00 - Intro
0:26 - The Discovery
1:08 - The Targeted Software
1:58 - A Very Good Thing
3:13 - How It Started
5:57 - The Attack Finally Begins
7:58 - The Hackers Are Forced To Hurry
8:45 - The Full Implications
10:29 - What Else Is Out There?
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
• My Gear & Equipment ⇨ kit.co/ThioJoe
• Merch ⇨ teespring.com/stores/thiojoe
• My Desktop Wallpapers ⇨ thiojoe.art/
⇨ / thiojoe
⇨ / thiojoe
⇨ / thiojoetv
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ - Věda a technologie
Imagine how pissed off that guy who put the backdoor in is, years of work gone, all cos some guy wanted a fraction of a percentage more performance
He was definitely punching the air
It won’t be the only one he’s working on…
It's scary to wonder how many hidden backdoors are out there, but remain unused.
Actually it wasn’t a faction of a percent it was drastically slower than it should be. If I recall it was like .5s slower than it should be which is a lot in the computer world
Imagine having foreseen this and loosing one of 300 similar backdoors😉
Moral of the story is never come between a database engineer and performance.
It is amusing that, for benchmarking purposes, the engineer who found it was sending SSH requests that shouldn't even pass a sanity check ("wrong username, etc") which explains why he got suspicious of some excess cpu cycles so quickly.
If he could be working on windows' explorer/taskmanager instead 11's wouldn't be slow as it is.
@@Lollllllz usually one can keep taskman on all the time. not on win11
@@Stratelier lmao
remember: the best backdoor is already running, is everywhere, and no one knows about it
Except the NSA / MSS / FSB / Unit 8200
Its called braking in irl. (This is a joke youtube, pls no ban)
Ah, fearmongering, my favorite
The virus Microsoft puts in win 11
Humans..
I must now change my password from "1234" to "12345" to protect myself.
Technically no matter how strong your password is this back door completely bypasses all passwords because it injects the hacker’s ssh keys onto the infected device.
@@samuelhulme8347 I remember my first joke too...
How did you know what my password was?🤔🧐
I don't know how you got my passwords but you don't scare me I already changed it into something more secure. With six digits it almost impossible to guess mynew one.
@@SereneStrategist-kk7mk is it “123456”?
To quote that old adage, "You have to be good all of the time. They only have to be lucky once."
I'm going to have to say: it's not an adage, it was a (very real) threat to Margaret Thatcher.
It's still applicable in this case, just a better phrasing might have been:
I'm reminded of the IRA's threat to Thatcher: "Today we were unlucky, but remember we only have to be lucky once - you will have to be lucky always."
You don't have to be good all the time*
@@NinjaRunningWild No I think he was correct. We (the good guys) gotta be good all the time, they (the attackers) only gotta be lucky once.
@@NinjaRunningWildpoint:
You
I think Seytonic covered this a month ago. But it doesn't hurt to remind ourselves: 1. Social engineering is a thing, 2. Pay developers what they are worth.
I don't think anyone ever gave the guy any money at all. Then he gave up, and the bad actor(s) took over.
How are you going to pay software engineers working on open source software for free?
@@Fircasice Many so called free software are open to donations. And people donate. Some of the money could be paid to the developers.
The fact this was just discovered by chance really brings into question how many other packages have similar backdoors. This is the kind of stuff that should spur a major investigation.
@An_EqualNot the FBI or CIA obviously, one of the founders of telegram said the FBI was trying to trick him into using open source libraries for telegram.
@An_Equal Beats me, but it's probably not just gonna be one singular entity.
Like Andreas said himself, this was just incredibly lucky, just a massive coincidence that he happened by chance to be in the perfect position to find it (and _just barely_ in time). A confluence of events like this rarely happens, so it's possible that there is indeed a lot of stuff going undetected. 😕
There is a "major investigation". This backdoor has sparked discussion on how to prevent something similar from happening again, made some free software contributors try to audit other software and once again demonstrated the absurdity of a "software supply chain" where the companies don't pay a dime to their "suppliers" yet expect them to do the most rigorous work to avoid hurting their (the corporations) bottom line
@@internet8080 Ok, post proof then.
This is the biggest weakness with OSS, but also the greatest strength of it. Anyone can worm their way into a seemingly innocuous part of the Linux ecosystem and taint it. But also anyone and everyone can topple years of nefarious actions through simple curiosity.
the fact this got caught, while Windows CVEs get put out and many admins don't update, leaving vulnerabilities in place for years! Notpetya took advantage of an years old vulnerability in Windows, and caused over $11 billion globally
By the same token, it's not hard for a nation-state entity to get an agent hired at a private software company
Yeah nothing's going to be 100% secure at all times, the payout from successful attacks is just too big. What OSS has is a living immune system, the ability to heal.
Bro is flexing proper subtitles 😎 my guy
Time stamps, subtitles. Bro is a role model
Invaluable
The whole linux sphere has been talking about this a lot, but yeah, I think you're the first tech channel with a more general focus I've seen bring it up.
Not just the linux bubble, the whole Cyber Community.
@@UmVtCg I bet you're probably right, I just don't tend to haunt that corner of the net.
@@UmVtCg I wouldn't say I'm in the Linux sphere and I'm definitely not in the cyber community and I still knew about it. It was pretty much impossible not to hear about it if you're in the "IT scene" in any capacity.
@@marcellkovacs5452 Its irritating the title is "Almost Just Happened"! no, over a month an a half ago. Clickbait.
Two Thio videos in one day is a win in my books
правда радует
yess
Agreed
That's what SHE said!
real
The xz backdoor story is crazy.
Not so crazy when you consider that at some point some one probably got themselves hired in order to put in the Juniper back door. This was found about 10 years ago.
As a software developer, I have no doubts that this kind of vulnerability (probably multiple) is already deployed everywhere, undetected. Never underestimate the power of social engineering, and these attacks being very easy to miss. Also I remember when ThioJoe had very few subscribers, I'm delighted to see the channel grow like this. I wonder if he remembers me 🤔
Probably like Pegasus on every phone.
Hardware backdoors are amazing for government use but amazingly we don't hear much publicly about that obvious attack vector...
Drive-By Mining. You have to give these guys credit for being innovative.
As someone on the defensive line working at scale (170,000 users), you do what you can with the control that you've *got* to avoid these issues, but you are mostly at the mercy of others. Where you *really* need to focus your efforts as a defender is being able to detect *when* you've been breached. Our goals are pretty clear - detect within 10 minutes, contain within 60 minutes. That's how fast you need to be, and some would argue that's not fast enough.
My Dad's mate was managing server infrastructure at a hosting company around 2010, and decided to deploy a crypto miner as a cheeky experiment for his team. It was a bit after a fortnight when the team found out, and they chewed him out for misusing company resources, but he immediately returned the blame to them. 'You're saying, if there was actually a piece of malicious software running on our systems, it'd take you two weeks before anyone realizes something's wrong?'
We've gotten so spoiled with our technology, we need more code and more programs and more features to cover every base. Thing is, the more we have, the more hands and minds work on the code that run on our machines.
That definitely comes with its risks.
The truly scary thing to think about is that... logic dictates that the worst is yet to come.
And thing is, we can live without most of it, too. Our ancestors even 200 years ago did.
Nothing had made it to the news where I live regarding this. Some tech channels on YT I follow covered the bare bones when this was first discovered, yet the background you've provided has created such a broader and more chilling account of what was really happening.
Remember: As an open source maintainer, you should keep an eye on the stuff coming in and just not accept incoming stuff if you don't know WTF it even DOES. (That's the technical term.) But I also realise that if you have relinquished the nominal control to someone else, you're not culpable.
I fully agree with this, but the problem was these so called contributors were intentionally bringing up so called "problems" causing the developer to burn out. Nobody remembers that these people do all this without guaranteed pay, they volunteer their time to better the open source atmosphere. It's really sad that people have to take advantage of good hearted people like this. This is why I always chip a few dollars their way whenever I can. We should try and keep these people happiness high. The actual owner was on Hiatus and gave the reigns to a person he thought he could trust, well that person was taking advantage of his burnout.
My first contribution was the Harvard's cs50 class CLI tool, translating it to my language Indonesian so my high school students can use it more easily. The maintainer raised this exact issue, "how do we know he pushed something legit, not troll translations?"
That's how I realized that while open source contribution is a cool way to collaborate, some people might have malicious intentions and maintainers should try their best to prevent it.
@@dputra They could probably chuck it into deepl translate and most of it would make sense. Having a native speaker translating seems like an improvement over any automated translations though.
@@NotAghostSpeedruns deepl is not even there yet at the time, only google translate which sucks at translating indonesian to english.
2:20 norton disliked this video
Haha, I couldn't believe they tried that. What a scummy company.
@@volvo09 What are you referring to? What did they try?
@@milentoshev8409 Their antivirus software became the virus. Granted it was opt in but there were multiple popups urging you to opt in telling you how great crypto is. They failed to mention about the wear and tear of hardware and the performance impact on other tasks. To top it all of they would not only skip paying the electricity bill they also took a 15% cut from your earnings
@@milentoshev8409 I dont remember all the details, but norton or one of their products had or has a crypto miner within them. They stealthily made it opt-in by default, and when found out tried some justification.
@@milentoshev8409 Norton tried to install crypto miners in their software without making it clear in the install process lol
Scary... I needed to check this
Wouldn't be surprised if this has happened to other bits of open source software at some point.
Like the xkcd comic Joe showed said, there are a LOT of bits of archaic code that underlie the world's software. We've seen cases where half the Internet broke because software relies on a single function that someone wrote for themselves 25 years and everybody copied. Software is more fragile than people would like to think.
At first I thought this video is quite a bit late. I've already seen multiple videos about this backdoor right around the time it was discovered. However I'm glad I watched till the end because this video provided some additional information and context I didn't know of yet.
I did not expect it to be a topic of XZ... thought this video was some windows thing
Also yeah way late
I love your coverage on topics like this. I find it so interesting and you do a great job of explaining the process. Great video.
You can do your own micro benchmarking and analysis with Process Monitor from sysinternals and run it as administrator.
Picks up background accessing.
Excellent info & video. keep up the good work!!!!
Thanks, ThioJoe
I'd heard of this, but not the full story. Essentially just heard "some guy was drag racing his computer for fun and noticed a tiny inefficiency which was a brand new back door, catastrophe prevented" and not all the cool details you gave! Thanks for this video.
Thank you for your work.
Jia Tan is a Chinese name, Jigar Kumar is an Indian name. People who want to stay anonymous won't use their names, but also doesn't want to introduce a rival nation to investigate (so he didn't use a german name for example), so quite likely a hacker of russian origin.
Isn't geopolitics wonderful.
So the backdoor would likely quietly delete itself if it detected a Russian keyboard.
Making it legal for your citizens to attack any system as long as it's not one your nation owns is a stroke of genius, ngl. I wonder how much money they've saved on buying day zero exploits from the usual sources?
Non-sequitur. Nothing can be deduced from the name.
'Cheng" is a cantonese name while "Jia" isn't. This indicates however made the backdoor just tried to think of a name that sounds Chinese enough. Such sloppiness is typical of the US
But trying to deduce the perpetrator from the name is stupid anyway, we could go in circles all day talking about potential 5d chess by the perpetrator
@@mega_gamer93The OP just have the politics brainrot. The culprits might be multinational anyways.
"Gee-yah" isn't Chinese, it only looks that way. In Chinese it's only one syllable.
First time I heard about this, thank you for sharing.
Zoinks!! Wow Scooby that was a close one.. Whew!
>>>>>>>>Is this>>>>>>>>>>>>>
The mantainer needs acknowledgement too. Having a life helped to deter the attack.
Love the format and the story itself.
This backdoor only affected amd64 systems (so ARM computers wouldn’t have been affected) and it would likely take some time before it got into Debian and Ubuntu LTS (used by a ton of servers), as they only receive non-security updates every ~2 years, so if it was discovered 1 month later, we would probably be fine.
1. If I remember correctly, there's a code that check specifically for amd64 (and x86?) architecture for it to run. (sus imo)
2. We're very lucky that the backdoor was found before it was released into stable Ubuntu LTS 24.04 release on April 2024. That might be the attacker's main target.
3. The fact that it was found by coincidence by microbenchmarking, ~500ms delay, is very concerning.
4. The attacker will learn from this mistake and might pull something like this again / another party is inspired by this move will do it in the future.
For some value of "fine". Yes it wouldn't have been a disaster but some servers would have been compromised for some time.
Explained very well. Heard from another youtuber but he made it all the way more complex
When this happened it got me thinking maybe it's time for a big code audit?
Hey, I just want to take a second to say thank you and congrats. I found you ages ago through all the pranks. Was funny at the time, but I can see why you moved away from it. Over time, you've given us some really amazing videos that are very informative and make it easy to digest for those who are less educated on tech. Thank you for the years of entertainment and information and congrats on how far you've come. Much love, bro.
This guy deserves a medal.
“And I would have gotten away with it if it weren’t for that benchmarking kid!!”
Heard about this the other day on the 2.5 Admins Podcast and the Late Night Linux Podcast, good to hear from some other people. It's a pretty big deal.
Was aware of this but good it is still getting coverage. Really feels like this house of cards is not gonna stay up much longer.
Yes, I have heard of it before. I watch a few IT channels. Your unscheduled video yesterday reminded me about this backdoor in that it looks like certain actors are attempting long cons to create vulnerabilities.
thanks for breaking that down, very interesting
Great explanation and I heard about this threat about a month ago on another channel.
Wow ThioJoe. Second video for today. I ❤ it
I'm not a programmer and I immediately spotted the "." because I don't like when it's not tidy 😂
I had heard about this, in general terms. Thanks for a detailed explanation.
I remember it when it was just discovered, but i think your video lacks the explanation of just how BRILIANT this backdoor is, and the code behind it.
Those antivirus softwares are useless. We need more performance tweakers.
I had heard about it but, like you said, it was only from tech news outlets.
Thank you for making a bid about this!
Yup. How could we know if backdoors already have been installed? Until...they are discovered. It could be a million or it could be zero.
You are a saint for documenting what will be the history of tech.
I bet the NSA planted the ultimate backdoor into silicon long ago. It is absolute logical
This is an excellent demonstration of both the benefits to security that open source software has as well as the threat to it. Had this been a closed source project this back door had it been implemented would never have been caught but also it would have been much harder to implement.
"it would have been much harder to implement." -- Yeah, it'd be really hard for a 3-letter agency to give that company generous funding, in exchange for a backdoor... ;3
And in case they refuse, remind them that accidents do happen...
idk what to comment (nice video thio, keep up the good job)
Crazy story, even crazier to think about what's out there that we don't know about.
So one thing to add, this was included in rolling release ditros like Arch as well, but my understanding is that, the way Arch used xz and the way Red Hat and Ubuntu used it were different enough that it wouldn't actually effect Arch systems
Arch doesn't have ssh compiled against liblzma (which debian and co had)
@@keit99 thanks for adding, couldn't remember exactly why Arch was different from the rest in this regard
@@keit99the distros did not link sshd with liblzma. The distros patched sshd on a way that linked it to libsystemd which is then linked to libzma
@@mega_gamer93 right that was it. IT's been a while since I read about the backdoor properly.
The method and approach they took were very refined and systematical. I wouldn't be surprised if this was only one of the many similar attacks.
Low Level Learning covered this right after discovery. His video is also worth watching.
He also said it's the end of open source and linux.
I knew about the backdoor since the time it was discovered in March 2024. The backdoor was discussed, it seemed, everywhere, I also watched a few videos which explained what it was and the consequences if it weren't discovered in time.
ThioJoe got a 2 vid in a day Streak
I knew about it during the Easter weekend thanks to a general channel Discord and some Linux/programming youtubers.
Arch really quickly updated the package and posted to their news page when discovered. Also just happened to be 1 week after I updated Arch WSL for a Samba setup involving Windows 98.
OpenSSH does not normally use liblzma but got patched by Debian/Fedora/systemd systems to work with libsystemd which did use liblzma but then 7:58 this pull request was going to make systemd not automatically load liblzma all the time which pretty much doomed the backdoor.
It's kind of reassuring that software improvements make backdoors in other projects obsolete. If the good guys keep fighting, they can plug holes they don't even know about.
Performance is number one
Safety Third
I wouldn't really say this "just" happened, it was a decent bit ago, but yeah, it could have been bad. One thing that's kind of annoying though is, well actually sort of two things, but both are alarmism
1 : "HOW MANY MORE COULD EXIST THAT WE DON'T KNOW ABOUT?!?!?!" No, this was discovered like a month after the impacted version first released precisely *_because_* everyone had eyes on it. This is the point of open source, *_everyone_* is watching, so millions and millions of tax dollars mean nothing. One random dude doing some basic benchmarking spotted that what *_should_* have been a basically instant no-op was taking over half a second and a decent chunk of compute, then identified this backdoor years in the making. If you are wearing a bulletproof vest, then get shot, and it gets stopped, you don't then say "woah, imagine how many other times I've been shot and I don't even realize it! I mean sure this vest I put on specifically to stop it stopped it, but that was just pure chance!"
2 : "See?! Open source isn't more secure!", except, 2.1 : see above, 2.2 : this exploit specifically abused how XZ was being distributed to have what is basically a closed source component be delivered with the final product. We do not have the source code for the exploit so, definitionally, the exploit was not open source. Since it was part of the project, that means part of the project was not open source. This attack had to reinvent the wheel several times over to hide itself ( and, again, still got caught instantly ) precisely *_because_* it was in an open source repo. 2.3 : your jordans are fake; this argument relies on the tacit assertion that this is an attack that hasn't been carried out on proprietary software several dozen times over. Not finding a vulnerability or not having it disclosed isn't real security, it's just feigned security.
The reality is this is a prime example of exactly why open source is so secure. Allllll of this time and money, full fledged psychological warfare, completely innovative attack vectors, etc. were all rendered completely and utterly meaningless because, against the millions and millions of nerds running automated checks and test scripts, they *_are_* utterly meaningless.
That's exactly how I feel too. Tons of doomer talk over this attack even though literally nobody was impacted lmao and the attackers wasted years of effort to be shutdown over night.
Malicious code can't just magically silently interact with software and hardware. It always has some impact on the system via the fact it has to run code, and that can be detected. Files not in the source code can be found with programs as simple as winmerge. This was always a doomed attack vector hinging on luck that no one would notice before they hit their target.
@@uponeric36 I wouldn't quite go that far, this is a very real attack vector that does need to be addressed. The fact that the wider open source community was *_ever_* okay with non-source-controlled files being included in the release builds is a massive red flag, as is the fact that build pipelines aren't themselves included in the repository.
This wasn't a purely 'luck' attack, it did exploit some legitimate vulnerabilities with the culture around open source, particularly around people's willingness to accept "ohh it's just a hobby project their not getting paid for" as an argument that works in tandem with " *_the entire linux ecosystem relies on this package_* ". The reality is the build and release pipeline for open source *_has_* been overlooked as meaningless tertiary shit, when that's far from reality. There are some legitimate security holes in that area of how open source is typically managed, but that doesn't mean it's doom and gloom.
Those are vulnerabilities that we should address, but my point is that a lot of people treat this like it's some cautionary tale when, by basically any account, thjis was handled borderline flawlessly. Sure, there was some evidence going back years, but if we're honest, go back to January of this year and all of that evidence is circumstantial and non-actionable. It was a few dodgy accounts that might be sock puppets, a few odd commits, etc. Realistically speaking this was caught like a month after it was even possible to catch it.
There are definitely some lessons that need to be learned here, but it's not the sign of the end times so many people are acting like it is.
@@robonator2945 You're right, I'm just layman lol. It is wild to me in retrospect that having files like that in the final build is "part of the culture"
I still feel like this was a perfect storm moment - the attack formed partially by negligence, but also against the odds of all the other security measures in place. Ultimately, the odds worked against it.
There's what I was considering luck, but "pure" *is* a strong word.
I did hear about this through mainstream media but I don’t remember where. I do recall the detail of it being discovered at Microsoft but there was no more detail than that.
Someone needs to write a screenplay for this and make a movie. So much drama.
If it doesn't happen until I'm a multibillionaire then I'll make sure to organise it and make the movie free-to-watch and free-to-pirate.
7:07 - very nice idea - must try it sometimes :)
I love your vids
i remembered how much of a nerd i am when he said there wasn't much coverage about it but all my news feed was about it
hey thiojoe i have a question, if i delete the microsoft folder in registry editor , will it brick my pc?
I did hear about a problem with vulnerability with the xz software but it wasn't widely covered.
As an IT Professional for over 30+ years, the world is a SCARY place. I just wanna go live in the mountains. Tired of playing whack-a-mole all the time. No matter what we do, there's always someone or something else out there that is better.
Scary stuff
I always wondered how they kept open-source software safe from back doors and other malign code. Turns out...they can't.
How many other attacks like this have succeeded and remain undetected?
I did read about this just after it was revealed by the Microsoft employee. I think I found it in one of my Flipboard items. However I did not have all the details that you just spoke about.
Theo, you do a cold impression of a North Korean Cyber 🪖 Officer.!
I once thought about open source risks and then forgot, since a whole community watches what changes... but if the hacker is patient (2 years!), it finally can happen... No I'm not afraid at all. _casually updates USB hub firmware, nothing bad can happen_
this is crazy i remember you telling me to tape batteries to my cat5 to make my internet go faster
XZ Utils is not really a behind the scenes thing for most Linux users though. Almost every Linux user will come across and extract .tar.xz files regularly, and sometimes even compress their own xz files, both of which require XZ Utils.
In other words most Linux users are aware of xz and use it frequently.
Ugh, this is just gonna be a feast for proponents of closed-source...
I hope Windows 10 never gets hit by backdoor hackers O.O
Totally enjoy your video TJ. Peace brother.
This was an amazing idea...
Super cool... an OCD technician wants to make the performance ideal, and thus saves the day.
as a owner of few public servers, i almost shitted myself
I just came up with a theory. The backdoor was actually discovered by some counter-intelligence agency obtaining information about this backdoor but not wanting to burn their asset, they came up with a cover-up story about some Microsoft engineer noticing a 100 millisecond delay.
Warns about one malware, advertises another.
This is an awesome story! Important as well, not much was mentioned about it to normies. The only things I saw in the avg person scope was trying to use this as a way to badmouth open-source software and that really did not sit well with me
Bit defender endpoint for Linux is for file hosting servers to check if the files they are hosting are malicious, it's not for self protection
to me what is scarier, is imagining all of the exploits who got through…
I'd seen a headline about this, but never clicked on it to read it. This is frightening stuff.
This would make me paranoid about security if I wasn't already paranoid from the time I (temporarily, to test something) opened up SSH access over the internet to a Linux machine on my network and saw it immediately get hit with constant brute-force login attempts.
I wonder if any of those bitdefender exceptions allow you to hijack the mining so the mining profit goes to you instead (robbing from the robbers...if a machine is being used that way the owner is the rightful beneficiary)
Yay, another video about the xz backdoor…
the person should not have added a new maintainer
that's why you shouldn't trust anyone
it sucks that contribution was abused
Trust = Weakness
"Just keep doing unpaid labor despite being burned out"
People all over the world: Oh no, my gaming pc is compromised! I will share this on my social media!
I already knew this because i have subscribed to morning brew!
It's even bizarre that Jia Tan contributed to Google's repos to disable or remove some packages so that his backdoor doesn't get added to Google's repos, he obviously knows that they'll be able to detect his malicious code. Very meticulous indeed.
It was discussed by Dave's garage on April 4, and other sources as well.
Nothing worse than not finding the backdoor!
I heard about it a couple of times before... But i fall several tech specific channels that talk about stuff like this all the time