My Server Flooded
Vložit
- čas přidán 7. 07. 2024
- lmg.gg/secretlablmgclips
Thanks to Secretlab for being the title sponsor of LMG Clips this year! Check out their ergonomic gaming chairs at the link above. Your back will thank you!
Linus’ server racks flooded.
Watch the full WAN Show: • Elon Musk Admitted I W...
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloatplane
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US ON SOCIAL
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
TikTok (LMG Clips): www.tiktok.com/@_lmgclips_
Twitch: / linustech - Věda a technologie
Didn’t know about his sister. Sorry to hear that Mr Sebastian
Damn. He had his sister on the show once on a build video. Is that her?
@@FrietjeOorlogno, he told it wasn't her
@@FrietjeOorlog No, he addressed it in one or two WAN's ago, that it wasn't the one in the video
yeah that hit me like a gut punch. My condolences, if you read this Linus
@@misterpuff797 thank god, the one in the show was so young, hope it wasn't premature tho
Sorry to hear about your sister.
Can someone explain?
@@DRSDavidSoft Linus had an unexpected death in his immediate family that he talked about a little in a previous WAN show, turns out it was his sister. He mentions it pretty quickly in this video.
@@TheSencatShit is that the sister we saw on video?
@@TheSencat People need to acknowledge why this is happening so frequently so it can finally STOP!
@@dakoderii4221 It's too late to stop it. At best, we can place the blame where it belongs.
I'd watch that "resurrecting my water damaged UPS" video. It actually sounds super cool
Sounds more like a fire hazard, so super hot actually.
Something like a UPS you just dont risk, same goes for a PSU. Its safer, depending on why it died, to just get a new one. Water damage can make stuff do fucky things, and a UPS flowing quite a lot of power can still fail after some time even if its "resurrected"
There's not much to a UPS. If it's just him swapping internals from a donor there's no real risk as long as the chassis is completely dry.
@@renderedpixels4300 As long as you are competent you can repair a UPS, its functionally more like working as a mains electrician than a computer tech so you do need to know some things and the repair in most cases for anything now is replace larger assemblies - one FET died, replace the whole control board...
Multiple submarines have been lost due to water flooding into battery banks... I would not risk it. Just e-waste recycle it too be honest. If Navy folks cannot even figure out how to handle water inudation into battery banks then that UPS is a waste of time to try to salvage and dangerous.
My condolences Linus. I can't imagine the pain of losing a sibling.
May she rest in peace.
Sorry for your loss Linus, and well done for having the restraint in that moment to not rip the server rack apart and smash it all into tiny pieces.
I understand the whole point is for this to not happen to begin with but… there’s homeassistant compatible devices you can put on the floor to detect water leaks, then you can use that to fire off a critical emergency shut-down
I would like to see some kinda of device that can detect a change in flow or pressure, out side of a window, and fire off events
@@DarkAxi0m can’t remember what it’s called but there’s an LTT video about a device that keeps the entire water cooling loop under a bit of vacuum, so that if it has a leak the pressure will change and it can sound an alarm- but no water will get out because of the vacuum.
But this has giant, open reservoirs so that wouldn’t work
@@PentrossAquacomputer LeakShield
@@DarkAxi0m Or, a level detector in the tank itself. If it drops too low then somethings wrong and it needs to kill everything. As well as CPU temp monitoring if the pump fails
@@Pentross That's a bit over killed and would require the pump to be at the end of the loop etc.
Because they use a static flow rate pump, a single flow sensor on the return line would be enough. If the flow decrease even a little bit, you know either your pump is bad, or you have leak (be it small or big). You could complement that with a water leak sensor at the bottom of the rack and use them to turn of the pump in case of a problem.
sorry for your lost linus best wishes
"...I'm not saying it went well, I'm just saying that I'm not a f*cking idiot." -Linus 2024
Linus we watched the home build series.
maybe a voltage differential between 2 metallic parts in the same loop?
next time make shure every part is eighter grounged ot completly isolated.
In 1988 I was working at IBM, in a brand new (unfinished) 56 level tower near Paris. And one day, we saw water falling from the ceiling, slowly at start, and becoming more and more like the Niagara’s fall. The building had 4 reservoirs of 30000l on the roof, for water usage in the building, and as a fire suppression, and one brand new valve broke, letting 10000 gallons of water, going down the 56 floors, devastating everything in its path. Fun time.The worst was the water getting more and more dirty at each floor, with the particles of ceilings tiles, and other crap.
that sounds like a spectacle
Why store underground and use pumps if you could just play the world's worst game of Jenga!
That is horrible
Where in this were you?
We had the same thing happen, but in an 8 story residential building and with the water heater. We were running all over with every bucket, bowl, and trashcan we could find trying to save musical instruments, tech, and furniture
Air Cooler: I will overheat this cpu for fun.
Water Cooling: I will end this man whole career.
"I once used a flimsy intel stock cooler and now I think all air coolers suck." /s
@@xXRenaxChanXx technically,
@@xXRenaxChanXxthey suck and blow, its an air cooler thats how they work :^) Badumtss
reasonable AC will cause your CPU to throttle if fans stop but you still will be able to finish your work for the day, just a bit slower
if a pump stops... oh well
and if it leaks...
Rip Linus sister I hope you're doing well Linus I know it's not easy to lose family that close.
Sorry for your sister first of all.
Your story really helped me today. Had a IT crisis today. Came back to a dead SSD after whole day of work and dealing with the gov for my ID. Saw this clip while frustrated. I got hope back and, following linus' precedence, left to get food instead.
Thanks for the motivation. Hopefully it'll work out for me too
Everything looks better on a full stomach
Manifesting everything working out Man U got this
Thanks for the likes! My SSD is completely dead and I had to reinstall my OS on a new one, but my backup setup worked and I got all the data back from the cloud. So everything worked out in the end
There's some universe where Linus just said F it and cooled directly with the pool water, and his entire basement filled with water.
Have you tried putting it in rice?
He means well 😅
to summon Dennis?
The sister?
@@joaopedroalbernazoof
@@SignalJones Lmao
The corroded connector sounds like it was caused by electrolysis induced by a grounding loop.
What if in the construction someone took a slightly identical connector made by a different material? It could be an human error
Al vs. copper
0:41 Damn.. Our condolences Linus.
In the aquarium hobby there is a sensor that sits on the floor and alerts you to water leaks.
That it the FIRST ff thing you do when doing shit with water.
My condolences for your sister Linus, may she rest in peace
I'm always surprised at how resistant tech is to water. I'm still sticking to air cooling.
Yeah, I also got lucky with hardware and water: a few years back I was moving, my desktop was already outside to be loaded into a car, I did not notice it rained for a bit while I was inside finishing up. Once in the new apartment I connected everything and fired up the PC, which returned a black screen after a while I noticed water was coming out the bottom of the case, I my heart jumped, I let it dry for a day or so, tried again and everything was working just fine.
Yes, never had an air cooler kill a PC in almost 40 years of owning systems.
@@mastroitek Even if its not water the number of times I have had a issue with a piece of tech whether mine or my friends and I replaced it or set it aside intending to throw it away, check a warranty, evaluate possible fixes Etc only to get busy or forget about it and have it work perfectly fine once I got back around to dealing with it is kinda astonishing.
Just knew about your sis, Linus. Rest in peace, and my soul and heart goes to your family
Oh, this brought back memories! I work for a large ISP. A few years back we were upgrading the incoming fiber to one of our contracted housing associations. This required drilling up and re-sealing the concrete floor of the server room, after which the association hired another contractor to sand and polish the floors. During this time, the existing hardware was still up and running.
When we came back the day after, we realized that the contractors hadn't covered the racks AT ALL. Every single system was CAKED in concrete dust, and they'd all self-terminated due to overheating. The entire room was covered in dust. We were less than pleased with the association, and told them that they would have to get the contractor to clean up their mess before we started to try to get everything up and running again.
Some of you just had a chill run up your spine, and yes, it is exactly as you feared. When we came back, they had literally pressure washed the entire room, including our racks. There was water dripping from the switches.
I still remember vividly how my boss got the call, stood up, and exclaimed incredulously - "They pressure washed the fucking racks.". I don't think I've ever seen a more apt example of someone having lost their faith in humanity...
They are 100% going to pay for damages
Maybe you need some sort of "drip tray" every few machines in the rack, which could divert the water away somehow. Like a rack shelf but with a fall towards one side and a drain hose away from the rack. This would isolate any failures to only a portion of the rack rather than losing a whole rack. If you also added a water detection into the tray then you'd know there was a fault. I didn't see anything with quick googling but I'd guess some 3D printing could produce something.
I think you could just make trays from sheet metal for that. Also yeah having flood sensors should be a no brainer as he's already running home assistant anyway, so all he really needs to do is to buy some.
I would add drain holes and tubes to all the systems which end in a single drip tray/container to limit them dripping on each other. Finally add a water sensor to the drip tray.
@@Iisakkiik Home assistant is a terrible way to deal with cutting power because you're relying on so much to work right; Best way to deal with it is to simply have a hardware water cutoff that cuts power, a close second is to use the tools that are likely built into the PDU's/batteries and have a graceful shutdown.
MY condolences go out to your whole family. I am sorry to hear that you lost a family member. It hurts so much, I can't imagine what you're feeling and totally respect you for coming in and still doing the things you do. Keep being a reason for me to have pride in the people of this city.
So sorry for your loss Linus, wishing you and your family some peace in these times❤
Dissolved solids in the water, and probably leaking current from the pump. I have zapped myself on a few occasions with hydroponic gardening and pumps leaking stray current into the reservoirs. Quickest way to find out put a bucket in the loop and stick your hand in there after a manicure if you can feel tingling on your cuticles its stray current.
Sorry to hear about your sister, thats heartbreaking
Im so sorry to hear about your sister. My condolences to you and your family.
Instal moisture sensors throughout the rack.
Sorry for your loss Linus
I do love when people forget that the IP rating isn't just about water. You tell them something is IP rated dustproof and they just look at you confused.
Ouch man sorry to hear about that very hard day
My condolences and hope yall get the opportunity to properly mourn your lost
A few months back I had a water pipe break in my house right above my lounge room where all my electronics and my desk are and water ran through everything in there for hours before I came home to turn off the water, went through a similar state of mind when diagnosing everything but was also extremely surprised that damn near everything survived, even batteries(I watched em like a hawk), after letting everything dry for a week. modern tech is surprisingly durable
It all made sense when he mentioned EK for his water cooling
I'm sorry to hear you lost your Sister Linus. I know it is poor consolation, but I'm sending my best wishes and condolences for your loss.
Sorry to hear about your loss
buy the thing, wanna see you fix it
would love to see the ups rebuild video
sorry for your loss Linus!
That's amazing most of the equipment survived. Way back when, my buddy had a 7990XT in a Mountain Mods Horizon case. He spilled a drink that ran off the edge of his desk right in to the open top, onto his 7990XT. I didn't see him online for a few months until he recounted this story to us. Lost the mobo and GPU to a cup of water.
Sorry to hear about your sister.
You need to install humidity detectors, alarms, emergency shutdown valves etc
My deepest condolences for your loss to you and the family, sorry for the server flood
Lost a best friend and an uncle in the past, my condolences. Losing someone close to you never gets easy.
A video fixing this and having monitoring sensors. Bonus points for having a video camera so you can check up on it remotely.
pretty sure he already as a surveillance camera in his server room
Cameras only give you a view of what happened after. And may not work if the nvr recording it all is soaking wet
Sorry for your loss
bro called xeon v3 ancient. My entire homelab runs on v3s and v4s
It is ancient.
Mine is also entirely on v2s and v3s.
Well, they're 10 years old
10 years is pretty ancient for tech
I wonder if there’s a way to add a mechanical shut off that runs from a flow rate monitor that triggers an emergency shut off when the flow rate spikes indicating a leak? Sounds like a Pii project.
Could do what factories do and put audio sensors that listen for out of place noises.
I'd say a pair of flow sensors before and after the rack, with logic that if there's less flow on the after than the before, you have a leak, and therefore shut down the pump and fire off a notification.
I'm sorry for your loss
I'm so sorry about your sister, I lost one of my brothers a few years ago and know how much it hurts. Losing a loved one isn't something you get over, it's something you learn to live with. My condolences and prayers to you and your family.
My condolences to you, the family and possibily her family and friends.
Why is it stuff like this always happens after an awful day.
So sorry to hear of the loss.
My condolences Linus...
I’m soo sorry to hear about the loss of your sister my condolences to you and family
Condolences on your sister. Losing family is never easy (I know I still have to properly deal with the death of my father over a decade ago). Best of luck to you and your family, immediate and extended and otherwise through this.
Have you considered reaching out to Jay (JayzTwoCents) about the water cooling issue? He may also have some suggestions including if you want to go with a different company rather than EK for whatever reason.
Pretty much all houses in North America will use 120v/240v two phase power, some apartment buildings and a lot of industrial sites will use 3-phase, where you can get 120v between neutral and a phase, or 208v between two of the 3 phases. So in theory appliances that use the ~200+ volt power could either get 208v or 240v, and there are usually fluctuations in the power grid so voltage can skew from those numbers as well.
Sorry to hear thoughts go out to you & your family. That's sad
aquaman tech tips, de-flood
Much love to you and your fam. Sorry for your loss, Linus.
My condolences.
Fitting corroding and exploding is terrifying. Looking forward to update video because my water cooled server is now making me anxious lol
My condolences Linus
The company I work runs on a 2006 Dell Poweredge server, that got flooded, with a ton of mud, we pressure washed it, replaced the thermal paste, left to dry for a couple days on a room with the AC on the dry setting, and it's been working 24/7 ever since...
My condolences. Sorry for your loss
Add a flow meter than alerts you when the debit is over a set value.
And as someone points out, it's probably an electrolysis because of a grounding short.
Check the other systems for possible grounding problem and electric tape them
My condolences to you and your family about the loss of your sister.
Goddamn linus goddamn, the server must be begging for mercy
When it rains it pours
Man, RIP to your sister and my condolences to you!
The first sentence is pretty much my experience with AMD as well. I only run a RX 6700 XT, but the eight months I have had with it so far were pretty uneventful. I had one minor bug that saw me roll back drivers to the last version and wait for the next one and that was it. Nothing I haven't experienced in the decade of Nvidia before.
I am sorry for your loss Linus, take as much time as you need.
5:49 now thats what i call open loop water cooling
Condolences on your sister Linus, greetings from 🇩🇰
Brass and copper are still susceptible to galvanic corrosion. Galvanic corrosion happens when two dissimilar metals are in contact with each other AND have some sort of electrical current running through them. It can even happen to the same metal with different levels of impurity or different alloys.
Rest in peace Linus's sister.
Really sorry about your sister. Very HAPPY about your Server survival
Sorry for your loss.
Oh man, that's rough, sorry about your loss Linus.
I had a server room flood caused by one of the cassette ac's which had an electrical fault which triggered the fire sprinklers (because it's in an office building, you're not allowed to remove them). We immediately killed power to the server room, removed all the servers from the racks... Rinsed them under running water (because the insides were dusty before the flood), stored them on their sides to drain and hit them with heat guns and hair dryers. I think our survival rate was around 85 to 90% as well. With all of them going for another 4 or 5 years until they were upgraded and replaced during the next refresh.
watch out on the lithium ups if water got into the bms it can cause a fire after a few charge cycles depending on how good the bms is once water gets in it can cause it to get stuck on losing the over charge and cell balance protection (why most ebike /scooter fires happen )
Damn Linus. Condolences
Sorry for your loss Linus.
first thing I always do is spray on another conformal coating on all PC parts pre install and after assembly , helps stop dust causing corrosion in humid area's and protects against any other liquid damage that might happen , amazed you got away with a flooded server rack , I normally use water to water exchanged loops so they liquids them selves are isolated per machine and the main loop just exchanges the heat from 1 block to another , sometimes impurities just get into the loops , I also nickle plate everything myself
Sorry for your loss and the incident.
Sorry to hear about your loss, Linus.
I wish your family the best
❤
Sorry about your loss bro.
My condolences Linus. May she rest in peace.
We had a freak weather event here on Friday. Rain at a rate greater than 4 inches per hour. Overwhelmed my massive curtain drain for the first time in 20 years since installation. Water was 3 inches up the basement windows but thankfully didn't blow out the windows. Stood in 8 inches of water pumping out the basement window wells. My neighbor had his basement windows blown out and his finished basement flooded AND his pool full of muddy water. Have a whole pile of lithium batteries on the basement floor for the solar system - that was my first concern. Also have an unpopulated rack with servers on the floor waiting to be installed.
What I do is use a heat exchanger for each machine to my cooling loop and use Koolance 705 non-conductive coolant. Highly recommend, been running like this for years.
You should maybe build and fit some drain pans between systems. Easy to design and build a 1U drain pan to catch water that might drain from a water cooled system. ;)
Also, here is a most outrageous video idea.
Step 1) Gather 2 identical motherboards and build identical systems with them using inexpensive components and test them.
Step 2) Disassemble one system and its components, spray some clear enamel spray paint on the circuit boards of each component (including PSU) on just one of the builds. Be careful not to coat heat spreaders, connectors or contact pads that might be needed for them function. Coat enough of each board so that it might be considered water resistant.
Step 3) Reassemble the now "water resistant" build with components and spray the system again while it is assembled so the paint will coat PCI-e slots, power connectors etc... Test that it still works like in step 1.
Step 4) Get wet! Moisten both systems, coated and not coated, until they crash. How much water can they each take before they are completely unusable. Try some number of times to dry each system, test its stability and soak it again. Be sure to do this safely and away from any mains electrical outlets or cords. Do not touch any power cords with wet hands or touch any wet power cords near either end, they will shock you... ask me how I know! [insert random employee laying on the ground saying, "I'm okay. He told me not to..."] 😂😂
Step 5) If any system is still running mask off the CPU heat spreader, spray some clear coat around the CPU and socket edges and then try to directly water cool the CPU with a garden hose.
DIRECT WATER COOLING... WITH A GARDEN HOSE ! ! !
Profit and have a laugh. ;)
Thinking about an automatic fault detection. Something measuring the flow of water going into the loop and back out of the loop so you can measure the difference and if there is a difference, shut everything down.
sorry for your loss!
Condolences to you and your family
Sorry about your loss Linus, I wish you and your family well.
Condolences to the Family Linus. Also is the sister is the same sister that build a pc with him on one of the LTT video?
he said last wan show that it wasn't
Sorry about your loss.
I'm very sorry for your loss Linus ❤
i have no doubt the time she had was enriched with the moments you had together
Thin sheet of plastic between machines can help direct the water off the machines below!
sounds like a upgrade video, of adding in a fail safe. so if the water gets low in one of the tanks an alert goes out. if it gets lower it starts killing power to pumps and rails to keep a fire from happening.
My condolences to you and your family in regards to your sister.