My Biggest Server Rack Change Yet
Vložit
- čas přidán 24. 06. 2024
- Thanks to Linode for sponsoring today's video. Get a $100, 60-Day credit just for signing up at linode.com/CraftComputing, and start hosting your servers in the cloud!
What better way to spice up your server rack than retiring a $3000 Epyc CPU and replacing it with two Intel Tiger Lake 11900H mobile chip engineering samples. Today I set out to restructure my entire server, not only trying to boost performance and network speeds, but reduce power consumption, and in turn... my power bill.
But first, what am I drinking!?
Mixology March is winding down, and I'm running out of time for projects. So I'm just drinking a cranberry juice today.
Links to items below may be affiliate links for which I may be compensated
Check out the parts from today's build:
2U RackChoice Chassis: amzn.to/3lRecPT
Erying TigerLake 11800H 2.2GHz 8c/16t: s.click.aliexpress.com/e/_DES...
Patriot Viper 2x32GB 3600 Kit: amzn.to/3zewiP1
Silicon Power 1TB A60: amzn.to/3zdw2zI
Intel Box Cooler for LGA 115x/1200: amzn.to/3M6qP4t
350W Power Supply: amzn.to/3npJI81
nVidia Tesla P4: ebay.us/OjMWTE
Grab yourself a Pint Glass or Hoodie at craftcomputing.store
Follow me on Mastodon @Craftcomputing@hostux.social
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
The following music was used for this media project:
Music: Morning Stroll by Steven OBrien
Free download: filmmusic.io/song/10523-morni...
License (CC BY 4.0): filmmusic.io/standard-license
0:00 - CAT
1:35 - Welcome Back
4:08 - Unboxing and Build
6:21 - WHOOPS
6:25 - Memory and Storage and Cooler
8:34 - Graphics Cards
9:51- SERVER MONTAGE
14:15 - Post-Mortem
16:53 - Why Did I Do This?
18:24 - Sign Off - Věda a technologie
I can forgive using a stock Intel cooler, especially the newer, cooler looking black one. Paying $11 for said cooler though, that is unforgivable :)
To comment on "Why Did I Do This", this is the exact reason why I keep watching this channel. I like to see how creative you can be to pull out option of homelab without having to spend 3k for a decent setup. Or maybe I'm following because I like beer... :) IDK.... Has always, keep up the good work Jeff!
why not both!
Great video! Thanks again for the video about the erying motherboard/cpu because I put my erying 12800h into my server as well! It was quite the upgrade from my older server setup with a xeon e5 2470v2, PLUS my server idles so much lower now. I love it
I'm glad power consumption was addressed in this video Jeff. I know electricity is pretty cheap in your neck of the wood but for most of us around the world it's not the case, or soon won't be. I do really enjoyed watching your home-lab/server builds but always thought " I'd never use X or Y components Jeff is using in my own home-lab as the daily power consumption, heat and noise would put me off. Those Erying Tiger Lake boards look like a real winner! I'm waiting patiently for when they release an ITX version........ Well, would you look at that? Erying has released a 12th gen version in ITX flavor! Sweet! I just picked up an i5 12500H (12C 16T little beast) for $400 AUD. You should pick one up for a ITX build for the channel Jeff?
Imagine sticking a couple of those ITX boards in a 1U case, that'd be some pretty dense power efficient compute power
I wish I didn't just scroll down and read this... you bastard!
@@SenZubEanS I know what you mean! I thought just on the off chance I would check AliExpress as I heard Erying were working on a Intel 11th gen ITX board. Low and behold 12th gen CPU/mobos were listed!
How are you using the 12th gen board? I am looking for something to setup cloud gaming vm's with a P4 myself and curious how promox uses the E-cores.
Great video ... looks like lots of fun. Appreciate the reviews of the Tiger Lake board ... have one on the way.
Ooo a new GPU, that last review on the 70W card was mind blowing in how capable it was. Would love to see a comparison.
I have to admit the reasons for doing your own computer builds both servers and workstations is fun. Been doing computer builds since mid 1990's. More than 8 up and running on two homes. But now both of us retired my wife has other ideas, like boating. riding motorcycles. Here in Florida everyday is a Saturday. :) Thumbs up Video!👍👍👍 😀
I am super excited to see how these TGLs fair in virtualized gaming workloads. Tesla T4s and P4s are very very interesting little GPUs.
Enjoying the videos. Keep up the good work. What did you use for the drive bays?
Love your videos Geoff, making me wanna host my own cloud services more and more....
Always enjoy watching your vids 👍
I’m curious, are these severs standalone proxmox hosts or clustered? Also I’d love to see how you utilize the ssds into your setup of the two erying servers.
Love the Linode Intro today Jeff, Thanks :P
I noticed when I was testing out some things on my ERYING board (I found and ordered one about a hour before you posted your first video on them) that if I used both M.2 storage slots (I also had a wifi card in at the time) the x16 slot was only running @ 3.0 x4 speed with my 1070 in it. I ended up only using one SSD in the top M.2 slot and my old 1080 Ti for the GPU with everything working full speed as normal so I never looked into it, but seeing you put two SSDs in made me remember that issue that I never bothered to troubleshoot. IDK if mine takes lanes from the GPU for those other slots or if maybe I set something crazy in the 100% easy to use no confusion to be found BIOS. /s I know for overclocking this one it's running windows so I just decided to use Intel XTU (had to enable an extra setting in an obscure menu for XTU to see it as an OC capable system but works great now) Little undervolt here, touch of extra PL1/2 there, and up those multicore muiltipliers and it's a beast for the money. I want to order the new 12th gen version they haveout now, but they're all non ES and I'm too cheap for that.
I have the Erying ES 2.6GHz version. I put it in the cheapest case I could find with 16GB 3200 (soon to be at least 32GB), a 2TB PCIe 4.0 M.2 and a Red Devil 6900XT. This will be my room scale VR machine. I did overclock it and the CPU is pulling almost 120w. I went with liquid metal between the CPU and the copper spacer, then added an Arctic Freezer 34 eSports DUO cooler. The liquid metal stopped the thermal throttling at 97c to a max of 74c! And got me up to just shy of 5.1GHz on most cores.
This is really cool Jeff!
Nice new layout. Currently just running a single truenas on 6th gen intel but with the stock cooler also. It works…
I love your "Blue Peter" moment. Well done.
Freaking love it. Great job on the project. Definitely jealous of the power draw to performance ratio.
It's been a while, could you possibly show us a diagram of your network layout and how you have things set up for the lab topology? I'm curious how you VLAN this stuff.
I have to agree that swapping parts and rebuilding machines is the best part of having a homelab or just multiple computers for that matter. I've always preferred playing with hardware over software.
Without the software the hardware is useless.
hahah loved that intro! :D aswsome :D love the video overall :)
Such a satisfying video!
I loved everything about that intro
Would you consider purchasing one non ES board for the contrast point of what doesn't work with the Engineering Sample boards? This is especially relevant since you are doing virtualization on the systems.
Mine arrived today. At the Wall its pulling 150w at full boost. Not quite the 45w mark. Idle is about 15-30w.
Best Linode AD so far!
Look at the X10DRC-T4+, it has the same slot layout but they are x16 slots and it's dual socket.
Nice job, however a bit sad the 1 U blue server is going away. It was such an interesting machine!
I have to admire you, using a Engineering Sample CPU to run a server is ballsy :) Though I am about to get a similar board based on 13th gen from them to use as a low power HTPC
Cluster tutorial! Cluster tutorial! Cluster tutorial!
I think I spot SNK vs. Capcom: SVC Chaos on the arcade cab in the back :)
I've been sceptical about SP SSDs. Now I see you use it in server, I'll give it a try.
this is beautiful to watch
Fun video as always, I'm updating my 1st gen thread ripper that has been running truenas and all my plugins. New version will be Ryzen 5700g to cut power and need for a GPU.
The "paradigm couplers." That's awesome.😂😂😂
new servers are looking pretty nice
Fascinating !
Great video! I keep meaning to ask what do program do you use to play games remotely on your cloud gaming servers?
Okay, but the server montage music was 🔥
HAHA, I love the linode sales ad. 😁
I might actually steal this idea for hosting some gaming servers at a remote location. I think it's time I retired that ancient r710
I'd love to see how the performance of Epyc does compared to my aging Ryzen 2700. Tried to do a two-pcs-one-cpu type thing on it before but it wasn't satisfactory for the wife, hehe.
The Message at the end is so true. We have the luck to get our hands on an 12 year old used server. But it surely would run on other hardware as well
You turned me onto those Erying boards and just got my 11800 ES 2.2GHz with the vrm heatsink upgrade. I felt your frustration with that one, they sucked. Cant wait to get the rest of my parts and use it to replace my Dell HTPC with a i7 2600. Going to move over the HDD and SSD in it as well as the blu ray drive and the GTX 1650.
Glorious ad, as always
Hey, I think a lot of us would like to know what services you are running. We have heard snippets of it here and there but it would be great to have a dedicated video on the running services
I dodnt know I was watching a linus tech tips video.
Damn jeff, get it together!
Just picked up a 2nd gen thread ripper for this exact reason.
Checking back in a year later to say, I built one of these erying 2U servers for my homelab and have been enjoying it. HOWEVER, the intel stock cooler randomly died. Weirdly enough, the heatsink was enough that the CPU was fine--just limped along at 90C until I got a new cooler.
Wait....I'm confused.
If you've retired your EPYC system with Tiger Lake systems, aren't you also giving up all of the PCIe lanes that the EPYC gives you that the Tiger Lake system does not?
I couldn't help but purchase a p4 after your last video. Using some obscure Google driver and a virtual display, I've been gaming on it through parsec. For 70 watts, and $120 CAD, it's seriously impressive.
What driver did you use if you don't mind me asking? I've been trying to figure out a good way to deploy a P4.
I'd love to see a video on the bartop behind you
What memory speed did you end up getting those boards to run at? My 3200 Corsair memory should arrive tomorrow! 🤞🏼
I also did away with my dual-socket X79 system and replaced it with a first gen Threadripper 😁
rambo its back
just watching this video makes me sneeze.
Herein Australia 🇦🇺 we’re paying around 25c (USD) based on our hourly pricing …. So a 40%+ saving on power will definitely save a tonne of cash on the quarterly electricity ⚡bill 😇💪🤯🤩👍. Man looking at that home server …… 🤤🤤🤤🤤. I’m starting small trying a smaller itx build with multiple SATA SSDs for archiving, movies, photos, backups, a little smart home setup just to get my feet wet in my home server 💪😱. Fingers crossed 😅😂🤣🤞🤞
Loved the in case of AI revolt bat'leth
We need a diagram of all these fancy moves and builds :)
your cat's making me sneeze(and I'm not allergic) LOL Cheers!
How did you get past the pci passthrough issues on the chinese board?
Your power prices make me jealous. Residential prices in my area 11.2 cents per kilowatt hour.
Hi, what do you use for blowing dust, is that a datavac or just compressed air or a normal air blower?
nice. in the end when you talk about the total cost of the board, cpu, vga etc. around ~$600, did you use the Tesla P4 or the Tesla T4? i think T4 is about $600-$700 alone. How much did you buy the 2 T4's that u used is the servers?
I have my rack in my dining room/Lab and they look dirtier than yours does the day after I clean them.
It all comes down to how much dust/pollen/mold/smog is in the area and how old the house is.
Note: Non smoker. And my kitchen is vented separately so no airborne oil either.
Lost a mainboard on my home server last week. Friend has a spare, hoping the RAID5 (shhh, I know) is fine :/
My server stack consisted of a cheap ~$115 used Dell CloudEdge C2100 with 12 3.5 inch drives and dual Xeon L5640
But because those werent fast enough for plex+transcoding, i needed another machine for that, back in 2016 it was a dual X5690 build, but after the 5700G came out i switched to that.
I recently switched to a 7950X to hopefully go down to an all-in-one box for file server+plex recording+TDARR CPU transocding+Photoprism AI image recognition.
Unfortunately TrueNAS does not support ROCm GPU acceleration for kubernetes/apps out of the box, and a recent bug means that none of the kubernetes/apps can even access the files on my file share meaning i still need the 5700G for plex and TDARR
While idle the stack has gone from ~600w, down to ~220w, and that includes the upgrade from a 16 port gigabit switch, to a 48 port gigabit+4x10G+2x40G
With the 5700G out of the picture it idles at ~130w and should peak at only 350w.
Man those 1TB optane 960P drives are awesome for accelerating pools and reducing load on the HDDs
Oh, and IIRC that 220w number might include my redundancy server(currently my primary until i figure out networking and move everything over to the new enclosed rack), but this server is really low powered as is all flash (18x2.5 inch) IIRC idles at 40w with much of that probably being the SAS controller and 128GB of ECC
The Cascade Lake boards are coming down in price as the Tiger Lake boards are being cleaned out fast. 12650 and 12500 options are decently priced.
I like the goal of reducing power consumption. My webserver is an HP office PC with an i5 9500. It needs just 7W in idle.
Any update to running gaming VM's with your new Nvidia T4 on those ERYING motherboards?
New server day!!!
These are nice upgrades!
Though Tesla P4 are only listing 8GB, are you sure it's 16GB?
Also what's the drink?
T4 is 16gb and the P4 is 8GB, he has both.
cranberry juice...
no, seriously, read the description lol
Me: What chassis is that? ATX PSU in a 2U frame without doing some jank bracket and double sided tape work? I'll take two.
Jeff: I linked the currently available equivalent in the description.
Me: Hopefully they ship to Guam.
Can you check the C-State compliance of the board, I'm looking to move away from my power guzzling 2697v4 pair, to a Proxmox HA Cluster with Ceph but need something that actually drops down the C-States
How about the PCI passthrough? There are the same issues with proxmox like your previous video on i9 motherboard?
The question of the day is have these come back down in price since your 1st video? Would check but am at work and my work internet doesn't like aliexpress.
Man i wish i had these prices on electricity... just the fact that your price is in the single digit cent range feels like another universe or smth... meanwhile im paying 54 cents per kw/h and i'm already crying when i see the 100w idle from my virtualisation server =(
Can't you fashion an intake filter (3M HVAC filter material) to keep all dust from entering your computers?
Did I miss something with these Erying motherboards how do you add networking more than the builtin 1gbs?
Is ECC memory no longer of importance?
Which filesystem are you running on these?
ofc it is
Isn't that the Tesla T4 and not the P4? Think you said P4 both times when you held up the T4. P4=Pascal and T4=Turing. Making sure we all are following along.
Yeah I had to rewind a couple of times to check if Jeff said P4 twice, and sure enough he did...
Would almost any other cooler be "better"? Maybe, at a bare minimum 2x cost. Would you be able to tell without having to pull out the micrometer? Likely not.
Can we get a video of the B roll of building it?
How about 4K remote gaming on that Tesla thing on that tigerlake box?
Yea I separate out my system. I have one bathroom upstairs and one bathroom downstairs. I also have 2 power meters on my house. One for each level. Though my server problem is hot water. The water heater only connects to the panel upstairs. Oh, Wait!! This isn't about throne redundancy? :)
I love the Star Trek sketch.
Intel Mobile processors really are difficult to beat when you need efficiency more than you need performance or thread count.
i would love to have a HomeLab based of those "Erying TigerLake 11800H 2.2GHz 8c/16t" damn.. I think i would even forgive it for not having IPMI..
What Budget Mobo/CPU combo do you recommend for “Fractal Define-5 Case” that I want to load up with hard drives and 2 LSI HBA controllers ? It just a home server nothing super fast will be replacing in WD MyCloud NAS drives.
@craftcomputing isn’t the Turing version of the p4 the t4 ( the silver card?) I have a p4 in my r620 but want to put a t4 in my epyc system. Any chance at a side by side on those two?
Your power bill makes me die a little inside.
Our power used to be the lowest in the country at $0.059/kW.
In the last 18 months it has risen to $0.34/kW.
Right now half my rack is just off.
I'm at the point where spending $800 on a new system that uses 1/3 of the power will cost me less total money in about a year.
its pretty wild, I live maybe half an hour south of Jeff's area and our power is 0.14/kW. 2 years ago we paid the same 0.07/Kw he does
I do wonder about the money you save, 200 watts less, but what saving in energy bills is that to how much it cost to replace the hardware? This is one of those questions i plague myself with, solar panels would be awesome for the house, but i put $60 in the meter a month. if i spend like $7000 how long is it going to take me to pay off. lol. Though the one advantage straight away would be being able to run way more electronics guilt free.
How long did it take you to find the screw that fell on the floor?
Apart from being a dual slot card, is an second hand RTX A2000 not a better value/choice?
The T4 is around €500 (cheapest option) P4 around €120 and the RTX A2000 (6GB) I can get around €250.
And thats with cooler.
edit: stupid question after trying to get vgpu working on a rtx 3090 (Ampere) for hours/days🤭
This is nerdgasm, love it!
Does the chinese boards allow PCI-Passthrough to virtual machines?
Erying saw this video, that's why they released boards with mobile xeons. No ecc support though.
E3-1275v5 at idle draws 15W from the wall 😋
well the cost of the Mobo and Proc are going up now that they are being bought when i cheked today it was alread up over 200$ US to get these and I cant imangine the price will come back down... ill have to decide quick if I want to try these myself.
@5:20 see how glossy the heatsink block is. it's damn near perfectly flat.