Upgrade Problems - Cloud Gaming Server Edition
Vložit
- čas přidán 17. 09. 2021
- Thanks to Linode for sponsoring this video. Visit linode.com/CraftComputing to get a $100 60-Day credit when signing up for a new account.
First world problems are real, especially when trying to do things in a homelab no one has done before. My three GTX 1070 Tis just aren't quite fast enough to run 12 Gaming VMs at the same time in my Cloud Gaming Server. So, I upgraded...
But first... What am I drinking???
From Treehouse Brewing comes the Double Shot Imperial Stout (8.9%). If you know what to expect, which is essentially German Chocolate Cake, this is a pretty great desert beer. I just wasn't expecting cake.
Links to items below may be affiliate links for which I may be compensated
Check out parts from today's video:
AMD Epyc 7742 64-Core CPU: amzn.to/3Cmdmxd
AsRock Rack ROMED8-2T SP3 Motherboard: amzn.to/3hKfmYj
256GB Crucial DDR4 REG-ECC 2666: amzn.to/3zoZLDn
Synology M.2 SNV3400 800GB NVMe: amzn.to/3ks95lT
Asus Hyper M.2 4x Expansion: amzn.to/39ksUoO
nVidia GTX Titan X Pascal: ebay.us/e3FOMk
be quiet! Dark Power Pro 12 1500W: amzn.to/3zmxu04
InWin R400n 4U Rackmount Server: amzn.to/3nLbNVB
Get your self some merch at craftcomputing.store
Follow me on Twitter @CraftComputing
Support me on Patreon or Floatplane and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
www.floatplane.com/channel/Cr...
Music
Bossa Antigua by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
License: filmmusic.io/standard-license - Věda a technologie
this "local cloud gaming" project is totally out of my league but I really enjoy the tinkering that comes with it
Tbh I thinks it's totally doable in a lower end form for people I don't wanna go the server rack form
This is literally IT in a nutshell. "Well it should work. why isn't it working. rinse repeat"
Jefffffff leave the back half of the backplate off the card when you stack them like that. That's what it's designed for!
No need to choke them more than needed
This. Maybe he could even remove all of it so there is a bit more of space between them.
@@TheMinigato you have a point. It's minimal but you would get airflow over the PCB where there currently is none
Maybe the video card likes it rough....... someone had to do it
@@StephenPreston like your mum? 😂
Yeah was gonna say this.
For the future, give the "watch" command a shot: watch nvidia-smi
Optionally with the -n flag to specify a refresh in seconds (defaults to 2). :)
Or nvidia-smi -l 1
I came to the comments to say the same thing. Figures that at least one other person already knew!
Yes, In this specific case you can do it with an extra flag on the Nvidia command itself, but it's much easier to just know how to use "watch" since it works across the board without having to know every flag of every individual command that you want to have a live updating readout for, or if that command even has a live readout to begin with.
@@davidg5898 Yep! I can't live without watch, but given the choice between one command or two commands piped together, I'll choose one every time.
came here to say this
I just learned about watch after concatenating files together for the first time! I was just thinking that too
"People that get mad at youtubers showing of multiple GPUs should look away"
The cool thing about CZcams is that you can learn anything for free. I learned basic PCB repair from Tech Cemetery and Louis Rossman videos to necromance a dead 1080 from ebay (the best I could afford) back to life. Why get mad, when you can get gud?
Thanks for opening my eyes to Tech Cemetery, didn't know it existed!
North ridge fix is also pretty based and redpilled
Me thinks if Jeff had a garage sale he would be able to add on a new wing to his house.
No yard sale, but a new wing is coming ;-)
he is in it for the long time investment. These cards are never going to lose value again
Neat little thing I found with my adventures in doing the same thing with virtual machines, you can passthrough your mic to a VM using VoiceMeeter and TCPStreamer which is awesome and great for games that need voice chat.
Love these cloud gaming server videos. Looking to do the same thing for my kids computers. I've been running a virtualized desktop and gaming computer for a few years now but separating the GPU for multiple VMs solve the issue of lack of PCIE slots or poor spacing of the slots in order to put enough GPUs in one system.
You'll love my next PC build then. Stay tuned 😉
@@CraftComputing which we could use rtx i have 2 3090s that could be use in a least 6 high end or 12 1080p machines 😕 im running unraid for now until the rtx30xx are supported 😥
Yes!! Saturday, A cold beer in my hand, And a new video from Jeff!! Things cant get better
Yep. Me too
Same for me except it's Dr pepper as I'm legally not allowed to drink 🤣
it's honestly astounding to watch this series after so much has already happened. thanks so much for sticking with it!
Any quantity of GPUs shown not mining (and not intended for that either) is good to see.
Excellent work as always. I am excited to see this in action.
Video's like this never interested me that much, but you have a way to make it interesting and made me wanna follow it till the end! :D Good job, keep it up!
Oof. That video/audio sync issue that happened around 11:00
Always happens with his face-cam
I love nvidia-smi. I use the -l flag to repeat it by a chosen number of seconds that I choose.
Alternatively, you can use the `watch` command to automatically refresh every second (configurable)
That card swap edit was flawless!
Love your channel, you do things that are interesting but no way am I putting my self through. Keep it up.
It gets pretty expensive when this guy gets bored lol
What are you, my wife?
😉
What is the impetus for having 12 vms? With 24gb of vram you could give 4 vms 6gb of vram which would really give a fantastic experience. Even 3gb of vram seems a little stingy. Are you ever running 12 at once?
great video. Smart thinking.. I would just ordered waterblocks and done custom water cooling... :D
The next video should be watercooling those cards!
I’m also pretty into virtualization and love what you are doing.
This is very interesting, would be cool to water cool the server and pipe the heat out of the case
I love your crazy quests. Now that have nearly conquered this quest, what is the next quest on the horizon?
Some editing jumping around errors at 11:09
Gotta love when your graphics cards cool themselves with a fanspeed mod
I had a Titan X Maxwell that ran extremely hot, to the point people made fun of it at lan parties because you could feel the heat when walking by my tower. I ended up water cooling the titan x which really helped keep temps in check, dropped it down 20+ degrees. Best thing about liquid cooling older gear is you can generally find water blocks for next to nothing.
I was expecting the previous video on this build to be in the description or in a pinned comment.
You can do `watch -n2 nvidia-smi` instead of manually checking it. Hope it helps
I just got 4 Tesla m40s. If those work I will sell my 4 1070s. I do machine learning and 3x memory would definitely help.
Nice! vGPU partitioning works on the M40 with vgpu-unlock, but you can only use 8GB of memory because of the nVidia profiles. I'm hoping to find a fix for that eventually.
@@CraftComputing what about the k80?
@@CraftComputing yeah I hooked them all up to my p9x79e ws motherboard and got a endless qr code cycle. I think it would stop at d4 and start over. All of them ended up working but I could only get one working at a time and it had to be in the last slot. Well, that's when a 1070 is occupying the first slot. I'll have to try one of my actual server boards with integrated graphics to see if anything will happen.
Wow! How are you splitting these up 4 ways!?
Would that work for a Ubuntu based install with occasional VM passthrough to VM?
Would this solution work on a 2070 Super or a 6900XT?
Jeff, 9:55 plz explain the covers for the internal fans? Is it a noise thing, or is it just aesthetics?
Just aesthetics. Who doesn't want jet engines in their $10k server?
Son of a gun, now I have to give it another try!
Hey so as others have said, remove the back half of the backplate for stacking cards like that. These backplates have no thermal pads so they're just cosmetic
Or repad and add pads to the backplate too
@@theangelofspace155 well you remove the rear half of the backplate to give clearance for the fan on the next card. they're 2 piece backplates. you can leave the front piece on and give it thermal pads too
“These cards were never designed to be stacked right on top of each other choking them” jeff, did you forget about SLI already? 😂
I Wonder if unraid will be able to partition a GPU...nicely done nonetheless you always amaze me with your knowledge.
vGPU partitioning doesn't work in Unraid.
I had to do something similar to my heavily modified Vega 64 that also reaches 1800mhz, but I had to plug it in to a noctua fan controller with a physical knob because Dell motherboards don't like fan control software.
Got my Tesla M40 Currently working on fitting a 980ti cooler to it. Sadly I'm still waiting for my power adapter. I will be first at making a Tesla M40 with a decent cooler!
I just received the first cooler for my Tesla M40s. Will be doing a video on that shortly.
Next, watercooling the Titan X Pascal(s)? Imagine how much faster the server could be if water-cooled!
You have enough space for some water block terminals on the top of those cards… might be worth a try, should get you in the low fifties of high forties.
This is a great channel.
Jeff, are you able to do a video comparing proxmox and hyper v in terms of performance ? I am not sure whether it is my settings, hyper v consistently using a lot of CPU when running the same game. Most of time, the CPU usage is consistently 100% in the 2019 core i5.
I will be looking into performance differences between hypervisors in the next month or so. Putting together a build/testbench to look at a number of different video cards and configurations.
@@CraftComputing dont forget unraid plz...!!!!
im surprised you didnt put a spacer between each gpu to flex them apart 1-3cm so they could breathe better
Jeff remove the back half of the backplate !
It's specifically designed to be removed when stacking the cards
I am also looking to build a similar rig but I wanted to use Moonlight to stream instead of Parsec. Do you have any experience of using Moonlight with such a rig?
Im a bit late, but how is the scalability of multiple gpus for cloud gaming? Does it have the same drawbacks of sli? I'm looking at doing 6 tesla m40's spread between 3-4 VMs.
Are you using VirtIO for the storage? Also, what software do you use for the thin clients?
At the moment, no, just standard IDE emulation. I'll be looking into block storage shortly.
Thin clients are connecting with Parsec.app, and I've also been experimenting with Sunshine/Moonlight open source nVidia Game Stream replacements.
@@CraftComputing Cool thanks, I use Moonlight. It's great.
Nice!!!!!
Jeff, I noticed your Noctua cooler was oriented so that the airflow doesnt move in the typical front to back config. Is this a limitation of the specific cooler or is this a implementation choice by you?
some coolers can only be mounted length-ways on some sockets. not sure if this is the case here, but it is plausible.
I saw your comment on LTTs video on GPU virtualization. Decided to come watch what you did with your implementation. If you're interested - I have two PNY Quadro RTX 6000 24GB that I could sell or lend if you'd like to upgrade your hardware or just demo another. Let me know I guess, I have had trouble finding a good home for these since they fell into my lap last fall.
sorry if this is a stupid question but what happens if you wanted to run a game of a 4k tv screen or something like flight sim, is it possible to run a sort of fake SLS style where you use two GPU for a virtual machine? i know sls is kind of defunct but there must be a moment where you want great picture and performance but still want everything running off the server? or am i barking up the wrong tree here?
I'm in a similar boat with threadripper pro and a pair of 2070s (blowers) and a 1650s in a supermicro chassis - the 1650s is running solo, while the 2070s are halved for 5 total machines. I've been hacking on this thing for the better part of a week, and I feel your pain lol.
Maybe the solution to your cooling problem could be buying tesla M40s instead. They are intended for use in servers. I'm not sure how much did you paid for these but in general M40s are quite cheap for how powerful they are. But I would most likely buy titan x too if I could, just because how beautiful they are ;)
While you can unlock vGPU for the M40s, vGPU profiles act as if there are only 8GB available, rather than 24GB. Speed would be around the same as the 1070s I replaced anyway.
by the sounds of it the cards might have done better with the shroud and fan removed so the server fans could just blow over the heatsink, maybe 3d print an abs server shroud to duct the airflow etc. depends on the chasis tho.
11:08 Jeff.exe has stopped working...
People have been commenting that you'll wear out the fans by running them full speed all the time. If you really are running them full speed, you might want to consider switching to an external fan controller which has temperature probes.
GPU out of stock :) You are from Mars :D
Traded for one, eBay sniped the other two, as they weren't listed as Pascal cards. ~$400 each.
11:09 we got a glitch in the matrix
I was getting a little antsy, since I can't build at a similar pace
Wonder if splitting up the gpu is possible in unraid?
I think the word you're looking for Fondant
wow the beer should be call a early 30 heart attack
You should replace all the pads front and back with thermal right pads. It will cool the board off which will keep the gpu itself cooler. Should knock at least 3-4c off. That was my experience at least.
Noted: Simply just plug in 12v power to fan to dust out my GPU! ✅
11:09 Nightmare Fuel...
currently looking to do something similar over 2x Titan V's, although ive heard volta is untested and may not work, have you played with volta for Vgpu by any chance?
I've been looking for some Volta cards, as a little birdie told me they're incredible for vGPU.
@@CraftComputing oh that sounds great recently got my hands on 2 of them for Free so i may try play with them and see what i can do!
Hot damn! I've been waiting for them to drop below $1500 for years!
@@CraftComputing same, my friend surprised me with them as a gift and it was a great upgrade from my titan xp
@@dylan-xq1qh Sounds like a hell of a friend. Hope you have a blast with them.
Is it possible to take care of the air from the gpu's? Probably doesn't matter.
Why not use watch to keep it updating automatically?
Take the blowers and shrouds off. The server airflow over the newly exposed heat spreaders might be enough.
Glad my titan black doesn't peak at 50% fan speed
Since it’s in a server why not remove the cooler all together and add server fans to the case?
.stl for your coaster??
Yes nice 3 cards
I wonder if someone makes or you can make a y cable with only the fan sense wire on it and plug that into the gpu just so it reports fan speed.
Take the fans and shrouds off, leave the heatsinks, and strap deltas to them
Is it bad for the fans to be running full tilt all the time?
Fans are cheap to replace if they fail. Titan Xp's are not.
As Jeff notes, they're inexpensive to replace. Their lifespan will be less, but they'll still last a long time (relatively speaking).
One unavoidable issue, aside from noise, is increased need to monitor them for dust intake/need for cleaning. If operating in a very dusty/particulate rich environment, the fans can get coated/filled with dust and particulates a lot faster. This will make them both louder and less effective.
So, depending on your environment, with a setup like this, you might want to blow them out with compressed air on a regular basis. How often depends on your environment, but maybe (I'm guessing here, as this is not something I've needed to do yet) once every 3 months?
(Now I'm curious how often Jeff has to do this; his servers are all semi-outdoors...ish. :P )
I would probably have tried to watercool them, but I understand that's a much more expensive, bulky and expensive solution.
My guy. I need some help. I want to buy a new gpu (2070 super, 3070) and i don't know witch one and where to get it cheap. (buget only allows for 450-600$)
Let it burn.... Your new motto
What are your thoughts on liquid metal?
It's great. But a major headache.
Too much double checking not to mention gravity can do you in in the long term for things that are vertical.
@@CraftComputing Understood. For me the biggest headache is actually spreading it. I don't know if you've tried this before, but if inject the liquid metal into the q-tip you can spread it like a paint brush instead of trying to spread the ball that typically forms.
Blowers are only impractical because of the inverse of why you wanted to use them, because consumer PCs have different airflow concerns than servers and even some workstations have
If it were my system id run titan zs but unfortunately theres no more driver support
Water cooling?
I think there's an L at the end of virtual ;) j/k nice job!
Uuuhhhh, let's make it liquid cooled. Well, you probably need more space, money and work. But it would help greatly.
I'm a little bit surprised that Nvidia thinks it's okey with no fan sensor, but I suppose it's prepared for water cooling.
Muta just came
why is this build so expensive? do you have a cheaper build?
This is awesome. Are you selling the 1070tis?
Lol. Just bought 3 of these before i even saw this
You gonna drink that beer or just have it be in the way?
Did u re paste the gpu’s? Those cards are getting kinda old
I did it on camera.....
@@CraftComputing 🤦♂️ maybe I should pay more attention. Yup right at 7:10 man I’m blind. My bad
What is the use case for this? Are you really hosting LAN parties with 12 people?
I've hosted a 6-player WAN party on my last CGS. Hoping to get 12 people on the new one. That's the nice thing about it, it works local or remote!
@@CraftComputing isn’t the quality downgraded based on your wan bandwidth? what kind of lag/quality are people experiencing remoting in? I’d like to see what your remote users experience vs your lab environment.
@@jengdes1gn you realize every game us remote? You play against other send and receive packages. Can you stream 4k video on you internet provider? Then you should be able to play remotely.
@@theangelofspace155 Ok, assuming everyone on both sides has a stable connection, do they all keep identical game settings in their VM? What if their connection is dropping frames and it’s laggy. Sure, the quality on his end might be nice, but if they can’t take advantage of it, he could use the 1070s and drop their resolution to 720p. Asking for the end user perspective, not the host.
Liquid cooled cloud gaming server!
idk why you didn't just take the shroud and fans off altogether and rig a fan to the back of all three cards
Now I'm asking myself, what is a German chocolate cake? As a German, I know some chocolate cakes... Let's check Wikipedia 🤷
Ah it has nothing to do with Germany en.m.wikipedia.org/wiki/German_chocolate_cake
I'd say Schwarzwälder Kirsch ;)
Hi Jeff
Give me that 1070 ti!
No, I don't think I will.
Are you sure he didn't send you canned pressed coffee and not an IPA?
Hi can the vgpu_unlock script work with the TITAN RTX ???
Yes!
@@CraftComputing OMG thank you !!!
Do you know where can I found a Titan X Pascal or rtx for cheap ????
heres what I use for looking at GPU temps .... watch -n 0.1 nvidia-smi
You took the cards down that far... just watercool them at this point. External radiators are a thing
That's a whole other bundle of maintenance and monitoring that should be done at your own risk, especially in a rack as he has his.