Trying out SR-IOV - Cloud Gaming Server Part 7
Vložit
- čas přidán 5. 11. 2020
- I decided to give AMD a shot at cloud gaming glory with a trio of FirePRO S7150x2 GPUs. Then this project just went all 2020 on me. But hey, at least I'm having fun.... right?
But first... What am I drinking???
Wichita Brewing Valley View Vanilla Porter. I was actually moderately impressed with this one, which is a first from this brewery. It's a pretty solid porter, but there is absolutely no vanilla in this.
And scotch. Lots of scotch. Highland Park Magnus Single Malt.
Links to items below may be affiliate links for which I may be compensated
Check out the AMD FirePRO S7150x2 on eBay: ebay.to/2U3id2g
Parts from this build:
Fractal Design Define 7: amzn.to/2Wx6ji3
EVGA SuperNOVA T2 1600W: amzn.to/3aUPRx7
Noctua NH-U12S TR4-SP3: amzn.to/2zTr7bT
Supermicro MBD-H11SSL-I-O Socket SP3: amzn.to/3f8H79P
Seagate FireCuda 510 1TB NVMe: amzn.to/2y9XrXu
Seagate IronWolf 110 1.92TB NAS SSD: amzn.to/2SmSBwP
Noctua NF-A14 PWM Fans: amzn.to/2SIwgtV
OLOy DDR4 32GB (2x16GB) 3600MHz C18 - amzn.to/38dTC30
(THIS WON'T WORK ON THE SUPERMICRO BOARD)
Find the parts I recommend on my Amazon store: www.amazon.com/shop/craftcomp...
Drivers for the AMD FirePRO S7150x2:
www.amd.com/en/support/previo...
Follow me on Twitter @CraftComputing
Support me on Patreon or Floatplane and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
www.floatplane.com/channel/Cr...
Music:
As I Figure by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
Cool Vibes by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
License: creativecommons.org/licenses/b... - Věda a technologie
"So I guess if watching an idiot fail at niche projects is kinda your thing"
I mean I'm an expert in idiocy and failing, it's right up my alley.
I can relate 😂
for 6 years I built up a race car using a Mitsubishi sedan on an engine platform that basically everyone has forgotten about, still an accomplishment, even though it cost so much more money than any other platform and doesn't have as much appeal
What an absolutely stunning cat!
You've done the impossible - you've cultivated a comments section that's wholesome and sympathetic!
Yourself is wholesome too.
what i like about this guy when he fails, he owns that shoot and keeps trying until he gets it right.
Listen. If I wanted to watch someone fail at niche projects, I'd put a mirror next to my desk.
😢
Look i want to watch someone fail at niche projects, so ditch the mirror and put a camera over there.
Listen. If I wanted to watch someone fail at niche projects, I'd watch ByteMyBits
Still waiting on that game invite... 🎮 Hopefully after part 8! Looking forward to it!
Why doesn't he copie your setup in getting a network gaming machine? Is it something iam missing?
@@aldervall Good afternoon.
Whereas T.T.'s pass-through deployment seeks to provide gpus to VMs at a ratio of one-to-one (roughly speaking).
Whereas C.C.'s pass-through deployment seeks to provide gpus to VMs at a ratio of one-to-many (roughly speaking).
Warmest regards, friends and neighbours.
I truly enjoyed the hell out of this. Feels like a real story, with multiple chapters each with it's own barriers to overcome.
It's one thing to see those LTT style project videos, so well crafted, with beginning plan, then "a hurdle for our heroes to overcome", which is always resolved immediately by the simple fact that they've gotten so large, they either have something else on hand thatll work/sponsor send them something, or they can break out the double-diamond black series executive ultra credit card knowing the video will easily cover the hw expense. I still watch them on occasion, but I miss that "in the trenches" feel of some of the older videos. Remember when whonnock died and they were desperately trying to recover their data? Or even further back, when they tried to build a PC in an aquarium? THAT was what I enjoyed - the level of detail is also a lot lower these days. Part of the reason I want Anthony in more videos - they bring him on whenever theres real nitty gritty to be done, and that's what so much of the tech youtube sphere is missing for me.
But now... now theres craft. Doing the kinds of things that make me feel just a little bit less nuts about my own (utterly ridiculous) projects, and presenting them much as any of us would experience were we to undertake the same task/project. I'm currently trying to plan out how to add this 8 bay sas backplane to my fractal case in a functional way so I can maintain sas hotswap functionality without killing cable management/moving to a bigger or louder chassis/etc. It just feels good to watch, reminding me I'm not alone in the tech nuthouse!
Wow, this hit the nail on the head in every way.
LTT has gotten so... mainstream (at the risk of sounding edgy).... they just want to appeal to the widest audience which means that videos of them messing around with enterprise hardware trying to make unRAID do something crazy are a lot less common.
Craft is scratching that itch.
I like LTT more these days, I think they have matured into something better.
The cat didn't expect this coming :)
I'm looking forward to seeing the next part of it 😂
"You know what this is." Kitty! "You know why we're here." To pet kitty!
Great job! I appreciate watching your 'failures', it's a valuable learning experience on my part - considering a significant number of the other virtualization videos out their that leverage a very specific path.
I love your content on a lot of your home lab stuff because it’s pretty much the normal experience; Its rare that anything works properly or as it should on the first try anyway.
These videos are like a journey, it’s entertaining IMO and I even learn a few things.
THANK YOU for saving me the time of trying this same thing lol. I was looking everywhere for a couple S7150 to use in a VM box due to their SRIOV support but saw your series and figured I'd wait to see how it went, considering how much trouble I was having with just a single GPU passthrough. You definitely saved me some time and money with this video!
Love this video. The honesty is so refreshing. Awesome channel. Keep at it please. ❤️
Hey man, you're only human and the fact that you undertook this project is impressive by itself. I wish you luck in your endeavors and I hope you can get your own personal Stadia server in your home.
Effort is highly appreciated sir.
It's not a failure, it's just another way of how NOT to make this work. A number of people now will not make the same mistakes. Sharing knowledge, both successes of failures, is NEVER a bad thing. Thank you for taking the time to do this!
Man Jeff I just love your stuff! Such a cool little niche you have created! I love the super computer builds and being an IT infrastructure engineer in my day job you take all the risks and I get to watch! ;)
This is my favourite series. I'm very keen to do something similar, can't wait till you figure it out. Best of luck!
Failure is Value! You learned from this build and this project. You will take that to the next iteration, and the next one will be amazing. I know you don't need words of wisdom, bit it is more impart to all of us who struggle with the lesser challenges that you may have already handled and learned from. It is nice to know that even guys with your skillset still have trouble from time to time. We all learn from videos like this. for that, I thank you!
Great that you published this video Geoff, it's really nice to see the 'real world' CZcamsr. Well done chap!
That server is the gift that keeps on giving to us the viewers of your channel.
I've been really enjoying this series!
I am so sad, i made a 3 gamer 1 tower workstation because your videos inspired me.
Asrock x399 taichi.
Threadripper 1950x oc 3.9ghz on water.
64gb of ddr4 3000mhz ram. (16gb/pc)
3 rx480.
Proxmox works like a charm with this configuratin.
I really hope you will get this thing to work. :)
Dude, I can relate to your enthusiasm as you go on the video(s) :D I also relate to your "mistakes were made" -part. Thankfully I always got my bad projects hardware sold forward.
So a big thanks for doing this video. Now we know what to look for and what to avoid.
love you vids and you honesty mate. keep it up . cheers !
Don't beat yourself up on this Jeff, at least you gave it a try and made it entertaining. Thanks for the video.
I've really enjoyed this series, I'll be sad to see it end.
You present the fails in such a good way, keep em up!
Hey Jeff, haven't really been keeping up with this series but today I got gpu pass-through working in centos with parsec. Parsec requires a monitor and on some of those firepro/quadro cards there should be a monitor/edid emulation feature. In my case I have a regular consumer card, an RX 560, so I needed a headless dongle. If you're interested my full process let me know.
You put so much effort into this, fair play you! 👍👏
I'm convinced that having a CZcams channel allows you to buy these computer parts and says it's for work. Here's to supporting this channel!
Well, yes and no. Writing things off as a work-expense is not quite that simple.
For starters, you have to be able to document that the purchase is for work *ONLY* , so having it as a home game server kinda pokes a few holes in that argument...
Not to mention, some purchases aren't completely covered, so you only get part of the cost written off.
But yeah, here is to supporting Jeff and the channel. 👍
Nice to see someone else with that track ball.
Don't be sad, you did good. It is a pleasure to watch the steps.
I get excited when I see there is a next installment in this project. I want to build something for me and my kids to enjoy gaming throughout the house. I've bought the Tesla K10, lol.
Love this series and cant wait
If at first you don't succeed... Worth it, just to learn what *doesn't* work. Thanks as always.
Nice one mate, don’t you dare give up on us. I’m waiting to see you crack this one.
Oh man I was excited for this thing to work!! Please do a part 8 if you can think of another way. By the way that cat is quite handsome
The whole reason I started watching your videos was a way to vicariously live out my mad scientist fantasies, oh and you've got great taste in beer!Maybe someday I'll start a channel and we can have a good laugh at each other's failures!
wow, project which I tried to pursuit long time ago, but failed because of lack of budget ... so sad :(. However, it's too cool to have you here :).
Waiting for your success!
this is great content just for the fact you highlight how even seasoned operators can make mistakes - you really have to do your homework and mistakes/oversights happen to the best of us
An honest video - great to see :)
Love the RTX ON part! That nvidia feature is really awesome!
Rambo deserves hazard pets for that one.
Love your videos, I'm not smart enough to do this on my own (and definitely don't have the money!) so i'm enjoying following your journey through this mad project :)
best series of videos of the life
Live and learn. Good luck.
Think of the feeling of success you will have when you get this machine right!
Waiting on part 8!!
I love your server videos,. But your cat! So adorable!!
Good deal on some oLoy ram coming to amazon warehouse soon. Thanks Jeff!
Just got around to watching the whole playlist tonight, and this was quite the journey. I haven't had the opportunity to try out sr-iov yet, but from what I've seen it can be a bear when it doesn't work. If I'm going to do it I'll be going the standard route of having a monitor/keyboard/mouse direct connected for gaming. Just need a second GPU to run my host OS as it'll probably be a desktop Linux box.
i knew it wasn't going to end well when you were relying on luck being on your side. Well done for showing all the failures this is much more like real world IT compared to some of the other channels!
At least you tried a very nice project !!
VCE (hardware encode) on the S7150 only works if you pass through the entire GPU. A similar setup is used on Azure NVv4 instances that can pass down to 1/16th of a Radeon Instinct Mi25 card. As can be seen in your video, they call this setup MxGPU. The MxGPU host drivers on the hypervisor throw away VCE completely (can be seen in the Linux driver source) and don't bother partitioning and giving individual VMs partial access.
I suspect you passed through only 1/16th or 1/8th of your GPU and hence got the expected performance.
Ah yes Jeff. Seeing people install drivers are my favorite past time! 😩😂 (Kidding lol but good job on the video! Hopefully when you come back to it you’ll have better luck.”
6:56 Poor Rambo...
Definitely dropping the sympathy like. I salute your effort sir.
I get just as excited to see a new cloud gaming video as I do a new Mandalorian season
Damn I wish you would have used Proxmox! Also I had no idea that these cards lacked hardware video encoding, thanks for testing all of this!
No one fails at so much niche as me. Damned proud of it.
At least you tried :) A nice car on the table
Can't wait for part 8!
I was soooo Close to try that with one 7150x2 in my server.
Thx for preventing this failure. :D
Stay tuned.........
"you know what this is, you know why we are here". Entertainment. And beer recommendations. Keep up the good work :D
2:30 Doh. The color of the LED is wrong. Should be orange. Like the network cable
Big oof, hope you can recover some costs. That’s for your content!!
Steam has options for hardware encoding. You might want to use that either with another thin client, a phone/tablet, or a real Steam Link device.
PS: the AMD drivers offer desktop streaming to phones, maybe other devices too. I tested it even with Stem VR games over to a Google Cardboard compatible setup.
Blower fans so loud the cat despawned
Hahaha the cat was like wtf is this wizardry human
At least you got the best showcase for RTX voice.
Hey, I work for Tesla and I would LOVE to see you review the Teslaquila!
Well, you still got a like from me, because, well, seeing how someone fails is a good lesson on what not to try. I'm a retired systems integration engineer that specialized in fault tolerant servers, and dealt with sales people in the company selling 10's to 100's of thousands of dollars of equipment promising things they should have spoke to the engineers about first.
I setup a smaller version of this a few years ago for my daughter to learn her way around a computer.
I have unraid running on my server mainly for plex, but I have a dual socket 1366 board with 2x x5670's and 24gb ram. I put in a gtx 1050ti and created a windows 10 vm through unraids built in kvm. I was able to pass though the 1050ti after adding the cards vbios to the VMs config. Once that was done, I used a raspberry pi 3b with moonlight embedded installed and set it to automatically run and connect to that VM when it boots. It worked perfectly fine. just had a keyboard, mouse, and monitor hooked up to the pi in the living room, and was able to network stream the mstmc.exe through nvidias gamestream to the vm.
the only downside to doing it this way is you have to wait for the vm to be booted to windows and logged in before you can connect, so I had to use windows netplwiz and set the account to auto log in at boot. otherwise the pi would not be able to connect to the machine because the os was not yet unlocked. For a single user per vm it worked fantastic though. I was able to pin specific cores and threads per vm and pass through a whole disk to the VM.
maybe worth looking into doing something like this before you give up on the idea
Q. How do you keep you cases clean with fluffy cats around, A. High velocity fans keep cats away!!. In all seriousness I love these little niche projects, keep it up, it'll be all the more satisfying to see it work
i have never called you an idiot but what you learn from this is more valuable than you thinking you failed you didnt fail at all you know what not to do next time have fun :) thats what its all about :)
Great series!
I wonder if the CPU had a little bit more single threaded power (higher clockspeed for each of the 6 cores per VM) if the performance would have been better with the SP7150x2. I'm quite interested - I have a thread ripper 1920x, while only 12c/24t the clockspeed is a bit higher (and potentially could hit 4.0GHz).
I have what I lovingly refer to an the umbilical cord of video and USB cabling run about 25ft from my server to what serves as a second workstation for me. Would love to drop the cabling but without the headaches :)
Clicked for the kitty, subbed because you're enjoyable!
I'm right there with you man
I know exactly what you're going through and I'm going through similar issues with my cloud gaming server.
What was the issue with the GRID K2 cards? I setup a similar system using a 3rd gen Ryzen 7 system on esxi and had a very playable setup using Blast or Pcoip over horizon view
I have been toying with the idea of virtualizing my gaming rig, It is already in a rack case but has no need to only serve that task. I would be passing through an entire 1080ti, But it would be really cool to get a single GPU I can share between 3 different VM. I'm bummed you are putting this on the back burner. BUT, I totally understand why. I look forward too when you bring it back.
I so had my fingers crossed. Just don't leave us hang'n.
Don't worry i get into niche ideas very similar also with mixed results 😉
Nice video, dude. Hard to watch a fail, but I've spent the last 3 nights figuring out a Geforce 2 GTS doesn't work, so I get it.
Aw man, I totally feel your pain when going experimental!.
Also, if they don't have hardware encode, how are you supposed to use these cards for vdi?
To the viewer it’s just as interesting when it fails, I’ve learnt lots from this series. Please spend more time showing and explaining the issues - don’t just skip over them so quickly. And obviously you can’t give up on this!!
Yeah, was gona recoment these ones on the previous video precisely due to SR-IOV hehe
Ooops, HW ENC, you need different models hahahaha
Made it work every boot! use the guide on the eevblog-thread (page60) to have proxmox spoof the 690-gpus to k5000.
Nice! I'm well acquainted with the thread. I'll give it a look 😊
Realized now that i posted this comment in the wrong video, i thought i was on the latest. sorry about that🙂
Oooof you saved me some money, I almost got a pair of S7150x2 thinking it's not going to be thaaaat hard.
ended up going the save way buying an old 1080 for my one client that needed it, until I actually start having more clients I don't think I'll touch grid/grad (nvidia/amd) server gpu
Thank you for saving us several thousand dollars! XD it's a shame you couldn't make it work, for some reason Nvidia, Microsoft and Google have so well kept the secret of streaming games
Liked for the beast of a cat and not for the fails
14:45 is a reenactment of a scene from the movie First Blood, Rambo getting cleaned up.
Ah yes, the series that refuses to end for some reason.
I love how kitty looks disapprovingly at the 128 Gb of ram 🤣🤣🤣🤣
I got asked to look into sr-iov networking (which is a colossal pain in the ass!) for use with a voip application at work. At the same time, I benchmarked its network performance against plain old LXC containers using openvswitch networking in proxmox.
There was just a 2% difference. I figured that given the additional complexity, the sr-iov networking wouldn't be greenlit.
WRONG.
Next week I'm building a proxmox cluster with sr-iov networking. Time to take up day drinking.
Did that benchmark you did show any latency differences? Id be curious to see if your work wants the smallest latency it can get, especially in Voip setups.
I was semi successful with 3 r9 295x2s and a huananzhi x99 dual with 2 E5 2678v3s both with 3.3ghz all core turbos and 128gb ddr3 ecc. Issue was trying to get linux to recognize the two gpus through the PLX chip, which it did in the end, but only with a 20% performance cut to the secondary card on each gpu. So, what I ended up doing is using 3 "4 core" systems and 3 " 3 core" systems for them secondary gpus and made them a lower performance tier the 3 core systems got 12gb of ram, and the 4 core got 20, the rest being used for OS and hyperfast drive caching. Hardest part was making sure the memory was correctly connected in VMs so the cpus didnt have to ask eachothers memory controllers for access to one another for each VM (for the obvious stutter increase that is caused by it)
It's not failing, it's learning.
More cat cameos please! 😻
Whenever he cooperates....
Same gpu I was gonna buy
keep making these videos