Proxmox GPU Virtualization Tutorial with Custom Profiles thanks to vGPU_Unlock-RS
Vložit
- čas přidán 20. 03. 2022
- Visit Keeps.com/Craft to get 50% off your order!
If you have a compatible nVidia graphics card, you can build your own Cloud Gaming Server in as little as 90 minutes! This video is a complete step-by-step tutorial for installing Proxmox, nVidia GRID, and defining custom GPU profiles for allocated memory and supported features.
So why am I posting this updated tutorial less than a year later? Mainly, this update fixes a number of problems with driver compatibility, as well as memory allocation in Maxwell GPUs. Previously, only 8GB was allowed to be used in total on Maxwell-based cards. Now GPUs like the Tesla M40s, Titan X, etc, can use their full 12GB/24GB to divide between vGPU profiles. The installation is easier, more stable, and customizable depending on your needs.
But first... What am I drinking???
From New Image Brewing comes the Can You Milk an Oat... an Oat Cream IPA.. This one caught me a bit by surprise. Very thick and creamy, but with a crisp finish. This is an IPA that doesn't know if it's supposed to be a Winter or Summer drink, but manages to pull both of them off.
Links to items below may be affiliate links for which I may be compensated
Build your own sub-$1,000 Cloud Gaming Server!
Intel Xeon E5-2690 v3 12-Core: ebay.us/ewszQG
Jingsha/Bewinner x99 ATX Motherboard: amzn.to/36lDlKm
OLOy DDR4 32GB 3600MHz: amzn.to/3tdfhlS
16GB DDR4 ECC-REG (Buy 4x): ebay.us/lBdn7x
Tesla M40 12GB: ebay.us/2OD6VL or ebay.us/40kdCU
Tesla M40 24GB: ebay.us/FW9GWQ
be quiet! Pure Base 600: amzn.to/34K6JJO
be quiet! Shadow Rock 3: amzn.to/3I99wde
be quiet! Silent Wings 3 140mm: amzn.to/3isPosl
Fractal Design Ion Gold 850W: amzn.to/3q514FG
Silicon Power 1TB NVMe: amzn.to/3IdVIOH
Delta BFB1012EH Blower Fan: amzn.to/3w4XRKa
nVidia Tesla Cooling Shroud: www.thingiverse.com/thing:474...
My previous Tesla M40 Benchmark video: • $220 for Titan X Perfo...
Instructions and Source links
Step by Step Instructions - drive.google.com/drive/folder...
vgpu_unlock Github: github.com/DualCoder/vgpu_unlock
vgpu_unlock-RS Github: github.com/mbilker/vgpu_unloc...
Register for a 90-Day Virtual Workstation Trial at nvid.nvidia.com
vgpu_unlock Discord server: / discord
Game Streaming Clients (Comparison video soon!)
Parsec
parsec.app/
Sunshine Streaming Server
github.com/loki-47-6F-64/suns...
Moonlight Streaming Client
moonlight-stream.org/
Head on over to craftcomputing.store and grab yourself an official Pint Glass
Follow me on Twitter @CraftComputing
Support me on Patreon or Floatplane and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
www.floatplane.com/channel/Cr... - Věda a technologie
Man, your content has just been getting better and better, once I got used to the brew aspects of your videos I really dig your interests. I have followed a few of your setups successfully for fun and then just sold the parts once I was able to do the project. Thanks for all the fun ideas!
Very cool tutorial! This is one of the most accurate guides available to date. I always were interested in this technology and how this is going to evolve for using in Enterprise environments. Will certainly keep following you. Thanks Jeff :)
Awesome tutorial Jeff! I don't have a practical use for this type of vGPU setup, but I like watching and learning what can be done. It's amazing what you can do with somewhat new hardware.
Milk that cow, I love seeing it! I've been wanting to do something similar to this for a long time, but I've just been too busy \ lazy to do so.
Your kick off made me laugh like crazy. Great job. Further update. I was the guy who quickly said hello at CES 2020. Sadly I had to run to a meeting but I'm glad to see that you've updated your channel and are doing so well!
I greatly appreciate your clarification of what a gaming server is, the benefits of having one, and how to build it! Excellent video! Thanks Jeff!
OMG. Absolutely amazing. Thank you so much for taking the time to go through this. I am in the process of sorting out a setup like this for a plex server for encoding. This is exactly what I was looking for. Thanks so much. :)
LOVE THIS JEFF! For those struggling with the Nvidia profiles, make sure you choose a profile with a Q beside the sizes. There are several letters, but I found this buried in the vgpu issues section and it was the solution!
Fabulous video. Probably the best Proxmox / virtualisation video I’ve ever seen. Impressed. Subscribed!
Awesome, can't wait for next video for more streaming options.
Amazing tutorial Jeff, love it. You, sir, have inspired me to make my own gaming server. Just wondering when you were planning on making that video about different streaming programs. Was it going to be about different programs other than Parsec? Hopefully, the video drops soon. Would love to get started on my project as soon as I move and have my computer room all set up.
Keep them coming Jeff! I love all these proxmox/tesla tutorials... Is there any chance you are planning on doing a guide on setting up a Linux guest to use the k80/M40 teslas with accelerated graphics? I had things partially working on Pop!OS VM over VNC with a RX570 passedthrough(video out), but i can't figure out how i did it now (much less with my K80).
This video is GOLD! Thanks for taking the time to cover this so thoroughly.
Looking to render with an RTX3060, in a MacOS VM...I think you see where I'm going with this...
KEEP GOING! - This is something I've wanted to do for so long, and I'm knee deep in my own build. ATM I would make a suggestion but it seems you've covered all the bases. This is not an easy thing todo, just the right combination of software and hardware must come together. Oh - How about a demo with Teradici PCoIP? - I'm not sure if it's free but I know it can do 3D graphics.
Hey man, thank you so much for doing this. I discovered your channel while trying to do this on my server; for anyone wondering, I was able to follow these directions on Proxmox 7.4 Kernel 5.15.102 & dual Tesla P40s
hmmm im using 7.4-3 and getting errors with the dependencies
Never stop the virtual machine videos, next you should do bhyve on FreeBSD with gpu passthrough and hardware acceleration.
Definitely doing this sometime. Thanks Jeff!
Great tutorial - Been using Proxmox for a while now as my daily rig is a linux box. But I still want to game so I use use steam play and make sure the windows VM is on the same subnet. works a treat
Thanks, Jeff, for all your work on this! Do you know if any of this has become a little easier in Proxmox V7.3 or is this tutorial stil how this is done right?
I'm puzzled. I thought this was gonna walk down the Hyper-V route. That being said, as always - ace video, nice information.
Thank you for the detailed tutorial! I'm looking forward to building something similar (but probably lower-spec) for my two kids. In a follow-up video can you try out some popular games and tell us if there are issues with DRM systems that prevent the game from running on a virtual host? Or are there techniques to hide the fact that it's running on a virtual host? Also - what's the best way to pass through a microphone on the client to the VM so that you can talk while playing? Thanks again!
It's usually not the DRM but rather anti-cheat being hit or miss depending on title. For microphones you probably can just do a USB passthrough by the Proxmox Web GUI (or via LibVirt if you're using something else).
Caught that Archer reference. Nice touch.
Looking forward for the next video, the reason I don't like this setup is I don't want to have to connect through parsec to the same PC i'm using.
Could you also explore DDA (Discrete Device Assignment) in hyper-v? It's supposed to only be available on windows server for GPUs but maybe there's a way to trick it on regular windows
With regards to keeping the M40's cool @Craft Computing. I have heard people say that the Kraken G12 kit works to mount one of the Kraken AIO coolers to the GPU die
As a mostly retired techie I found that a great description of something I'll never need to do but like to know that it can be done. But I'd be very worried about a beer that is that strong but doesn't tastes like it isn't. I've been caught out too many times by these.
Finally this video is what I needed Since I have the Tesla M40 24GB model
Only for the fact that it's in a Dell PowerEdge R720
@@christopherspivey9117 I have got a R720 and does it all work out for you?
@@repairman2be250 Yes it does provided that you have the proper EPS12V to PCI-E
You will have to bridge both the 200W PCIE Power cards in to 1 GPU since the GPU is way more than 200W
@@christopherspivey9117 Thank you for your reply. Will keep an eye on for a M40 card.
Can’t wait for this to become a thing in unraid. I’m thinking a Tesla M40 might become my go to for a steam link/moonlight box.
does anyone else do this? say im installing something new or setting up software. i can know fully how to do the job, but ill still usually check a tutorial just as a sanity check that i do know how. surely this isnt that odd. jeffs vids reminded me that i did it. we do alot of the samethings lol. but always glad to see your new video, keep it up :)
thanks bud that's a massive effort! hats off to you good sir :)
I'd like a step by step disaster recovery, of-course this would need to be "simulated" by Jeff, but this would really teach us some very important basics! Creating servers is fun and all, but disaster recovery won't be if you don't know what you're doing :)
Basic explanation of scrubbing and the different pool states would be really helpful.
Disaster recovery = backup proxmox disk before doing any changes.
@4:17 you should look into using Ansible to setup desired state configuration. I'm just doing this for Proxmox as I'm installing my 'X'th (where X > too many) machine and it's like groundhog day will all the "tweaks" needed on a base install.
Lkg fwd to using this "latest and greatest" version 👍
Thank you Jeff.
Very cool! This is something I have been wanting to do, I'm wondering whether this will be possible with the latest 12gen Xe integrated GPU or even the coming Intel graphics card for a remote desktop for youtube and streaming. Would you be interested in digging into that?
I am using moonlight works perfectly fine. No need for parsec with newer gpus.
Thank you for the video, Jeff! How about Kepler based cards? I've got K80.
I love this content ebaut gpu virtualization Thanks for doing it
Please don't stop making these videos
Pretty cool 😎. I have installed proxmox before as a test but forgot to partition the hard drive to give me space for my Mac VM. I was doing this on a older intel msi laptop with a ivy bridge cpu and 16GB of ram. 🤣😂 appreciate the video.
You should perhaps re-visit Proxmox with some more powerful hardware. VMs really need SSD storage it's worth noting.
@@wayland7150 appreciate the advice. I was considering that.
Amazing video tutorial.
I really want one of those Tesla cards but they much more expensive here in UK (around used 1080Ti prices). There are loads of cheaper Grid K2 8GB cards, i don't know if those would work with this system. Need to research that.
Great job! ..please make a tutorial for linux and macos! ..It would be interesting to know if it works with those OS
What are your thoughts on virtualizing a CAD workstation? I don't desire remote access for it either but dedicated hardware (graphics) is a must.
Having said that I opted for a z840 workstation with 2x xeons, 28, 2 x RTX A4000, 128 GB Ram and I am happy with the duct inside pushing a lot of air where it belongs for perfect cooling.
This is pretty usefull for thoose who gaming who can find tesla cards for cheap and want to game. Thank you for the guides
This can be done with any Maxwell, Pascal, Volta or Turing based nVidia cards. They don't have to be Teslas.
@@CraftComputing yeah fair point but how i localy looked a maxwell based tesla was 1/3 the price of a GTX 970, yet had the perfromance of the GTX 980, honestly i would of gave it a crack with this guide now but the gpu was sold and i got a R9 380x 4gb.
And if my internet was actually good would try eitherway. But in speedtest i got 20pin, 30mb up and 30mb down. Pretty bad
@@dexsters5643 I've heard the game streaming only uses about 10Mbit/sec. It's worth trying if you want to play around.
@@eDoc2020 i dont have a nvidia videocard, i had a choice GTX 750ti 2gb vor 75€ or R9 380X 4gb for 100€ i took the R9.
@@dexsters5643 You can still do streaming with other cards, the difference is you can't split it across VMs.
Would you mind doing a power draw/power usage update for us? I'm curious about comparing a setup like this to having 4 low-spec gaming pcs (comparatively) and weighing the power usage for gaming. For me, I'm concerned with the increasing cost of electricity, thanks to Idaho Power being a monopoly...
Love the guides!
I'd love to see an AMD GPU variant of this, especially since we're talking about older GPUs for something like this. AMD GPUs also wouldn't require dealing with enterprise licenses nor NVidia's terrible linux drivers.
Commenting for the algorithm. I agree with you! In his past videos, it makes it more difficult to follow when he doesn't show the AMD technique.
There is no AMD method for this that works with consumer cards. I'd love it if there was.
AMD hasn't released a vgpu capable gpu in many years, so the Linux solution just isn't there, but his videos on using windows hyperV to share a GPU (Google easy gpu-pv) does work for AMD cards.
@@CraftComputing That's disappointing, but good, to hear. I haven't done any digging myself, so it's good to know someone else has. Thanks for your content, and I appreciate it!
@@CraftComputing I've managed to passthrough a RX580 (2nd slot) in Hyper-V in windows 11 pro. The problem I encountered was that the pci-e DDA is not supported in my version of windows. I did some more digging and found that I need windows server OS in order to do what I wanted to do. As soon as I am feeling brave I will be attempting a 2 gamers 1 pc using hyper-v with my current system.
One thing I have found. If your install is EFI with zfs the grub change for IOMMU will not work. You need to instead create /etc/kernel/cmdline and put what cat /proc/cmdline returns with the iommu turn on command you need. Then run pve-efiboot-tool refresh and reboot. Being my system is a multi-function setup zfs was part of it which added a small twist.
You rock and so does proxmox
thanks Jeff, great vid
Great job! Super installation. The author is well done. I just don’t understand, I have a task to also assemble a machine for cloud gaming, but there are 1060 6GB and 1070 8GB cards available, could I run them according to this guide?
All of this video went over my head.
"or a hashtag....." i felt that haha
hey Jeff, cool tutorial. i'm building one of those on dell t5610 with 2x p100 card. i have 126gb ram on it but i kind of curious can you use RAMDisk as a boot drive for virtual machine? if it's can i want to upgrade to 256gb RAM then partition it to RAMDisk to the VMs
Now that there are custom profiles I really want to get back into this (merged drivers and Looking Glass)
But the newer Linux kernel is so goooood, with futex2 and lots of useful goodies for my hardware, and 5.17 just released with AMD-P state driver I need and 5.18 is going into feature freeze with better drivers for my mother. I don't think I can go back to 5.12 again to get the old Nvidia drivers :(
great tutorial! I've been waiting for this, I only have on question is how does this impact network traffic per VM?
Streaming to a client requires ~10Mbps upload at 1080p.
@@CraftComputing That sounds like it would be OK over WiFi!
@@wayland7150 yeh, i've streamed VMs to crappy devices over wifi just fine. main problem with wifi is latency imo; you need decent 5ghz or better wifi.
Thank you so much for "milking this cow", please don't stop! 😂
Also, would it be possible to do a similar thing on top of TrueNAS SCALE instead of Proxmox?
Awesome stuff, Jeff. Although there is one issue here...it's too late for me to prevent male pattern baldness with Keeps :/
There's always that Locks of Love sponsorship
1. How should I split a 11GB 1080ti? 2. Does this support non-windows clients? Like Androidx86 or ChromiumOS, or just straight up linux. 3. Can you configure mirroring on both display instead of disabling one?
already weird! After you have explained it in great detail and detail, other youtube channels will make the same video almost in a timely manner...... 😂
very interested to see a raspberry pi running a game at 1080p streaming from this beast.
@Craft Computing, 8 months ago you made a video on how to create 2 VM using Hyper-V including shared GPU. Now im confused, should I do it on Proxmox or on Hyper-V? I would like to access them remotely too. Thank you!
Hey Jeff, I know you love the data center cards. By any chance have you attempted gpup on HyperV with any AMD GPUs?
Just wanted to know how the experience is.
I know this probably has been brought up but could the tesla m40's be water cooled? I know it would bring the price up a bit but it would be quieter.
should it be bullseye (Debian 11) instead of buster (Debian 10) at 4:12?
curious why the framerate limiter wasn't taking effect during your example running Heaven? was the profile not setup or applied correctly? I would have expected to see 60fps from heaven, and less than 100% GPU utilization.
Have you try to watercool the Teslas? Maybe Kraken G12 could work well to use an Asetek AIO
I am getting close to building a new rig, and I am wondering if I can set up my workstation with a vm for video editing I can run remotely. The second question is what would be the best way to process footage when I am editing out of state? I guess the best way is to set up a NAS that I need to upload to?
And I guess along side that I am wondering if perhaps it isn't just better to use something like parsec?
Hi Thnaks for this video as well as all the others I have seen by you! Very informative and very helpful! Got this working on Proxmox 8 with the most current grid drivers. Windows runs great or good enuff using part of my Tesla p4. Wanted to alo use this with a Linux VM running Jellyfin and use the other half for Hardware Transcoding but whenever I attempt to do this the Nvidia grid driver breaks and I have to reboot the system. There are other guest drivers that came witj tje 535.104.06 package maybe Im not using the right one? been pulling my hair out over this one for weeks! any help would be appreciated! thanks!
nice, did you get this to work for linux guest vms? also, what proxmox 8.X version (kernel) did you get this working with? thanks
thanks for video, have a side question about passthrough gpu, mounted 3 gpu on my workstation, but can only use 2nd and 3rd, proxmox keep 1st in use and it gives error 43 when i try to use it.
when i remove 1st gpu the error appear in 2nd gpu ...
just little point, maybe i m wrong about vfio_iommu_type1, from version 6.4 or 6.5 it included by default in loaded models, no need to add it in modules file, except if there some other reasons.
when i check list of modules, i found it there
I do believe we're in desperate need of a refresh. It's 2024 and Proxmox 8 (with the dropped support of Pascal and older cards on 17.0 and later) is its own beast in and of itself. Getting driver 16.5 to work on 8.2.2 with Kernel 6.8.4-3 is a total nightmare.
if you use a card with a hdmi out can you use that for the games X4 love your work
Have you thought about water cooling? Apparently an NZXT Kraken G12 bracket will adapt M40s to an AIO (according to one random forum poster)
Stay tuned.... 😉
@@CraftComputing Loved the video, the G12 bracket also works perfectly! One thing I noticed, it looks like the value you have listed for doing an override to create an 8GB vGPU seems to cause an assertion error that prevents the VM from starting. After changing the value from 7874106708 to 8053063680, it started up normally. I think it's because the first value is not divisible by 1024?
Friend, I have a question. I want to play around with an Nvidia P4 for graphics processing with OpenCV, without modifying the clock or using it as a video card. I won't be modifying the standard clock. Do I still need to add an additional cooler? Or can I use its passive dissipation without adding any cooler?
Hello thank you for your great content, Is it possible to use gaming GPU such as RTX 3070 TI for a VGPU as it has Ampere architecture similar to RTX A5000 and A5500 which is in the list??
If it works with this setup then, I would recommend Moonlight over Parsec. Enable nvidia gamestream on the server then run moonlight on the client side. I get lower latency and better picture quality with moonlight over parsec.
I've actually been using Sunshine, an open-source alternative to nVidia Gamestream. Works on Linux VMs too ;-)
@@CraftComputing Nice, never heard of Sunshine I will have to look into. (or you could make a video on it, wink wink)
Next week (wink wink)
I had a P4 laying around and, as of last week, I have a brand-spanking-new Dell PowerEdge R740 to put the card in. I am wondering if you know of a tutorial to get this configuration running for virtual desktops running under ESXi. Thanks, Jeff. - Jeff N.
awsome tutorial! TNX!!!!
So, what do you do once your trial account with Nvidia ends? Keep making different accounts?
Tried using this awesome tutorial for Proxmox 7.2-5 and it doesn't work unfortunately. Trying to give my 1080ti a second life. Tonight i try a clean install of 7.1
Can you make a new video adding some information, e.g I dont want to use parsec (or at least not only maybe have it being used when I want to stream to an android TV box on my main TV and hook up an xbox controller on it) I want to use two physical displays connected on the GPU's physical HDMI/Display ports.
Also could you add some more info about how to fine tune proxmox for this scenario e.g how to auto run on the physical displays both VMs on boot or other performance settings for the rest of the hardware (e.g what about audio I mean how does it use the resources and split it to two computers? ) .
I tried this with a Quadro K2000 I had lying around, didn't expect it to work with that card but strangely enough it did.
this is cool. i have one too. i'll give it a swing
I may be able to get a K2000 so this is interesting. It seems similar to the GRID K1 which is the oldest card supported by vGPU. I wonder why support for this generation isn't officially supported by the script. Is it just an oversight?
Thanks a lot for another great guide, worked like a charm with Parsec!!!...
But I am having a weird issue when I try to use Moonlight (I rather use that one to stream to the phone)... It errors out during RTSP handshake, giving me an error saying Ports need to be forwarded (I am in my LAN!!!).... This Happens only when using the VGPUS... removed the mdevs, passed the GPU directly to the same VMs, doing that, with the full GPU passed through, Moonlight works no prob... Added the mdevs again, trying with different newer and older drivers. With Parsec, not a single issue... Moonlight keeps giving me Port Forwarding errors when I am in the same network!
I could not make Moonlight work with Vgpus...Waiting for your video on the streaming clients... maybe you had the same issue, and hoping you give a solution. :)
Your videos taught me a lot, thanks a lot once again!
I like the virtualized gpu series and I can think of quite a few more passive entries. 1. Quieter fan, 2. Cooler swap 3. Truenas scale 4. Tesla m60 5. Affordable dual Tesla rack system. 6. Tesla p4, Tesla p40?
Oh yeah, also use for workstation applications instead of games.
Is there a water cooling kit that would work with those Tesla M40?
@@JoshuaBoyd I am into that as well. I don't play games.
I'm still not sure why patch argument for nvidia driver installer is not used to patch all the necessary lines with a patch file
Something I think needs to be noted EVERY time we talk about device passthrough...
Memory Ballooning is NOT SUPPORTED when passing through devices. It WILL cause craziness.
...
I learned this the hard way after several weeks of troubleshooting. Someone just happened to mention it like it was a 'well known' problem. Which it is clearly not.
It is listed somewhere in the fine print, but I haven't seen it mentioned pretty much anywhere else.
Thank you. I passed through an HBA to TrueNAS but fortunately was advised not to use ballooning with TrueNAS.
Is there any api calls that you can make to start and stop VMs in proxmox so you can turn on and off VMs from the remote device?
Jeff I have a question. This method is for spliting a gpu for multiple VM's but If I want to get 1 physical gpu for every VM, I have to do the same thing like in this video or is easier to passthrough and no modifications needed?. I am a beginner in this world so I still have a ton of things to learn about VM's and stuff.
I second this
Passing through 1 GPU per VM is easier if you have the money. You have to do most of the stuff before the first reboot, but most of the stuff after the reboot is specific to this solution. There are guides on the Proxmox Wiki for GPU passthrough.
Quick system reboot? Not on this Dell server we're talking around 7mins time to post/boot with a SSD😅
Is it possible to use this for headless linux VMs that would require GPU work loads like LLM or Stable Diffusion? Think that will be in big demand going forward.
I picked up a Tesla P4. Which, if any, of the unlock elements can I skip when setting up?
"If your an IT admin worth it's weight in scotch" is the best quote I have heard in a while
awesome thanks for the tutorial
The slight cringe cut and subtle tone of disgust when you said "hash tag" was amusing lol
One question: I have a 2090 with 24 gb; when i'm not using a second OS on proxmox (lets say Mac OS), and I only want to game or edit videoson my main OS (Windows), *can I somehow easily "rejoin" the 2 vGPUs i create into a single one?* (so that i can fully use the whole 24 gb)
I noticed you advised to split the CPU's resources evenly among the VMs, but I was thinking that for best results, Proxmox needs a core or two, doesn't it? Or would it just use whatever scraps of CPU were left over from the VMs?
from my experience it's not a problem, unless you plan to hit 100% from all the vms simultaneusly. it depends from the use case of course. I have a Zimaboard running much more than it's supposed to run with only 4 cores.
Hey jeff, what about the Tesla k80's are they still not supported and not useable?
I thought you'd need a grid licensing server in order to allow for the guest to work or using the license bypass script, didn't know that you could mask the card into a different card in order to install drivers
So is there no way to assign a physical display output port to a vgpu so that you can just plug multiple monitors into the same GPU?
I want to do this, but is it possible to use 2 outputs on the same video card to run 2 VMs with seperate monitors?
Hi Jeff, is there any possibility to discuss issues? The discord server seems to be invalid link for me.