Swap GPUs at the Press of a Button: Liqid Fabric PCIe Magic Trick!
Vložit
- čas přidán 1. 06. 2024
- Easily allocate hundreds of GPUs WITHOUT touching them!
Check out other Liqid solutions here: www.liqid.com
0:00 Intro
1:00 Explaining the Magic
2:00 Showing the Use Case Set Up
4:41 The Problem with Microsoft
6:07 Infrastructure as Code Hardware Control
10:36 Game Changing our Game Testing and More
16:12 Outro
********************************
Check us out online at the following places!
bio.link/level1techs
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Music: "Earth Bound" by Slynk
Edited by Autumn - Věda a technologie
The computer isn't real. The fabric isn't real. Nothing actually exists. We're all just PCI-express lanes virtualized in some super computer in the cloud. And I still can't get 60fps.
Goddamn Nvidia all that money and keeping us fps capped
🤣🤣
There is no spoon taken to the extreme.
The hivemind secret guardians saw that. They will get you.
😂😂😂
So I can literally hotswap my 4090’s as they melt like a belt fed gpu pc?
thats like a gatlinggun for gpus. dont bring manifacturers on ideas.
Melt? Like turning them to liquid, pretty on brand. ;)
I don't think 4090's fit in that chassis based on how much room is left in the front with those other cards.
Lmao
That PCIe tech is just fantastic for software testing. I test my OpenCL codes on Intel, Nvidia, AMD, Arm, and Apple GPU drivers to make sure I don't step on any driver bugs. For benchmarks that need the full PCIe bandwidth, this system is perfect.
Please Liqid, introduce a tier for homelab users!
oh yea could do so much, proxmox systems on dual ITX all sharing GPU and network off one of these
I doubt this can be made affordable for common mortals
Please deposit your 2 kidneys and 1 eye before you make any such requests.
My humble dream setup would be a “barebones” kit consisting of the PCIe AIC adapters for the normal “client” motherboard and the “server” board that offers four x16 slots. You’d have to get your own cases and PSU solution for the “server” side.
@@marcogenovesi8570 you can tho, in terms of hardware it's just a pci switch the hard part is the low level code to match the right pci device to right cpu and on top of that software that connects it to workflows that can understand this.
WOW! imagine having 6 different CPUs and 6 GPUs, rotating through all 36 combinations to hunt for regressions! Thank you for sharing this magic trick!
imagine if windows worked well
Most likely the windows box would need to be shutdown before you switch or the OS will crash.
I do this daily in my lab. 16 different servers, 16 GPUs (4 groups of 4) and do fully automated regressions for AI/ML models/GPU driver stacks / Cuda version comparisons. Like I have said in other posts once you stitch this together with Ansible / Digital Rebar things get really interesting. Now that everything is automated... I just simply input a series of hardware and software combos to test and the system does all the work while I sleep. Just wake up review the results and input the next series of tests. There is no more cost effective way for one person to test the thousands of combinations.
@@joejane9977 It does. I mean, it works well enough that few people go through the hastle of conciously switching. It's more a default switch if people go start using a tablet as a primary device, do to not needing a full fat desktop for their day to day needs.
For perspective of where I am coming from: I have a trio of Linux systems, a pair of windows systems; one of the windows systems is also dual booted to 'nix. Used to have a macOS system but, have no need of one, and better things to spend money on.
For some stuff: Linux is great; thing is, I have better things to do with my time than tinker with configs to get things running - so sometimes, a windows system just works.
Hell yea, we need that consumerized!
I would love this in the home setting. If it is hot pluggable it is also programmable which means that you could upgrade GPUs periodically but instead of just throwing it away you would push it down the list on your priority. Hubby and Wifey could get priority on the fastest GPU and if you have multiple kids they would be lower priority. If mom and dad aren't playing at the moment though they could just get the fastest GPU to use. You could centralize all of your hardware in a server in a closet and then have weaker terminal devices. They could have an amazing screen, keyboard, etc but they could cheap out on the CPU, RAM, GPU etc because those would just be composed when they booted up. Similar to how computers will switch between an integrated GPU and a dGPU now you could just use the cheap devices iGPU while doing the basics but then if you opened an application like a game it would dynamically mount the GPU from the rack. No more external GPUs for laptops and no more insanely expensive laptops with hardware that is obsolete for its intended task in 2 years.
moooom?! why is my fortnite dropping fps lmao
I would have concerns about cross talk and latency from like, signal amplifiers, in that scenario. I could not imagine trying to triage the issues this will introduce. 😂
@@SK83RJOSH I think latency would be the biggest one. I am not sure what you mean by cross talk though. If you are meaning signal interference I don't think that would apply here any more than it would apply in any regular motherboard and network. If you are meaning about cross talk in wifi then this really would not be how I would do it. I would use fiber for all of this. Even Wifi 7 is nowhere near fast enough for this kind of connectiviy and would have way too much interference. Maybe if you had a 60ghz connection but that is about it.
Love Wendell's off label videos
I was skeptical that I'd find anything useful or interesting in this video since the use-case doesn't suit me personally, but Wendell could explain paint drying on a wall and make it entertaining / informative.
This is absolutely incredible! Wow, I didn't even realize how many possibilities this opens up. As always, another great video man.
I've been waiting for this video ever since Wendell first started talking about/with the Liquid people. Glad it's finally here!
Can't wait for the Linus Tech Tips lab team to announce their use of Liqid in two years
I mentioned this idea in his comments when Wendell was doing interviews with the liqid guys, but Linus being the dictator he is in his comments has banned me from commenting.
@@mritunjaymusale sure buddy
LTT does already use Liqid, just not this product. You can see in one of their videos they have a 2U Liqid server in their main rack. It seemd like a rebranded DELL server, but still from Liqid.
Ah TIL, thanks for the info @janskala22
Can't wait for you to type "ltt liqid" into CZcams search and realize LTT has videos from the last 3 years showcasing Liquid products.
This is wickedly cool. Practical or usable for me? Naw, not really. But seeing that messy workshop lived in, satisfying as the tech.
This would come in so handy in my small three node Proxmox cluster, assigning GPUs to different servers / VMs when necessary. The image would be streamed using Sunshine / Moonlight (similar to Parsec). I wish there was a 2 PCIe Slot consumer tier available for a price that enthusiasts would be willing to spend!
I use this every day in my lab running Prox / XCP-NG / KVM. Linux hot plug PCIe drivers work like a champ to move GPUs in an out of hypervisors. If only virt-io had reasonable support for hot-plug PCIe into the VM so I would not have to restart the VM every time I wanted to change GPUs to run a new test. Maybe someday.
I would love to see performance tests comparing the impact of the cable length, etc.
Essentially, the PCI speed impact not only in terms of latency but also throughput, the native solution vs LiquidFabrid products.
I have a hard time believing that this solution has 0 downsides, hence wouldn't be surprised that the same GPU has worse performance over LiquidFabric.
Cable length is a red herring. An 8 m electrical cable only takes ~38 ns to pass a signal and the redriver (not retimer) adds sub 1 ns, while normal PCIe whole link latency is on the order of hundreds of ns. However, the switching of the Liqid fabric will add latency as will redrivers.
There are most definitely downsides. Some PCIe devices drivers will crash with the introduction of additional latency because fundamental assumptions were made when writing them that don't handle the >100ns latency the liqid switch adds well. ~150ns additional latency is not trivial compared to the base latency of the device.
this might be very useful with CXL if it lives up to expectations.
Liqid already has demos of CXL memory pooling with their fabric. I would not expect it to reach production before mid 2025.
CXL already goes far beyond this as it has cache coherency, so you can pool devices much more easily together. I see at as an evolution to this technology (and the nvswitch stuff), which CXL 3.0 and beyond expands on even further with the extended fabric capabilities and PCIe gen 6 speeds. I think that's where the holdup has been since it's a relatively new technology and those extended capabilities are significant for hyperscalar adoption which is what drives much of the industry and especially the interconnects subsector in the first place.
If there's something you can cram into PCIe lanes, you bet Wendell's going to try it and then make an amazing video about it!
For some reason this is one of the coolest things I've seen in a while.
I came for the magic.. I stayed for the soothing painting above the techbench
I was looking at the vfio stuff to have everything in a different part of the house. Now this seems like just as good of a solution. Having the larger heat generating components in a single box and having the mobo/cpu/os where you are would be a nice touch. Would be great for SFF mini pc's as well to REALLY lower your footprint on a desk or in an office/room.
I had no idea something like this was possible. Very cool. Its not the subject of the video, but that iMac you were demoing on, is it rigged up to use as just a monitor? Or is it actually running? Looks funny with the glass removed
fantastic video!
Thanks Wendell. I'd like to see a video about network booting and imaging.
Hey if you have to record in that noisey environment again you can leave about 15 or 30 seconds of you being silent at the beginning or end of video to use as a noise profile. In Audacity use the noise reduction effect generate the noise profile than run it on the whole audio track. Then it should sound about 10x better, or nearly get rid of all noise.
I'm surprised Wendell isn't using a pair of communication earmuffs; hearing protection coupled with a boom mic (or a bunch of mics and post-processing) possibly being fed directly to the camera.
As far as I know a good, comfortable set of earmuffs, especially something like the Sensear brand (which allow you to have a casual conversation next to a diesel engine at full throttle) are more or less required equipment for someone that works in a data center all day.
This is really interesting and cool tech!
WoW.. this is truly amazing, impressive, I dunno... like, I usually expect smart stuff on this channel from my list of tech channels, but this time, what Wendell done is a complete another league.
I bet Linus was thinking about something similar with his tech lab, but now there is someone to be hired for his project with automated mass testing.
I don't want an expensive box in my basement. In my attic high and DRY. 😂
I've absolutely been wanting and dreaming of something like this
cool and so. is that the same notebooks do that have a switchable igpu and dedicated gpu?
Wendell does it fit in the Minisforum MS-01?
It woukd be a massive plus if it would and works.
RTX A2000 12GB is already good but this is a complete game changer for alot of systems mini or full desktop.
Guys, can you help me gather Wendel's most used phrases? Like
- coloring outside the lines
- this is not what you would normally do
- this is MADNESS
- ...
He uses "RoCkInG" 19 times every video like he's a tween that needs extra time to take tests.
I have been thinking about this use case for a year now, for UE5 development, testing and validation. Recently also thought about using image recognition with ML or "standard" computer vision (or a mix) for automatic validation.
I can see this being valuable for both developers and also for tech media benchmarking. I just need to allocate time to dive into this.... or get it served "for free" by Wendel
I really wanted to do something similar in my Uni's server for deep learning since we had 2 GPU based systems that had multiple GPUs using this we could've pooled those GPUs together to make a system of 4 gpu in one click.
Could this use all the GPUs for 3D Rendering in Octane or Redshift 3D for a single PC or is it just one GPU at a time? I know Wendell mentions SLI briefly but to have a GPU render machine connected to a small desktop pc would be ideal for a home server setup.
Wow very interesting !
Can you control power on those PCI devices ? I mean lets say only one GPU powered on at a time, the one that is currently used remotely.
Also how do you sent the video signal back to the monitor ? Are you using a extra long display port cable, or a fiber optic cable at some sort ?
Thank you.
PS: What is the approximative price of such a piece of hardware ?
Man, I'm not an IT guy. I know next to nothing. But I love this sht...
How about power usage when you have all these GPUs running? Do the rest idle when unused at reasonable wattage / temp.? It's also hard doing game testing due to thermals as you are using different enclosure from standard PC etc. There muat be noticeable performance loss too.
I can't speak for client GPUs but enterprise GPUs have power saving features embedded into the cards. For instance an A100 at idle pulls around 50 watts. At full tilt it can pull close to 300'ish watts. The enclosure itself pulls about 80 watts empty (no GPUs). As far as performance loss. Based on my testing of AI/ML workloads on GPUs inside Liqid fabrics compared with published MLPerf results I would say performance loss is very minimal.
Now we know what really happened to Ryan this week, Wendell was practicing his magic trick!
Thought of another question: Can the box that houses all the PCIe AICs hard-power off/on the individual PCIe slots via the management software in case there is a crashed state? Or do you have to do something physically at the box?
There is no slot power control features... There is however a bus reset feature of the Liqid fabric to ensure that devices are reset and in a good state prior to being presented to a host. So if you have a device in a bad state you can simply just remove it from the host and add it back in and it will get bus reset in the process. Per slot power control is a feature being looked at in future enclosures.
Again, thanks for that clarification. Would definitely appreciate the per slot power on/off control, would be helpful for diagnosing maybe defective PCIe cards and of course also reduce power consumption with unused cards not just idling around.
What sort of cable does this use? Could this be run over fibre instead?
Also can you have a singular GPU Box with a few GPUs and then use those GPUs interchangeably with different hosts.
My thought process is. GPU box in the basement with multiple PCs connected over a fibre cables so I can just switch GPU on any device connected to the fibre network.
The cable is a SFF-8644 cable using copper as a media. (mini-sas) There are companies that use optical media but they are fairly pricey.
so instead of a disk shelf I could have one of those racks, fill it with HBAs instead of gpus or replacing them all with m.2 cards would that work ?!?!!? @Level1Techs
Can this work in an internet cafe or would the latency be too big for competitive gaming?
I could see Steve and team at GamersNexus utilizing this for retesting older cards when new GPUs come out.
The Wizard at Techpowerup must be aware of this.
Let's say you have a WS motherboard with 4 expansion slots PCIe x16.
Can you dynamically activate/deactivate by software these PCIe slots so that the CPU can only see one at a time ? Each of the slot is populated with a GPU of course. This need then to be combined with a KVM to switch the video output to the monitor.
The real question is, does this support Looking Glass so you can do baremetal-to-baremetal video buffer sharing between hosts? I know it should technically be possible since PCIe devices on the same fabric/chassis can talk to one another. Yes, my mind goes to some wild places, I've also had dreams of Looking Glass over RDMA. Glad you've finally got one of these in your lab. Anxiously awaiting the CXL revolution which I might be able to afford in like a decade.
whats the price?
i might have missed it.. But would/will/could there be an option to shove around gpus (or AI hardware) on a server running multiple VMs to the VMs that currently need it, and "unplug" it from idle ones ? .. Well, ok, you would need to run multiple uplinks at some point i guess.. Or have all gpus directly slotted in your vm server.
The ability of Liqid to expose a single or multiple PCIe devices to a single or multiple hypervisors is 100% a reality. As long as you are using a linux based hypervisor hot-plug will just work. You can then expose those physical devices or vGPUs (if you can afford the license) to one or many virtual machines. The only gotcha is to change GPU types in the VM you will have to power-cycle the VM because I have not found any hypervisor (VMware / Prox/ XCP-NG / KVM-qemu that support hot-plug PCIe into a VM.
@@jjaymick6265 thanks ;-) you seem to be going round answering questions here 🙂
@@PsiQ I have 16 servers (Dell MX blades and various other 2U servers all attached to a Liqid fabric in my lab with various GPUs/NICs/FPGAs/NVMe that I have been working with for the past 3 years. So have a fair bit of experience on what it is capable of. Once you stitch it together with some CI/CD stuff via Digital Rebar or Ansible it become super powerful for testing and automation.
Wow this is so amazing, I bet the pricing is far out of the range of a small office like mine though
…could you hook up a second Liqid adapter in the same client system to a Gen5 x4 M.2 slot to not interfere with the 16 dGPU lanes?
Liqid does support having multiple HBAs in a single host. Each Fabric device provisioned on the fabric is directly provisioned to a specific HBA so your thinking of isolating disk IO from GPU IO would work.
Thanks for that clarification.
I might actually be able to use all my pc addiction parts now!!!!!
Very cool!
Hardware unboxed! Steve!!!!
Woooooo! PCI-E fabrics baby!!!
Magic Wendell? Now I cannot get that image out of my head.
This could genuinely be useful for massive engineering firms, most engineering firms nowadays use 3d modelling and thus having a server side gpu setup could technically mean every single computer on site has access to a 4090 for model rendering and creation without buying and maintaining 100+ gpus.
From where can one buy this hardware? I can't find any info on Liqid's website. Google didn't help either.
I know that GPU riser cables are common, but realistically, how much latency is introduced by having the GPU at such a physical distance compared to being directly in the PCIe slot on the board?
Oh my. This is something special.
Looking forward to a L1T + GN collaboration video on building this into a working gaming test setup (Pretty Please ☺️)
Throw Hardware Unboxed into the mix. I think both Steves would love it. Only if you could do the same with CPUs...
Does the card support 2 gpus at 8x each?
so you saying i can do this with my 15 gbs internet, and connect my monitor or pc to a server game and ps5??? just have to pay 20 per month right like netflix?
How much it costs??
My only question is how much latency does this add, even in a short run in the same rack?
With SR-IOV, Could I attach different VFs of the same PCIe card to separate computers?
Liqid does support SRIOV, but the VFs are not composable. The way SRIOV is leveraged today is a single card that supports SRIOV is exposed to a host and the VFs and SRIOV bar space is then registered by that host. That host then can present each of those VFs to a VM just as if the card was physically installed into the host.
What are the performance implications in a gaming scenario? I assume there must be some amount of drop, but I'd like to know how significant it is.
A few ms of latency I reckon
How's the latency? Every PHY jump induces latency, so considering all the hardware involved, this should have at least 3 additional points of extra latency. So maybe like 4-5 times the round trip of native pcie...
100ns per hop. In this specific setup that would mean 3 hops between the CPU and the GPU device. 1 hop at the HBA, 1 hop at the switch, 1 hop at the enclosure. so 600 nanoseconds round trip.
Such a good thing for cloud computing.
How much worse is performance in terms of timing/latency compared to the slot on the motherboard? I wonder if it would be noticeable for gaming...
How to do cluster for training LLM, first 4 x 8GB in one system and second 4x8gb over lan.
your test bench has an optical drive?
Wendell: “you may have noticed..”
Me: “I noticed that Apple monitor!”
Is there any PCIe rack-mount chassis that can allow this to be a rack-mounted solution?
Typical installation is rackmount. It is all standard 19 inch gear that gets deployed in datacenters around the world.
@@jjaymick6265 can you post a model number that you'd recommend?
@@jjaymick6265 Is there any particular model that you'd recommend.
@@mohammedgoder Somehow my previous comment got removed. If you are looking for supported systems / fabric device etc the best place to check is on Liqid's website. Under resources they publish a HCL of "Liqid Tested/Approved" devices.
I found it. Wendell mentioned it in the video.
Liqid makes rackmount PCIe enclosures.
How do I buy one of these switches, I must have one
Maybe I'm misunderstanding this, but wouldn't the latency between the CPU and the GPU be really bad here? Especially with the setup Wendell has in the video?
I just want this inside my workstation - a bank of x16 slots and I get to dynamically (or even statically, with dip switches) assign PCIE lanes to each slot or to the chipset.
Ah! He admits it, he is a wizard!
Where can I get a copy of that Wallpaper?
What's the extra latency?
Here take my money, this is amazing, me with a lot of computers and with I would be able to use my best GPU on all of them!
Have we cracked the code to pass a igpu to a vm in say TrueNas?
I didn't even know this was possible. How long can the cable run be without degrading performance?
In a datacenter setting using copper cables it is limited to 3 meters between host and switch port and 2 meters between switches and enclosures. (he did not show the switches but yes there are PCIe switches involved also) Host --> 3m --> PCIe switch --> 2m --> enclosure
@@jjaymick6265 Thanks for the info!
This could greatly cut down on hardware for render farms in VFX. Neat
Very cool
Cool as heck it is.
Id like to be able to hotplug GPUs in my running VMs as well...how nice would it be to have say two or three VM boxes for CPU and MEM, one SSD box, one GPU box and one NIC box so you could just swap any nic/gpu/disk to any VM in any of those three boxes in any combination.... that'd be sweet....i definitely dont need it but that just makes me want it more
Over the last couple days I have been working on this exact use case. In most environments this simply is not possible, however in KVM (libvirt) I have discovered the capability to hot-attach a PCIe device to a running VM like this... virsh attach-device VM-1 --file gpu1.xml --current . So with Liqid I can hot attach a GPU to the hypervisor and then hot attach said GPU all the way to the VM. The only thing I have not figured out how to get done is to get the BAR address space for GPU pre-allocated in the VM so the device is actually functional without a VM reboot. As of today the GPU will show up in the VM but drivers cannot bind to it because there is no bar space allocated for it so in lspci the device has a bunch of unregistered memory bars and drivers don't load. Once bar space can be pre-allocated in the VM I have confidence this will work. Baby steps.
I like some of this.
But let me look at the far end, outside of Parsec or similar, how am I re-routing the video signal or playback. Perhaps there needs to be a wink wink nudge nudge Level One KVM solution. But outside of this, walkiing down to the basement, to re-plumb the video cables old school to new host/ changed host, kinds degrades the magic a bit on the idea...
Fun.. not sure it would be ideal for competitive gaming exactly.. but very very cool
I don't know how to thank you enough without being weird man....XOXO
This looks like it might also eventually enable migration of VMs with PCIe passthrough.
Do we want to know what this costs?
Nice ❤
This is cool and all, but what's the CPU overhead?
Zero? CPU doesn't see the PCIe bus shenanigans. Its outside any sort of virtualization. Literally just electrical PCIe bus routing.
okay, this is game changing
Functionally Liqid software is not in the data path. The "overhead" if you wanna call it that would be the latency induced when installing a device behind a PCIe switch instead of directly connecting it to a CPU root bridge. That latency is advertised as 100ns per hop switch latency. To date... I have yet to have anyone reliably be able to show the measured difference.
What's your monthly power bill?!
Hold on, is that an old iMac you're using for a screen?
Sounds like GN is getting a new setup
Oh wow, that is neat, how does it work?
Let's go to the basement and let me show you...
nice try Level1, not gunna get me that easily.
Jokes aside, this is pretty awesome tech.
This would be great for KVM. Plug USB cards into PCIE, and send your peripherals to all your computers.
This seems tailor made for all those single slot consumer boards that get sold. It would make them much more useful.
I can imagine it could in some future reduce the cost of a recording studio with all of their specialized audio cards if their could spend 1/1th the cost on the motherboard, and share the cards across multiple equipment.
I could seen cryptominers using the best cards depending on the current pricing.
I could see switching GPUs depending on which gives the best gaming performance.
How about retro machines using older PCI-E cards with VM's?
I imagine the bandwidth of older GPUs wouldn't sature the bus, so you could connect them to the system and pas them through to individual VMs?
Or, some PCI-E 1.0 cards in Crossfire and SLI with a one slot motherboard.
Way overkill, but seriously cool tech.
Speaking of that, you could ger some PCI-E to PCI-X audio equipment, pass that through to some Windows XP VMs, and get that latency goodness and unrestricted access audio engineers loved in a modern one slot solution.
Enterprise side, I could see creating a massive networking VM set with one of these cards in each of the main systems slot, and attached to a separate PCI-E box, each setup with those multifunction cards. Custom bespoke network switch.
1/1th the cost?
@@tim3172 1/10th, lol.
The iMac bezel blew me away...