Video

The testing platform? That's a custom built streamlit/python/langchain app I built specifically for my lab - so it's not really an app I distribute

I actually have already tried it out, it's fairly impressive so far but I know there is some craziness going on around the model, repo, and huggingface/etc so waiting that out before we run it through it's paces. It's very lengthy - but 6x A4500 cluster on the current Q8 quant pulls ~6 TPS - to give an idea of how verbose it is our typical ~100 token prompt to create a Github Action results in a ~1300 Token response in an initial run 🤔

absolutely, but def 💰 involved. I have another video (and more coming) that pits a 3090 vs some other cards/etc if interested.

Hey thanks for that! Stay tuned, lots more coming!

Thanks for the feedback, I've tried to do better for those with smaller screens in more recent videos.

@@RoboTFAI Thank you.

Yep for sure, we do that in a lot of other videos. This one was just to focus on that scaling out GPUs isn't for speed, it's for memory capacity to settle a convo amongst some friends.

Coming! Though I don't do hypervisors much anymore (one node under TrueNAS but mainly for my github actions runners through ARC) - mostly bare metal nodes that are low power (n100/i5/etc/etc).....minus the GPU nodes of course!

1. Yes! Normally controlled by `gpu_layers` settings in the model - which determines how many layers to offload to the GPU(s), rest will use RAM/CPU 2. Nvidia just mainstream and their support with software/etc is pretty far ahead. AMD is def being used also - you don't hear about it as much but there is tons of large orgs doing big clusters of AMD based cards. 3. That depends on your needs, and your expectations of model response times (TPS). Most models can run on a good CPU if you are patient enough for the responses. 4. Not that important UNLESS you want to be able to #1 - and split models, or run them purely on CPU inference. If so you want as much RAM as possible (same thing we all want from our GPUs!)

@@RoboTFAI Thank you for your generous response. And I'm now a subscriber.

I don't typically test for accuracy, lots of channels out there doing that and it is very subjective depending on your needs. As an engineer most LLM's are good at "boiler plating" code or giving you a place to start and iterate on. All of rest the Quants for Mistral 7B baselines are coming in the next video and we will compare Q3, Q4, Q5, Q6, Q8, and even FP16. These will be used as baselines for future testing!

I haven't tried on pure CPU inference, but I did do it with distributed inference over the network in another video. We can certainly try as I have nodes with the RAM to do it in.

@@RoboTFAI oh, can you send me the link to the other video, I would be interested to see how you did the distributed setup.

@@ZIaIqbal czcams.com/video/CKC2O9lcLig/video.html - is the Llama 3.1 405B Distributed inference video. It's using LocalAI (Llama cpp workers/etc) under the hood: LocalAI docs on distributed inference: localai.io/features/distribute/ Llama.cpp docs: github.com/ggerganov/llama.cp...

Haha - but I have been known to burn up a power supply or three, a big UPS, couple breakers, etc - luckily lab has a few dedicated 20amp circuits these days. There is absolutely a fire extinguisher hanging in workshop/lab! Hey I would love to buy a 4090, and a million other cards! To be honest, I never really planned on a channel, I put up a video from a discussion with friends (basically to prove them wrong with data) and somehow you folks seem to like what this crazy guy does in his lab? if channel continues to grow and happens to make money one day happy to throw it all back in the channel. For now my budget is not much 💸

@@RoboTFAI Haha well you've earned this sub, curious what you end up testing next. "Not much" as you have a handful of $1k gpu's. Carry on good sir. o7

I think the 4060 (16GB) is a great card - people will flame me for that but hey. It's absolutely useable for interactive work if you are not expecting lightning-fast responses. Though on small models the 4060 really flies for what it is and how much power they use. Lower, lower, lower your expectations until your goals are met..... I did sell some of my 4060's but only because I replaced them with A4500's in my main rig so they got rotated out. I kept 4 of them which most days are doing agentic things in automations/bots/etc while sipping power or in my kids gaming rigs.

I don't work much with TTS, or STT - but we can go down that road and see where it takes us

That highly depends on your needs, wants, and wallet - if you want the ability to offload to RAM for any reason (super large models, or keeping KV in ram), the more RAM the better. Otherwise if offloading to the GPUs fully you only really need enough to run the processes which is fairly minimal. My rigs are Kubernetes nodes so I keep them stacked to be able to do anything with them, not just inference/etc.

The biggest issue you'll run into is the lack of PCIe lanes on the motherboard. If you have a consumer motherboard with recent gen Intel or AMD CPU, you will only be able to run 2 GPUs, 3 if you're lucky. You will likely need to use PCIe extension cables to space out your GPUs as consumer GPUs are generally 3+ slots wide, so you will get no cooling if you stack them next to each other. Once you go over 2 GPUs, you have to go into workstation or server motherboards to get the needed PCIe lanes; so EPYC, XEON, or Threadripper motherboards are needed. Best to buy used from eBay if your wallet doesn't support brand new Threadripper setup. Best bang for the buck will be used EPYC MB + used 3090s. You'll need muliple 1000+ W power supplies to feed the GPUs as well (I run with 2x 1600W)

Good question - want to sponsor me some to play with? 😁

If I get the funds. I was interested in them myself. For price over vram + power consumption. I would invest in a cluster software for these. Speed might be diminished but it's certainly a cheaper option then building a server for inference only using self contained devices.

Feel free to reach out, tis a community! I have several M40's from when I first started down this road that I would be willing to part with.... it's a slippery slope

@@RoboTFAI you're fantastic! Thanks. I'd love to reach out. Would appreciate to have your email or something so i can discuss! Maybe we can make a deal on your M40s of you have any spare of them that u don't use? Thanks.

@@k1tajfar714 robot@robotf.ai or can find me on reddit/discord/etc - though not as active as I would like to be.

Many thanks!

Sure, I think that's overdue at this point!

As much as you can pump for distributing the model - during inference it's really only about 10-20 MB/s per node

Better than stealing their GPU's out of their rigs right? 😂

I assume you are referring to Grafana (with Prometheus). Along with the DGCM exporter that is part of Nvidia GPU Operator for Kubernetes docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html

Thanks for the feedback!

Skynet is possible in your basement 🦾

Congrats!

Wow thanks! Much appreciated

Glad you enjoyed it!

How are you running it? Just local-ai directly or through docker? You would still need to set the correct environment variables (and with docker pass in "--gpus all") when starting the worker. If you are using docker just for the workers/etc make sure you have the nvidia-container-toolkit installed also (pre-req for passing NVIDIA cards to docker containers). If all that is covered, I would need more info on the setup. Feel free to reach out and I can attempt to help

​@@RoboTFAI Thanks for responding and let me say, that there are not many videos out about getting distributed inference running so your take on this is most welcome! I don't use docker and tried to keep it as simple as possible to reduce the amount of error sources. I just used to curl command on the very top of the installation page (curl http[:] localai io [/] install.sh | sh). This installs local-ai as service. So I changed and set the required environment variables in the service or the enviroment file (ADDRESS, TOKEN, LD_LIBRARY_PATH ) and installed the latest version of the Nvidia toolkit since my 12.4 and 550 driver were already too old and I got errors at first. Now I'm at toolikit 12.6 and driver version 560 and local inference works. So far I only tested with the Meta LLama 3.1 8b model in Q4, that can be installed directly via the web UI. I then enabled P2P and set the token on the server side in another environment variable, so it stays the same everytime. I created a worker also as service on my second machine to connect to the first using that token. The connection is sucessful and I can also do chats but only on CPU. I've then simplified it even more, disabled all services, switchted to the service user and ran local-ai (run with --p2p) as server on the main machine and another instance as worker on both machines, all in terminal sessions. Both workers connect but in CPU mode. I don't know if that is supposed to be that case but on the page in the screenshots you can see the same. What's in the log on your workers? I get someling like that: {"level":"INFO","time":"2024-08-10T16:46:54.792+0200","caller":"node(...)","message":" Starting EdgeVPN network"} create_backend: using CPU backend Starting RPC server on 127.0.0.1:44469, backend memory: 63969 MB There are no errors on any of the 3 running instances, the clients show the connection to the server instance and the server instance does server things. But it takes ages just to load the model and the inference is on CPU. None of the involved GPUs loads anything. Also I wonder how the model is supposed to work. I had expected that there must be a local version of it on the clients too but it doesn't seem to be the case. Yet transfering like 40-50GB of model data over the LAN each time you load a 70b model is very ineffcient. I couldn't find any documentation on this issue either. Edit: Reposted, seems mentioning a curl request is forbidden now...

Yea I agree on the loading across the network being fairly inefficient but the tech is also really new still. As far as the setup goes do you have your model setup for GPU (gpu_layers, f16, etc)? localai.io/features/gpu-acceleration/

@@RoboTFAI I had described every tiny bit in detail and even included some log messages but censortube deleted my answer. Twice. I didn't notice it did it again and since now month has past, I don't remember the details anymore. I've just checked that f16 is set in the yaml but I have not specified layers as the model should fit on either system's VRAM. It also runs in GPU model if I used it locally but remotely it connects only in CPU mode to the server system.

Adding in 2x 4060's won't really increase the speed over 1 of them, at least not noticeable. There is some other videos on the channel addressing this topic a bit. Scaling out on # video cards is really meant to just gain you that extra VRAM. So it's always a balance of your budget, costs, power usage, and your expectations (this is the more important one). Lower, lower your expectations until your goals are met! haha

Awesome, thank you! How to's coming eventually gotta find the time which I have none of!