I built a MONSTER AI Pi with 8 Neural Processors!
Vložit
- čas přidán 27. 06. 2024
- The thumbnail about says it all. It works! But what does that mean, exactly?
Some of the things I mentioned in this video:
- Alftel 12x PCIe Card: pipci.jeffgeerling.com/cards_...
- ThirdReality Zigbee Smart Plug: amzn.to/3X97MvC (affiliate link)
- Pi AI Kit: www.raspberrypi.com/products/...
- Hailo-8: hailo.ai/products/ai-accelera...
- MidnightLink's helpful Coral TPU setup instructions: github.com/geerlingguy/raspbe...
- CodeProject.AI: www.codeproject.com
- Raspberry Pi Linux PCIe issue: github.com/raspberrypi/linux/...
Today's CZcamsr outtro tribute: Chris Titus Tech
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: www.redshirtjeff.com
Main Channel: / @jeffgeerling
2nd Channel: / @geerlingengineering
Contents:
00:00 - You built a WHAT?
00:45 - The hardware stack
02:26 - Power monitoring
03:06 - Getting Coral and Hailo to work
05:44 - 55 TOPS!
06:30 - Not so fast... - Věda a technologie
Can't wait for a decade from now when they're packing 1024 TOPS into "entry-level" devices claiming "you definitely need all this power for current models"
1 BOPS?
640 TOPS ought to be good enough for anyone
@@heblushabusT = trillion, so B would be billion. Next step is Pflops for Petaflops
@@tuqe oh, right. bops sounded funny tho. so, POPS?
@@tuqeisn't T tera in this case?
It's nice to know that you could run multiple neural networks on independent NPUs! Like one for facial expressions, another for voice recognition, and another for text-to-speech!
seriously tho, 26 TOPS on that small thing is impressive by itself. combining a card with a few of those 26ers with something with PCIE-4x16 would make for impressive AI processing in a small package.
Yeah, Hailo makes a 'Century' card that does just that-hopefully as multi-NPU becomes more popular, programming for it also becomes more popular!
i would love a video on the home assistant power consumption!
+1 Yes, me too
Well that would be a 1 minute video 🤣
Oh thank god, I’ve been itching ever since you showed that b-roll
This reminds me of the Friends meme where Chandler gets a fancy expensive computer, and when asked what he is going to use it for he just says "Idk, games and stuff".
The infra is still fun even if you don't actually need the compute lol.
My favorite part of this video is definitely when the box gets identified as a cell phone and he holds it up to his ear 😅
Never stop doing what you do Jeff. Love the content, love the experimentation.
Jeff!!! The increase in content per week has been amazing. Don’t over due it but man I’m loving it
Bravo Jeff! That was alot of work on your part. Again, congrats and thanks for all the hard work you do for us!
I had no idea this channel existed, this is awesome!
and now you do, ha! this is the channel where things get crazy
hehe a pi mad scientist cobbling together contraptions no one thought of. love it! great scott!
🤗 Accelerate might be able to manage using all the NPUs at once. It's meant for using multiple GPUs for a single inference or training task but it might support NPUs too
That was fast
I was about to say that
My AI predicted it
that's what she said!
@@Level2Jeffu okey? u look a little red :p
Hey man! Wow. Saw this on Instagram and thought this would be on the main channel 😂. But awesome job man! This is so great 😂
OMG!!! Who are you going to be from now until the next video?!?! End of an era...
Ha!
It’s always fun to see you pushing the limits on single board computers. I’m sure it helps the hardware makers consider unusual use cases.
Jeff, have you seen the stories on the new research into IBD which suggests a genetic link and that certain cancer drugs might be effective? I saw the story on the BBC news site. Obviously I can’t post the link here, but I thought you might be interested.
Yes, I've been following that research-still 5+ years out from being practical but it's promising. The key is to find drugs that have fewer side effects than current TNF-blockers (which have similar mechanisms but target the immune system more broadly).
Love these kind of videos so keep em coming
"I've created a monster!
No one wants to see Marshall no more,
They want Jeff.
I'm like chopped liver"
Jeff the mad RPi guy! Love these builds when you push the envelope! 😎🤣
Level 2 Jeff is truly on another level.
The second one!
Here before 20k subscribers! Keep up the great work Jeff
thanks for the video!
I’m amazed by the things you are doing with the humble RPi.
Not as amazed as that fractal north case for the pi5 they are showing off at computex, but still amazed.
I love the crazy/nuts thing that is this set up! And, even if the "Coral Dual TPU" only shows up as one of them, I'll bet that it is still cheaper and faster than the USB version. Now, where was that AM power meter that your Dad has . . . ? ;-)
Jeff, I too would love to see a video on how to setup a Home Assistant Dashboard for Power Monitoring. You sir, are a Wizard!
We love you for your tinkering!
I'd love to see a version of this with the CM5 when it comes out. I think it would be cool to build an ITX-ish size ARM AI computer that you could use for all kinds of AI projects.
I would love to see a video going over your choice of power metering smart plugs & the integration into HomeAssistant.
Please make a power outlet video, I'm just getting into HASS now and power monitoring is next on my list - Also great video cheers!
Great video 😊 - slightly off topic but do you have any thoughts on using the overlay file system with raspberry and will it really protect the pi os from being turned off without shutting down? Would make a cool video perhaps to hook up a pi to a relay and have it turn on and off hundreds of times to see if and when the os corrupts.; have you done anything with the overlay / read only file system before?
Oh, man. That's a visually menace pi XD
Yesterday, you tried and SPAGHETTIBLY FAILED chaining NPU and now this. lol
Thank you for the attempts otherwise I would have tried myself. I think 2 NPUs can easily work with running 1 neural network on each NPU. This kind of configuration can realize many real world applications I have been dreaming of so many years. Thank you again.
Definitely! I think on the Pi 5 at least, that would probably be the ideal number of NPUs. You could stretch it to 4 okay too, but at that point the cost/build could point you to something a bit beefier like Jetson Orin.
I really like your videos they always bring novel information
Do you know when it will support running NVMe SSD and Hailo AI Kit simultaneously? (with the NVMe SSD used as the system boot disk)
Being able to run two Halo8's would be a cool project as in theory would be (2x 26, or up to 54 TOPs in theory). Combined with dual Pi cameras, would offer high frame rate stereo depth of field, or other fun video processing.
BTW: the Home Assistant monitoring is fascinating. Would be interesting in hearing more details.
Jeff, you have to write a book with a chapter on each of these types of Pi mods.
I appreciate that there are so many different TPUs/NPUs on the market, I’m just frustrated that we’re all beholden to nVidia when it comes to actually training models and running a lot of things.
Ditto. Wish at least AMD could offer something that would take the bottom out on either price or efficiency, but right now it is what it is :(
@@Level2Jeff I’ve got a used Tesla P40 with a water cooler in addition to another rtx 2060. Now that Intel works with Tensorflow and PyTorch, I’ve seriously considered just getting a 16 gig Arc 770 and paying for cloud computing if I need to train a model that needs more memory.
Hi Jeff, just wanted to let you know that you CAN use multiple hailo cores together, using the VDevice API - it automatically identifies the cores (granted it will only work with hailo chips).
HailoRT seems to have some multi-core configuration too... definitely some fun to be had here!
Hell yeah, finally you dont just tease😂 i was waiting soo badd, am i rpi addicted🤨
This would be useful to run Frigate AI detection from multiple cameras (e.g. PoE RTSP streams). Even this many devices is less power hungry than a 4080 to run 24x7, and compared to the cost of a good IP camera its not unreasonable.
0:04 Just flappin' in the breeze
"Why?" well to quote The Doctor from The Waters of Mars, "Fun"
Nice!
So, other SBC with PCIE Gen3x4 can leverage the full power of all AI chips installed?
Thanks for trying AND explaining the problems!
Can't wait for someone to write a "Kitchen Sink Aggregator" patch that'll normalize or make modules generically access -- ahh, I just wanted to use kitchen sink in a sentence.
We have level 1 tech,level 2 jeff what’s next level 3 steve 😂
Hey, thanks a lot for your videos. I'm wondering something that i'm sure has been adressed but i can't find a definitive answer on this. Does TPUs accelerate local LLM answer generations (Ollama for example) ? Thanks
300 TOPS sounds like a lot until you realize even an aging RTX 3080 has about 7400 TOPS. Sure the power use is different but point is you wont be running any big LLM models on this.
Is this really correct? I don't believe this.
@@pieterboots8566
I don't think that this is correct.
ai models can run in pipeline mode (if you have multiple tasks) for example yolo for bounding boxing a face and a second model that does check your expression and a third that correlates your face with known faces, and so on. I doubt it is possible to run these llms on these tpus - except they somehow split their model into n models
Hey Jeff! I'm still pretty curious about everything "AI". I'm just not sure how to take advantage of any of it right now. I'd love to see a video going over a bunch of different AI projects that these can be used for, either here or on the main channel. Obviously frigate is one, I've also seen some self hosted AI chat bots, though I'm not sure how well any of them would fare on a pi.
I know you're the "pi" CZcamsr, but I'm also curious about other applications of such accelerators. I wonder if an AI chat bot would work decently well on a 1L PC (or some similar micro x86 system) using something like the Hailo for processing rather than trying to cram a GPU in a small system like that.
If you know of a CZcamsr who is doing that sort of thing, I'm happy to check them out, just let me know.
Keep up all the cool videos. Cheers!
If you want 200 TOPS, just buy the Hailo-8 Century. It has 208 TOPS and is a single PCIE card. Although normal person probably can't buy it...
I'll just do it myself! Haha, so far I haven't found a reliable way to buy Hailo-8 outside of being an OEM partner. Hopefully they open up more individual sales at some point.
I know you won't tribute aVe, but, that would be hilarious!😂
How do GPU bit miners support the core splits? Some setups are looking at spreading a single calculation over thousands of cores. Maybe there's a way to port a miner task divider to NPU tasks. But I also wonder if we are still limited to running each TPUs individual Floating Point rating? Or could we run full 32-bit models?
I actually met the alftel guy and he is a GENIUS with RF stuff.
I don't doubt it. The contraptions he makes...
At first i was thinking the tribute was to Major Hardware, but the intonation and rhythm is completely wrong... it's closer to Explaining Computers, but the line doesn't match.
Ya got me on that one! @Level2Jeff !
I also recognized the tone as Explaining Computers, but the message doesn't match.
Check the description ;)
I've started to put the tribute into the bottom of the description to make it a little easier :)
@@Level2Jeff I skimmed and totally missed it! Guess I should've ^F-d instead! 😄
think you might be able to look over the nanopi boards since most have built in NPU? I know the Nanopi R5S has one. Not that much of one but still there
The trick to using those NPUs in parallel will be building a pipeline across them, where each NPU is supporting a subset of the layers in a model. This is a common technique in both training and inference, though I'm not sure if Tensorflow Lite supports it.
A pipeline would allow you to partition the model weights and compute across all of the NPUs, giving you a chance to run larger models then you could do on a single NPU. Your PCIe setup is very low bandwidth, but that is less of a concern here because pipeline parallel is only sending the activations (relatively small input tensors in inference) between the NPUs, not the larger weights.
Based on the limited information that I can find about Hailo and their sample hardware (a few of them have many Hailo chips on a single PCIe card) it looks like their software may support this.
Great video, does this also works with Pi4 ? Please also mention in future if the projects also works with Pi4 possible since not everybody has the latest and greatest :-)
It works with CM4, but not Pi 4 directly since the Pi 4 doesn't expose PCIe without a pretty serious hardware hack.
can this be used for llm's and stable diffusion or is it only useful for video tracking.
one day graphics cards will be replaced by AI accelerator cards, and all you'll need is a low power gpu.
@level2Jeff Where did you buy the Hailo-8 card? On their website they only provide an option for product enquiry, not for purchase.
I wonder how hard it would be for a Pi to orchestrate all these TPUs together. Could it offload some orchestration onto another TPU? I kind of want to write some code for this thing to see how it performs with multiple processors.
As price drops for NPU's people will use more then software will write the software for it. That is what the raspberry pi was designed for learning in a fun way.
Do these NPUs / TPUs work with ollama and the llama3 model? Currenting running ollama and llama3 on my gaming PC which has a 4060ti.
You choose to switch "geerlings" and it seems it paid off for the better :)
Interesting, but cool!
Great information in the video. Is there a PCI-e board like this that would work in a desktop, without buying the Nvidia RTX 40 series?
Hailo makes a 200+ TOPS 'Century' card that straps a bunch together and would fit inside a desktop case (full height PCIe card).
@@Level2Jeff Thank you so much. 🙂
Can you show us actual runs? Also would love to see the 10H version when it comes to your hands.
Might be interesting for scientific computing. Can you send instructions via MPI or something?
Hello Jeff. Odd question here. I have a Lenovo X230 that I will repurposing to "play" with local AI, specifically LLMs. It is going to struggle with the onboard GPU and I will eventually be moving everything to a different machine that will let me add a proper GPU card. Until then, is there a way that I could use a PCIE slot to NVME/M.2 adapter to put one of these little M.2 AI chips in the laptop? Since the machine has USB3 ports on it already, there isn't really anything useful I can put in the slot. If the adapter actually worked, would the extra AI board do me any good without custom software to utilize it?
Yes, at least under Linux. Not sure about Windows support for these things.
Question: Oobabooga (text-generation) Web GUI supports multiple GPUs and TensorFlow (TPU) with mixing + matching - would that work for your setup?
What's the best lowcost solution today for running local image-detection CNNs on a Pi4 or Pi5? Those USB-TPUs?
thanks for tinkering
r u gonna do it w/ the halio?
Arent they all sharing memory bandwidth? Are there any AI m.2s with onboard memory?
@Level2Jeff - Power Monitor Dashboard all the things - YES PLEASE
Oh man… this makes me think of how great a CM5 board akin to the CM3588 NAS board from FriendlyElec but for NPUs would be. Wait… could you just use that board for NPUs as is?
so Google TPU might be the LPU arranged in certain way? doesn't seems so, because LPU have huge memory chips... well the tops seems matched I wonder where this experiment will go into this is really interesting. Really interesting! indeed!
Btw what's the SSD and WiFi card for?
What is the best combo for pi hat AI NPU and also running an NVMe SSD all together?
probably either the NVMe BASE duo from Pimoroni or the dual NVMe board from Pineboards right now.
Yes, a dual m2 hat with a Boot and storage SSD, and that Hailo card for friggit, object detection , write to a db that is searchable (events) in plain text…😅
We need cheap PLX chips that take a newer standard bus in uplink and can provide many downlink at lower speed.
Also on consumer mainboard it is starting to be annoing. 20 or plus pcie gen 5 lanes where almost all hardware is gen 3 and 4.
Lot of throughput lost.
Couldn't agree more. Though those of us needing all that PCIe goodness are a slightly rare breed... and the answer till now is usually buy a big server CPU that gobbles up like 120W idle :D
@@Level2Jeff Even if we are the minority I think that the approcha that apple did with the MAC pro using the PLX and many lanes/slot is something that other HEDT/Workestantion OEM should start to consider.
New AM5 EPYC has been annoucned but lot of bandwidth is lost if you just plug a single HBA card on today mainboards x16 slot.
Can these boards be plugged into a full-fledged PC?
There a Qualcomm AI accelerator that use Dual M.2 slots, that board could be perfect for it (I've never been able to find the spec for dual m.2, so the spacing might be off), just one of those could get 200TOPS/25W
Good luck finding one though...
I think the name is probably "Micro PC city" and the u is used as a substitute for mu because it wouldn't be recognizable at that size.
before the WHAT it's compelling to know the WHY.
waited for the payoff but I guess I don't get enough why I'd want to use all these TOPs
I'm excited about using new tech but using it is the key. The excitement about the build comes after the excitement about the functional power. So I guess I don't get this video too much.
You need that 12xm.2 card with a 16x connector and put it in your ampere workstation.
Can you run LLMs on these NPUs?
Maybe call it “Zora” after the AI on the USS Discovery?
Well, technically, _you_ don't see _us_ but _we_ see _you_ ... ;-)
I had trouble getting my coral to work with codeproject ai long-term. It would work for a few hours but then stop responding. I don't know where the issue was on that. I mostly just gave up on it.
I had a similar issue in my testing, though didn't take too much time to debug it. One time it seemed to lock up the frontend of the Pi, had to force-poweroff!
It's cool to see that working _technically_, but in terms of inference, that's kinda the equivalent of having a 1000 CPU cores running on a single, shared stick of DDR4 memory. Do you hypothetically have enough cores to run a ton of applications at the same time? Yeah. But how much you'll be ably to do with the system will depend on the throughput of your memory. And it's the same with these little compute sticks. They rely on your Pi's memory so while you - on paper - have a lot of TOPS, running any task that operates on a reasonable amount of data or live streaming data will quickly saturate your onboard memory and pcie bus and prevent you from doing anything useful with multiple NPUs
Stop posting those AI generated video. I know it not you because you didnt recompile the kernel.
AI Recompile, go!
@@Level2Jeff You mean AI ReTrain.
Marco Reps at the end?
I wonder how feasible it is to put an abstraction layer in front of the 25 coral TPUs so that the software only sees one big TPU?
That's more-or-less what RAID drivers for ZFS and BTRFS do so clusters of 25+ drives appear as one block device. And what CPUs do to make scalar code think it's still being executed in-order as the only process on a single-threaded CPU in a virtual memory space, despite out-of-order execution, superscalar execution, hyperthreading, multithreading, branch prediction, and more, all happening in the background.
time to find some splitters for converting m.2 x4 into 4 m.2 x1 plugs for those dual-edge tpus
I know midnightlink irl
I wonder if Bend would work with this config ❤
Have you ever worked with a jetson nano by nvidia?
Yes! 😎🤖
Soooo does it like tell the time?
Jeff doing a Jeff impersonation? Tell me it isn't so!
So like, objectively I know that the Hailo isn't really meant for inference of production grade LLMs or anything.
...But like, I still want to see if a person couldn't do something silly like a bespoke MoE architecture with 128M active parameters and still get okay quality and speed.