my AI model box
Vložit
- čas přidán 18. 05. 2024
- Setting up AI models on the DAS and speed comparisons - visual studio / virtual machine tests.
Temperature/fan on your Mac: www.tunabellysoftware.com/tgp... (affiliate link)
Run Windows on a Mac: prf.hn/click/camref:1100libNI (affiliate)
Use COUPON: ZISKIND10
🛒 Gear Links 🛒
* 🍏💥 New MacBook Air M1 Deal: amzn.to/3S59ID8
* 💻🔄 Renewed MacBook Air M1 Deal: amzn.to/45K1Gmk
* 🎧⚡ Great 40Gbps T4 enclosure: amzn.to/3JNwBGW
* 🛠️🚀 My nvme ssd: amzn.to/3YLEySo
* 📦🎮 My gear: www.amazon.com/shop/alexziskind
🎥 Related Videos 🎥
* 🌗 RAM torture test on Mac - • TRUTH about RAM vs SSD...
* 🛠️ Host the PERFECT Prompt - • Hosting the PERFECT Pr...
* 🛠️ Set up Conda on Mac - • python environment set...
* 🛠️ Set up Node on Mac - • Install Node and NVM o...
* 🤖 INSANE Machine Learning on Neural Engine - • INSANE Machine Learnin...
* 💰 This is what spending more on a MacBook Pro gets you - • Spend MORE on a MacBoo...
* 🛠️ Developer productivity Playlist - • Developer Productivity
🔗 AI for Coding Playlist: 📚 - • AI
Repo
github.com/open-webui/open-webui
Docs
docs.openwebui.com/
Docker Single Command
docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main
- - - - - - - - -
❤️ SUBSCRIBE TO MY CZcams CHANNEL 📺
Click here to subscribe: / @azisk
- - - - - - - - -
Join this channel to get access to perks:
/ @azisk
- - - - - - - - -
📱 ALEX ON X: / digitalix
#machinelearning #llm #softwaredevelopment - Věda a technologie
Never bored with this crazy experiments 💜
Easy Alex. Just get a 8TB M4 Ultra next time 😂
Thanks for your videos ! Love them. It would be very nice to get your review on a Sinology NAS, I am thinking of buying 1 of those.
Great video.
Glad you enjoyed it
Great stuff
Get one of the Ugreen NAS. If you get teh 6 bay you get 6 3.5 Drive Bays, 2 m.2 Slots (aside from the OS drive) Finally it has Thunderbolt 4 ports in addition to the 10 Gbit ports
ordered :)
oh yeah, totally. i would run it on TB if you can afford to buy TB drives. for a poor grad student like myself, it’ll just have to be a makeshift NAS and external drives.
Do not buy ugreen nas, they suck
@@zezhenxu9113Ive heard “they suck” about every piece of gear I use from one person or another. What are your reasons?
They suck because I'm green with envy that I can't afford them. I don't know if ugreen with envy too.
Ugreen should have an Envy line of products but I think hachpee got that covered. Thank goodness it's not Compaq or eMachines haha.
Sorry I couldn't help it. Seriously though, wish I can afford that ugreen NAS. Have to make do with proxmox and truenas.
Quick question - how many tokens per second do you get on say 8B and 70B local LLM on the Mac ???.
I want to buy a server dedicated to LLM but adding an NVidia GPU to my PC is not what I had in mind - currently have a Radeon RX 6600XT - it spins up and makes a loud noise when using Ollama.
Just wait for the 400b model coming soon hehe
Hey Alex, great content! Would love to see some practical examples of what you have been using LLAMA for.
Don’t know if I got it right, but what are the advantages over using web ChatGPT, for example?
Please make a video which model a software dev should have and others model can be may have, because not all can have resources for this , thanks
My exact same experience. My new laptop is way too small for AI models. I can only do a few 8B p. models before my SSD is full. Cloud based models will not go away anytime soon they are better and they are fast despite being in the cloud.
Would you suggest the Samsung SSDs with or without the heat sinks for that particular setup?
I wonder if a external storage with network connectivity would be fast enough. You could match it with a VPN like Tailscale and have your models available anywhere.
i’ll let you know when i get my NAS :) although might need to upgrade my network first
why so many different AI models though, would be interesting to know what sets them apart
LoL I can't believe it! 4 SSDs in RAID 0 for gigantic speed, only to be bottlenecked by the 10 Gbps USB transfer rates :)) If that wasn't a bottleneck, those 4 drives, if they were decent PCI-E 3.0 ones, can go over 10 GB/s (that is gigaBYTES). Fast PCI-E 5.0 ones could probably go over 30 GB/s (I remember Corsair has a 10 GB/s SSD, so 4 of them + a bit of overhead should be able to do 30 GB/s). Anyway, the thing that triggered me was that Apple's SSD is much faster at 4:19 ... I really doubt it is. Compared to 4-RAID 0 normal SSDs, that is.
Also the breakdown at 1:37 is pure gold. Thanks Apple!
Edit: ok, wanted to check something and rewatched a bit. No mention of that USB 3.2 what type it is, but from the end tests on the Windows VM, reaching over 3 GB/s, makes me think it's actually a 20 Gbps (USB 3.2 gen 2x2 F*** the USB comitee for these absolute i-diotic names). Still, 20 / 8 means only 2.5 GB/s theoretical, more like 2.0 GB/s practical so where's the 3.3 GB/s coming from ? Not sure.
Also realized that the SSDs are Samsung 980 (not Pro), which is PCI-E 3.0, so around 3 GB/s each (it even says it on the box at 2:46 ). So the mention at 3:27 "It's only USB 3.2 But you don't need than 'cuz the fastest drive in there is gonna be uuhm 1 GB/s" is VERY wrong.
So using this solution, I should not care about space customization while making purchasing decisions?
Why keep below 34b than 7b. Or just keep the quantized version, you can delete or store in 16bit/8bit
No way, how big SSD is big enough for you, monster! 😅
Proxmox + truenas on an inexpensive mini pc (intel n305, if not thunderbolt) with 2.5 Gbit ethernet with thunderbolt or USB 3.2 port eith this DAS box.
i thought about doing this, but then just ordered the new ugreen nas instead :)
Just download more storage (and RAM)?
So much fun…
🤣🤣 I had the same problem and configured a NAS with some n100 mini pcs, then it wasn't enought so I got a new PC with a 4090 and like 16TB storage. LLMs are the perfect excuse to need storage.
Some day you’ll just buy Synology
you can buy the SD card adapter that allows an SD drive to be inserted sideways. I did that and put a 2T SD in there and with that card the the 2Tb SSD in my M3 I have a ton of room for models on the SD card. Love it.
what model do you have?
@@AZisk M3 Pro 16 with the Max chip and 64 GB 2TB. I bought this to hide the SD card. BASEQI UHS-II Aluminum microSD Adapter for 2021 M1 MacBook Pro 14 & 16” (Silver) Model USHii-420A
@@RedDragon72qWhat are the speeds like on it? I got a Transcend JetDrive for my M1 Pro MacBook and TBH haven’t really used the storage for anything. It’s too slow for most things but it’s there if I need it for storing large files. I keep a backup on my Parallels VM on there but it’s too slow to actually run from
@@DanielHarrisCodes standard speeds for an SD card, maybe a bit slower on read for some reason. I keep long term files and models on it. Loading the model takes a bit longer but once it it loaded you're all good.
haha. omg alex, seriously put your stuff on a NAS or an external drive. i put everything either on NAS, Dropbox (if i need to share with my lab), or on external drive (spinning disks). have you considered doing a hackathon for those near you?
i have a dropbox subscription. i’m sick and tired of the costs associated with it, and the lack of immediate availability of my data. NAS is next
وماذا عن كمية الشحنة الكهربائية داخلها كيف نعرف
why ollama not LMStudio?
What is your old time machine? Could you show us?
yes, i’m considering making a vid
If you're eating that much storage, then yeah, you should really be offloading them when not in use to a NAS or external media. It's not like you'll use all these models and whatever quantization or version they have at all times, right?
Why not to just remote desk into a desktop?
Wait wait wait! Hold up! So you are using usb 3.2? So maximum 20gbit, giving you like maximum 2500mb/second. Slower than what ONE of those nvme drives can do! What exactly do you think you gain by putting them in a stripe raid???
Probably the total capacity of all 4. Maybe a bit better than just JBOD.
@@DS-pk4eh With JBOD, he would not lose ALL data if one drive fails. The stripe raid give no benefit at all in this setup. Stripe raid is for maximising throughput at the expense of seek-time, as all drives needs to seek for one read. Does not hurt as much on SSDs of course, but still hurts.
That's a lot of money for "only" computing.
For your sd raid 0 you are limited by usb3.
Why use a das when you can use a nas, so you can also stream films and series + run your vms
already ordered
Why not use services like Backblaze to increase your cold storage space? Yes, you don’t get the convience of local redundancy, But it’s cold storage! If local HDD fails, get the copy online.
Ideally I should, but I don't like paying monthly storage fees.
Now put a sign on the black box that says, “do not feed the A.I.” 😀
🤣
Why didn't you get a dedicated AI server?
if i build out a server like that, i’ll want to spec it out with nvidia stuff, and i’m waiting to see what the 50xx series do
Wow, look at that shit.
Check it out!
Alex, you need to show #AI who is THE BOSS (you).
Put each LLM on a 2TB ext. USB so they don't conspire to take your fame & fortune and go to Las Vegas to gamble with other #AI
Fine tuning doesn't mean software development.
remember Bill Gates:
Here’s the legend: at a computer trade show in 1981, Bill Gates supposedly uttered this statement, in defense of the just-introduced IBM PC’s 640KB usable RAM limit: “640K ought to be enough for anybody.”
RAID 0 is not raid… the clue is in the name 😂😂😂😂
lol. i suppose we can just call it AID :)
@@AZisk I have at one time or another used:
Risky Arrangement Inviting Disaster
Really Awful Idea for Data
Reckless Architecture Ignoring Durability
Reliably Arranging Imminent Deletion
I guess professional video editors are the piggest ones
Wow, Alex DIDN'T get sponsored for this? Who'd you piss off man? Every other creator has one of these and they are all sponsored.😅
are you kidding me? if you download others along llm 2 tb is like a joke xD
Raid 0 is stupid your limited by usb not the drives .
second comment :)
Second!
Right off the bat, bro, Ironwolfs pro instead of exos? most probably you have overpaid dearly for inferior product...
they were pricey. the pros were recommended for das, why exos are better?
@@AZisk pretty much twice the MTBF, and twice allowed TB/year