*Summary* - 0:00 Introduction - 0:03 Artificial intelligence is transforming industries - 0:07 Access to AI technology has been limited by high costs and technical expertise - 0:16 GPU VRAM is an important factor for AI applications - 0:23 Consumer GPU price has been skyrocketing - 0:33 In 2022, a GPU with only 12 gigabytes of VRAM costs over $1500 - 0:45 The situation hasn't improved much in 2023 - 0:51 Demand for GPUs has been skyrocketing - 0:55 Nvidia has reached a 1 trillion dollar market cap and becomes the world's largest chip company - 1:01 Inflation and interest rates have been skyrocketing - 1:12 To democratize AI, we need both low cost consumer hardware and smart designed software - 1:30 Introducing AMD APU, a budget-friendly hardware solution - 1:45 AMD APU is a series of 64-bit macro processors combining CPU and GPU - 2:05 The least expensive AMD APU is 4600 G priced at 95 US dollars - 2:17 A slightly more expensive APU is 5600g which costs 127 US Dollars - 2:30 The GPU can be turned into a 16 gigabytes VRAM GPU - 2:53 System costs can be as low as 400 US dollars for all brand new components - 3:27 The power consumption is around 66 Watts for idle system and around 96 Watts at full usage - 3:51 The operation cost is about 2.4 US dollars for a month or 30 US dollars per year - 4:29 The system can run mainstream machine learning frameworks such as PyTorch and TensorFlow - 4:57 The system can run state-of-the-art AI applications - 6:00 The system has future expandability for more powerful discrete GPU - 6:23 A demo of running stable diffusion using command line - 7:55 A demo of automatic web UI running stable diffusion Positive Points: 1. Affordable: AMD APU is a cost-effective solution. The total system cost can be as low as 400 US dollars for all brand-new components. 2. Energy Efficient: The power consumption is minimal, around 66 Watts for an idle system and around 96 Watts at full usage. 3. Versatile: The system can run mainstream machine learning frameworks such as PyTorch and TensorFlow, as well as state-of-the-art AI applications. 4. Expandability: The system allows for future upgradeability to more powerful discrete GPUs without the need to replace any existing components. 5. High VRAM: The GPU can be turned into a 16 gigabytes VRAM GPU, providing substantial memory for AI applications. Negative Points: 1. Limited Availability: Access to AI technology has been limited by high costs and technical expertise. While the AMD APU is a more affordable option, availability could still be an issue for some. 2. High Demand: The demand for GPUs has been skyrocketing, which could potentially lead to supply issues. 3. Inflation and Rising Interest Rates: The rising inflation and interest rates could potentially increase the costs of these components. 4. Limited Processing Power: While AMD APU is a cost-effective solution, its processing power may not match up to the more expensive, high-end GPUs in the market. 5. Requires Technical Knowledge: While the system allows for future upgrades, this still requires a certain level of technical knowledge and expertise.
Meanwhile, I'm just simply use DirectML version of Stable Diffusion. It just really works on AMD iGPUs. No hacker-level tricks, just configure the iGPU to the bigger VRAM and set up the Stable Diffusion DirectML version. Although it works, but slow, at least little bit faster than running it on the CPU. It needs at least 8GB of VRAM and of course, you will need at least 32GB of system RAM modules installed on your computer.
yes, that's correct. DirectML is good for several tasks such as Stable diffusion! However it's limited compared to use Pytorch directly. To run most of Pytorch code, you would need ROCm on Linux.
I bought 2 P40's ($175 each used) and cannot get them to work in my workstation. Shame the workstation has 512 GB of ram. And each of these cards has 24GB vram. But dell workstations purposely doesn't support memory ReBar... so that you have to buy a quadro p6000 ($1200 used) instead to get the same thing. Dell is the worst. Nvidia hates us too.
Excellent presentation. I had given up on using my two 4060 Tis until I saw your video on how to allocate VRAM per GPU via command. Keep forging ahead, you'll make a difference for sure.
i'm looking at your terminal, it's still using torch 1.13 + ROCm 5.2 , i think if you updated with torch 2.0 + ROCm 5.6 it'll give 10-15% performance boost (maybe more, since 10-15% boost is for ROCm 5.6 when upgrade from 5.5)
Thanks, that's great observation! I previously tried Pytorch 2.0, but at the time there may be bugs so it generated black images. It probably got fixed by now.
just wondering, if i have 128GB of RAM, and utilizing 50% to iGPU, so i'll have 64GB of "iGPU VRAM" right? With this i think train SDXL or LLM will be much much cheaper (although it's slower) than nVidia (which need at least A100 to achieve similar VRAM size)
Good question! I would like to know that too. 128GB of RAM is available for $400+, so it's really "cheap" compared to a $3000+ GPU with 40GB of VRAM. I would upgrade my RAM to 128GB and get a Ryzen 5700G and I have 64GB VRAM? That would be awesome. 5700G is also slightly faster than 4600G shown in the Video. Maybe 10% I think.
@@clarissamarsfiels7961 Sadly no , For now, some motherboards company's ( like ASRock ) tap out @ 16Gb allocation, to the GPU part of your APU. There are even motherboard brands, that give you, an max. of just 4Gb allocation, to the GPU ! So ..... UNLESS some motherboard company, or an good modder ( bios hacker ) can make an new bios, that allows more than 16Gb, to the GPU, It just wouldn't be possible ! Even an 16Gb allocation is quite rare, I only know one motherboard brand that does this, and that is ASRock !
Sadly no , For now, some motherboards company's ( like ASRock ) tap out @ 16Gb allocation, to the GPU part of your APU. There are even motherboard brands, that give you, an max. of just 4Gb allocation, to the GPU ! So ..... UNLESS some motherboard company, or an good modder ( bios hacker ) can make an new bios, that allows more than 16Gb, to the GPU, It just wouldn't be possible ! Even an 16Gb allocation is quite rare, I only know one motherboard brand that does this, and that is ASRock !
I'd love for ROCm to be integrated into TrueNAS Scale I'd love to use my 7950x file server's iGPU to not only transcode live TV for Plex(for playack only, long term storage CPU is way better than GPU when going from MPEG2 to H265 and AV1), but also for image recognition through something like PhotoPrism Dont get me wrong, the 7950x is an amazing CPU i can record and transcode in real time 6 channels while using only 75w at the socket using Plex DVR and TDARR with X265 CPU transcoding for ~1-2rd to 1/5th file size depending on efficiency of source.
Now lets wait for Quad memory channel next gen BIG APUs, some leaks show AMD developing them but only for mobile market. Depends to configuration you will have 256GB RAM and 128GB for iGPU possible.
Ultra helpful video, I've been waiting to confirm this 'list of tested applications!' You know about minu pc's right? Cheapest build, least hassle, and more power efficent!
@@tech-practice9805 Yes. The SER6 is a great pick right now. Beelink SER6 1Tb is THE best bang for your buck on the market when it drops to $420 and below. It has two ram already holding 32gb, and has two m.2 slots: One filled with a 1tb m.2. AND it's thunderbolt port can take on a whole external GPU!!! I don't even think you can get more upgradeable at this rate. Eventually it's maxed out and you just leave it running for hopefully eternity. Keep buying the next most efficient installments of mini-PC's every 1-3 years. Very power effective, and easy to build and manage server rooms!
@@tech-practice9805 Fourtunately, my dad has done some research 👨 👏 he bought my sister a 6600H APU mini PC for her birthday, and yestwrdsy did a good comparing of PC's. We're eyeing the the SER6 MAX mini PC (32gb ram/1tb M.2) at prices about $410, untill like next year the SER7 MAX drops under $450, it becomes the most effective PC for the price. A quote from my dad: "... 6600H is 6c/12 thread and has inferior 660M rdna2 graphics with just 6CU enabled. The 6800H and 7735HS are 8c/16 thread with 680M graphics for 12CU rdna2. That's probably 50% more fps on average in gaming. So the 6600H is more like a 900p@30fps LOW gaming machine, while the 6800H/7735HS is a 1080p@30 LOW gaming machine in the SER6 Max... as showcased by TechDweeb." And, he also recomends another slightly cheaper mini PC, if you plan on buying based on availability. I'll paste the specs from his sheet (converted from CSV): If priced at ~$380: Name: GMKTech K1 ; passmark: 23,801 // Locked TDP and RAM speed; GMKTech cuts corners on cooling so 45W TDP is also the realistic max. // CPU: 6800H / iCPU: 680M / gen: 12CU RDNA 2 // DRAM 16GB / or Max 64GB DDR5 (speed) 4800 max. 1TB / 2TB 1x PCIe 3.0x4 No swcond Nvme slot, there is a 2.5" HDD slot. 3 displays / 2x HDMI 2.1 // 45W TDP. Idle/game/max = 16/58/79W // x3 USB A 3.2 & x1 USB C 4.0 % x1 USB 2.0
So you are saying we can game on iGPUs now on CPUs from both Intel and AMD or just AMD alone? Like playing old console games on PS1 (Duckstation) /PS2 (PCSX2) /PS3 (RPCS3) emulators on Desktop PC?
You could always play games on every iGPU, but not very demanding ones. New(er) AMD iGPUs have been really powerful, much more than Intel's. You can easily play PS1 and PS2 emulators on new iGPUs. Don't know about the PS3 one because as far as I know it's a bit resource intensive. CZcams search is your friend.
@@92kosta I think PS3 games can be played on AMD's APU G-series like the 5700G's Vega 8 iGPU with 8CUs (Core Units), almost equivalent to a GTX 770. correct me if i'm wrong
I play DayZ on one and get a steady playable 50-60fps on a low/med mix of custom settings Here's RPCS3 emulator on 5600G. Looks pretty bad. czcams.com/video/YmEGIyUrw-M/video.html Wouldnt expect much at all from newer, flashier titles. I never bother with them so couldnt tell you about that but running UT2004 on max settings at 240fps is still very fun. edit: here's 40 games running on a 5600G (no GPU used) with only 16gb Ram czcams.com/video/hCbybCYLJog/video.html
Thanks for your insight! Found yout article on Medium and came here for details I'm still tryiyng to wrap my brain around running AI models locally. Do you think it will work also on Intel iGPUs or is there any technical limitation?
Hi! Intel iGPUs are weaker than AMD ones. Also they use different software stacks which I don't have experience with. But ideally I hope the iGPUs can all be used for local AI.
@@tech-practice9805 Thanks for reply! I'm aware that Intel's iGPUs perform worse that AMD's but I'm already on Intel platform, and I don't want to change for now. My GPU has only 6 GB of VRAM, so I cannot run most of AI models. So far I've been using online things but I'm really excited about running it locally. I'm saving for better GPU but maybe try experimenting with running something on Intel's iGPU, if I have some time to spare. Thanks for your work!
How does this perform for running LLMs? I have an M1 Max Macbook that is actually faster, and can run larger LLM models than my 7900XTX because the integrated GPU uses system RAM as vram. If an AMD apu can do something similar, it would be a much cheaper way to getting large models running. I can run them in CPU mode on PC, but it's 1/10th to 1/100th the speed of my Macbook. It would be nice if I could just get an AMD APU, slap 128 or 256GB of RAM on it, and get 1/2 the performance of an M1 Max.
Did you follow the same installation procedure for this as your AMD GPU video with the dedicated card? I have been researching and attempting to get this going on a 5650GE to run whisper for home automation, but I had trouble installing on my system. I will be following your guide on Ubuntu 22.x to test if it works, then attempting to run that on top of Proxmox in the future.
Sorry about it. I have few posts about it also if that's easier: medium.com/@ttio2tech_28094/democratize-ai-turn-a-95-chip-into-a-16gb-vram-gpu-beats-most-of-the-discrete-gpus-51a8636a4719
Very skeptical about this. Stable diffusion 1.5 requires 4GB of VRAM. You can get a 4GB graphics card for very little money and it will be faster than this. For larger models this CPU will need so much time it would practically be unusable. And the budget 4GB graphics card will still completely outperform it anyway.
@@songyang80 CPU-only is not the way to go. I mean you can get an A100 at this point lol. Pair it with a cutting edge SSD and you will probably get orders of magnitude better speed. I would argue that a 4080 alone can beat this. GPUs are just better at matrix operations. That's their purpose.
Thank you for this insightful tip. Maybe help out and make a bit of money by providing a list of affiliate links to all the parts in this build? Everybody that watches this video is going to want to see the parts used and some will use your links to purchase.
It should work, but the question is how much RAM will that ASUS motherboard allow you to allocate to the iGPU. Some motherboards top out at 4GB of VRAM.
@@tech-practice9805 Do have any examples for the speed? Benchmarks? Token per seconds? Did you try it with a 13B or bigger one? 16GB or 64GB? How does it compare to CPU inference? Maybe make another video about it.
All AI tasks will work on CPU. (In principal at least... Somebody still has to write a script for it.). You just have to be willing to wait. Even As fast as modern CPU cores are, there's still at most a dozen or two available. GPU'S have thousands. That's why on CPU an image generation takes three minutes, whereas on GPU it's closer to three seconds.
@@Techonsapevole That's pretty dang quick for CPU inference. What CPU did you use to get those times? I'm genuinely curious. I've been an advocate for repurposing old server GPU'S for hobbyist use. Nvidia Tesla M40'S came with 12 or 24GB of Vram, they can crank out Stable Diffusion images at about 12s per image, and they're getting CHEAP since data centers can't afford to feed whole racks of them at 250W each. That said, if power efficiency is your goal, then that's not gonna be the way.
More VRAM is not always better :) Your render from 11:00 took you 97 seconds. While my GTX 1080 with "only" 8GB of vram did the same render in 24 seconds. This video is pretty misleading as someone might assume that with 16 gb of vram he/she can do high res renders / batch renders / local lora training. Technically yes. But he/she would have to spend weeks waiting for this APU to do the job.
Yeah like, I think this is rad as hell because of the way the GPU market is right now, it's awesome that people can do this, but my 3090ti does that workload in literally 1 second. ONE. I'm usually doing 50-150 iterations, sometimes more, and at higher resolutions with high res fix on top. I suppose if I was coming from having nothing and wanting to gain admission at all, this would be cool, but if I had to go from this card to an APU like this, I'd likely just quit, tbh.
Few observations from your video: 1) you have a lot of "noise" on CPU activity side. Even before you actually started inference, there were like 30-40% CPU used, just visually from Gnome System Monitor. This is no good. You might be running some extra heavy shit on a background which is interfering with your test. OBS? 2) before jumping to a conclusion like "hey just buy Ryzen APU and allocate more RAM to be used by iGPU", you should have made the same inference test on CPU only. I do own couple of Ryzen powered machines and the least powerful 3400g is having pretty much same figures (2 minutes for one image generation in SD 1.5 via a111) using CPU only. And my other beefy 7940HS is great at inferencing Mixtral 8x7B 4 bits quantized, giving me like 10-15 tokens per second in llama.cpp gguf.
Hi, i'm using Ryzen 4650G CPU (same Vega 7 iGP as 4600G), with Ubuntu 22.04 and ROCm 5.4.2, HSA_OVERRIDE_GFX_VERSION=9.0.0 . all installation is successfull, but all generated image are black. do you know what's wrong with it? i'm also trying ROCm 5.6, but it gives me error about HIP kernel and checkpoint model did'nt loaded successfully
@@tech-practice9805 i am using A1111 webui. as you said in other comment, that torch 2+ROCm5.4.2 is give black image result. it’s same as this problem. do you successfully running A1111 SD webui with torch 2 + ROCm 5.4.2 / 5.6?
16 GB isn't really enough unless you are okay with taking those HUGE accuracy losses and slow replies... Just isnt that good. I have 64GB and still need way more.. You want at least 128 for GOOD LLM fun. You REALLY want like a quadro or a pascal40 but those are too hard to get working on anything but very specific hardware. Artificial lockouts against workstations...Some people get them to work in certain gaming boards but then you are still limited by the maximum memory of gaming borads at usually 64GB... They have intentionally crippled ram on new systems also. soldered in, or capping new boards at 32 GB... they don't really want you messing with these LLM's in any REAL capacity... Not unless you got money. Or corpo/militech/government funding.
That's pretty awesome! Love to see it. We've been testing different hardware as well, I hope to post a video soon about it
Subbed, hope to see something soon!!
*Summary*
- 0:00 Introduction
- 0:03 Artificial intelligence is transforming industries
- 0:07 Access to AI technology has been limited by high costs and technical expertise
- 0:16 GPU VRAM is an important factor for AI applications
- 0:23 Consumer GPU price has been skyrocketing
- 0:33 In 2022, a GPU with only 12 gigabytes of VRAM costs over $1500
- 0:45 The situation hasn't improved much in 2023
- 0:51 Demand for GPUs has been skyrocketing
- 0:55 Nvidia has reached a 1 trillion dollar market cap and becomes the world's largest chip company
- 1:01 Inflation and interest rates have been skyrocketing
- 1:12 To democratize AI, we need both low cost consumer hardware and smart designed software
- 1:30 Introducing AMD APU, a budget-friendly hardware solution
- 1:45 AMD APU is a series of 64-bit macro processors combining CPU and GPU
- 2:05 The least expensive AMD APU is 4600 G priced at 95 US dollars
- 2:17 A slightly more expensive APU is 5600g which costs 127 US Dollars
- 2:30 The GPU can be turned into a 16 gigabytes VRAM GPU
- 2:53 System costs can be as low as 400 US dollars for all brand new components
- 3:27 The power consumption is around 66 Watts for idle system and around 96 Watts at full usage
- 3:51 The operation cost is about 2.4 US dollars for a month or 30 US dollars per year
- 4:29 The system can run mainstream machine learning frameworks such as PyTorch and TensorFlow
- 4:57 The system can run state-of-the-art AI applications
- 6:00 The system has future expandability for more powerful discrete GPU
- 6:23 A demo of running stable diffusion using command line
- 7:55 A demo of automatic web UI running stable diffusion
Positive Points:
1. Affordable: AMD APU is a cost-effective solution. The total system cost can be as low as 400 US dollars for all brand-new components.
2. Energy Efficient: The power consumption is minimal, around 66 Watts for an idle system and around 96 Watts at full usage.
3. Versatile: The system can run mainstream machine learning frameworks such as PyTorch and TensorFlow, as well as state-of-the-art AI applications.
4. Expandability: The system allows for future upgradeability to more powerful discrete GPUs without the need to replace any existing components.
5. High VRAM: The GPU can be turned into a 16 gigabytes VRAM GPU, providing substantial memory for AI applications.
Negative Points:
1. Limited Availability: Access to AI technology has been limited by high costs and technical expertise. While the AMD APU is a more affordable option, availability could still be an issue for some.
2. High Demand: The demand for GPUs has been skyrocketing, which could potentially lead to supply issues.
3. Inflation and Rising Interest Rates: The rising inflation and interest rates could potentially increase the costs of these components.
4. Limited Processing Power: While AMD APU is a cost-effective solution, its processing power may not match up to the more expensive, high-end GPUs in the market.
5. Requires Technical Knowledge: While the system allows for future upgrades, this still requires a certain level of technical knowledge and expertise.
Meanwhile, I'm just simply use DirectML version of Stable Diffusion. It just really works on AMD iGPUs. No hacker-level tricks, just configure the iGPU to the bigger VRAM and set up the Stable Diffusion DirectML version.
Although it works, but slow, at least little bit faster than running it on the CPU. It needs at least 8GB of VRAM and of course, you will need at least 32GB of system RAM modules installed on your computer.
yes, that's correct. DirectML is good for several tasks such as Stable diffusion! However it's limited compared to use Pytorch directly. To run most of Pytorch code, you would need ROCm on Linux.
I bought 2 P40's ($175 each used) and cannot get them to work in my workstation. Shame the workstation has 512 GB of ram. And each of these cards has 24GB vram. But dell workstations purposely doesn't support memory ReBar... so that you have to buy a quadro p6000 ($1200 used) instead to get the same thing. Dell is the worst. Nvidia hates us too.
actual demo is at 6:22
Excellent presentation. I had given up on using my two 4060 Tis until I saw your video on how to allocate VRAM per GPU via command. Keep forging ahead, you'll make a difference for sure.
thanks for sharing with us
Thanks A.I. will leap forward with inovations like this
i'm looking at your terminal, it's still using torch 1.13 + ROCm 5.2 , i think if you updated with torch 2.0 + ROCm 5.6 it'll give 10-15% performance boost (maybe more, since 10-15% boost is for ROCm 5.6 when upgrade from 5.5)
Thanks, that's great observation! I previously tried Pytorch 2.0, but at the time there may be bugs so it generated black images. It probably got fixed by now.
This is really great! Thanks for sharing...
Good video. Would be better without the loud background music.
just wondering, if i have 128GB of RAM, and utilizing 50% to iGPU, so i'll have 64GB of "iGPU VRAM" right? With this i think train SDXL or LLM will be much much cheaper (although it's slower) than nVidia (which need at least A100 to achieve similar VRAM size)
Good question! I would like to know that too. 128GB of RAM is available for $400+, so it's really "cheap" compared to a $3000+ GPU with 40GB of VRAM. I would upgrade my RAM to 128GB and get a Ryzen 5700G and I have 64GB VRAM? That would be awesome. 5700G is also slightly faster than 4600G shown in the Video. Maybe 10% I think.
Was thinking about the same thing!!@@clarissamarsfiels7961
@@clarissamarsfiels7961 Sadly no , For now, some motherboards company's ( like ASRock ) tap out @ 16Gb allocation, to the GPU part of your APU. There are even motherboard brands, that give you, an max. of just 4Gb allocation, to the GPU ! So ..... UNLESS some motherboard company, or an good modder ( bios hacker ) can make an new bios, that allows more than 16Gb, to the GPU, It just wouldn't be possible ! Even an 16Gb allocation is quite rare, I only know one motherboard brand that does this, and that is ASRock !
Sadly no , For now, some motherboards company's ( like ASRock ) tap out @ 16Gb allocation, to the GPU part of your APU. There are even motherboard brands, that give you, an max. of just 4Gb allocation, to the GPU ! So ..... UNLESS some motherboard company, or an good modder ( bios hacker ) can make an new bios, that allows more than 16Gb, to the GPU, It just wouldn't be possible ! Even an 16Gb allocation is quite rare, I only know one motherboard brand that does this, and that is ASRock !
I hope they can increase it after seeing my video :D it can definitely help their sales!
I'd love for ROCm to be integrated into TrueNAS Scale
I'd love to use my 7950x file server's iGPU to not only transcode live TV for Plex(for playack only, long term storage CPU is way better than GPU when going from MPEG2 to H265 and AV1), but also for image recognition through something like PhotoPrism
Dont get me wrong, the 7950x is an amazing CPU i can record and transcode in real time 6 channels while using only 75w at the socket using Plex DVR and TDARR with X265 CPU transcoding for ~1-2rd to 1/5th file size depending on efficiency of source.
Now lets wait for Quad memory channel next gen BIG APUs, some leaks show AMD developing them but only for mobile market. Depends to configuration you will have 256GB RAM and 128GB for iGPU possible.
that's exciting if it's true!
Ultra helpful video, I've been waiting to confirm this 'list of tested applications!' You know about minu pc's right? Cheapest build, least hassle, and more power efficent!
I heard about mini PC, but are they customizable?
@@tech-practice9805 Yes. The SER6 is a great pick right now. Beelink SER6 1Tb is THE best bang for your buck on the market when it drops to $420 and below. It has two ram already holding 32gb, and has two m.2 slots: One filled with a 1tb m.2. AND it's thunderbolt port can take on a whole external GPU!!! I don't even think you can get more upgradeable at this rate. Eventually it's maxed out and you just leave it running for hopefully eternity. Keep buying the next most efficient installments of mini-PC's every 1-3 years. Very power effective, and easy to build and manage server rooms!
@@tech-practice9805 Fourtunately, my dad has done some research 👨 👏 he bought my sister a 6600H APU mini PC for her birthday, and yestwrdsy did a good comparing of PC's.
We're eyeing the the SER6 MAX mini PC (32gb ram/1tb M.2) at prices about $410, untill like next year the SER7 MAX drops under $450, it becomes the most effective PC for the price.
A quote from my dad: "... 6600H is 6c/12 thread and has inferior 660M rdna2 graphics with just 6CU enabled. The 6800H and 7735HS are 8c/16 thread with 680M graphics for 12CU rdna2. That's probably 50% more fps on average in gaming. So the 6600H is more like a 900p@30fps LOW gaming machine, while the 6800H/7735HS is a 1080p@30 LOW gaming machine in the SER6 Max... as showcased by TechDweeb."
And, he also recomends another slightly cheaper mini PC, if you plan on buying based on availability. I'll paste the specs from his sheet (converted from CSV):
If priced at ~$380: Name: GMKTech K1 ; passmark: 23,801 // Locked TDP and RAM speed; GMKTech cuts corners on cooling so 45W TDP is also the realistic max. // CPU: 6800H / iCPU: 680M / gen: 12CU RDNA 2 // DRAM 16GB / or Max 64GB DDR5 (speed) 4800 max. 1TB / 2TB 1x PCIe 3.0x4 No swcond Nvme slot, there is a 2.5" HDD slot. 3 displays / 2x HDMI 2.1 // 45W TDP. Idle/game/max = 16/58/79W // x3 USB A 3.2 & x1 USB C 4.0 % x1 USB 2.0
Good to know that my work laptop m1 max is amazingly good with AI
From what I understand, it takes 16GB out of the box, right? Did you need to configure any kernel boot parameters for that?
++
That's configurable in UEFI
It also requires specifik motherboard and bios to allow Uma 16
Is it possible to run with 16 gb ram?
Thanks for the smart method :)
Thanks for putting in the effort.
deserve a subscribe
thank you!
So you are saying we can game on iGPUs now on CPUs from both Intel and AMD or just AMD alone? Like playing old console games on PS1 (Duckstation) /PS2 (PCSX2) /PS3 (RPCS3) emulators on Desktop PC?
I would say that AMD iGPU is more powerful than Intel's. iGPUs can be used to play those.
If from Intel GPU, you might want to give a try with Stable Diffusion DirectML version.
You could always play games on every iGPU, but not very demanding ones. New(er) AMD iGPUs have been really powerful, much more than Intel's. You can easily play PS1 and PS2 emulators on new iGPUs. Don't know about the PS3 one because as far as I know it's a bit resource intensive.
CZcams search is your friend.
@@92kosta I think PS3 games can be played on AMD's APU G-series like the 5700G's Vega 8 iGPU with 8CUs (Core Units), almost equivalent to a GTX 770. correct me if i'm wrong
I play DayZ on one and get a steady playable 50-60fps on a low/med mix of custom settings
Here's RPCS3 emulator on 5600G. Looks pretty bad.
czcams.com/video/YmEGIyUrw-M/video.html
Wouldnt expect much at all from newer, flashier titles. I never bother with them so couldnt tell you about that but running UT2004 on max settings at 240fps is still very fun.
edit: here's 40 games running on a 5600G (no GPU used) with only 16gb Ram
czcams.com/video/hCbybCYLJog/video.html
Thanks for your insight! Found yout article on Medium and came here for details
I'm still tryiyng to wrap my brain around running AI models locally. Do you think it will work also on Intel iGPUs or is there any technical limitation?
Hi! Intel iGPUs are weaker than AMD ones. Also they use different software stacks which I don't have experience with. But ideally I hope the iGPUs can all be used for local AI.
@@tech-practice9805 Thanks for reply! I'm aware that Intel's iGPUs perform worse that AMD's but I'm already on Intel platform, and I don't want to change for now. My GPU has only 6 GB of VRAM, so I cannot run most of AI models. So far I've been using online things but I'm really excited about running it locally.
I'm saving for better GPU but maybe try experimenting with running something on Intel's iGPU, if I have some time to spare. Thanks for your work!
How does this perform for running LLMs? I have an M1 Max Macbook that is actually faster, and can run larger LLM models than my 7900XTX because the integrated GPU uses system RAM as vram. If an AMD apu can do something similar, it would be a much cheaper way to getting large models running. I can run them in CPU mode on PC, but it's 1/10th to 1/100th the speed of my Macbook. It would be nice if I could just get an AMD APU, slap 128 or 256GB of RAM on it, and get 1/2 the performance of an M1 Max.
I am wondering the exact same thing (got a M1 Max too) but i can't get an answer and CZcams videos about the topic seems non-existent
Did you follow the same installation procedure for this as your AMD GPU video with the dedicated card? I have been researching and attempting to get this going on a 5650GE to run whisper for home automation, but I had trouble installing on my system. I will be following your guide on Ubuntu 22.x to test if it works, then attempting to run that on top of Proxmox in the future.
Yes, installation is similar to dedicated cards. There is also a step by step guide for APU: czcams.com/video/H9oaNZNJdrw/video.html
888 subscribers. yay !
No clue what you are saying because I can't listen to that music. Maybe remove the music and re-upload.
Sorry about it. I have few posts about it also if that's easier: medium.com/@ttio2tech_28094/democratize-ai-turn-a-95-chip-into-a-16gb-vram-gpu-beats-most-of-the-discrete-gpus-51a8636a4719
Very skeptical about this. Stable diffusion 1.5 requires 4GB of VRAM. You can get a 4GB graphics card for very little money and it will be faster than this. For larger models this CPU will need so much time it would practically be unusable. And the budget 4GB graphics card will still completely outperform it anyway.
It's surprising the use cases now for a < 4GB small language model. Llama file also works well on AMD cpus
what about if you make 10+ PCs with 64GB VRAM to run large models at once 😂
@@songyang80 CPU-only is not the way to go. I mean you can get an A100 at this point lol. Pair it with a cutting edge SSD and you will probably get orders of magnitude better speed. I would argue that a 4080 alone can beat this. GPUs are just better at matrix operations. That's their purpose.
Thank you for this insightful tip. Maybe help out and make a bit of money by providing a list of affiliate links to all the parts in this build? Everybody that watches this video is going to want to see the parts used and some will use your links to purchase.
Thank you for the suggestions!
Great, if only something similar was implemented for NVDIA gpus...
Hope so. It will be interesting. I heard Nvidia will sell CPU, maybe combine with igpu
Which of the cheap mainboards can accommodate / allow the highest VRAM?
I listed some of them in this video czcams.com/video/H9oaNZNJdrw/video.html
@tech-practice9805 Will this work with 4650G? You mentioned Asrock board, any chance it will work on any of the Asus motherboard?
It should work, but the question is how much RAM will that ASUS motherboard allow you to allocate to the iGPU. Some motherboards top out at 4GB of VRAM.
I have an Asus MB and it works! The maximum is 16GB for iGPU.
That is so awesome! What's the total RAM you have on there? Did you have to flash a custom firmware on the motherboard?
will that work with similar Mobile APU like ryzen 4500u or R7 5700u ?
for laptops, the motherboard bios may be locked. It depends on the brand and can check with their manufacturers.
@@tech-practice9805 build your own. Don't buy dell or HP. You will NOT be accessing any of the UEFI stuff you need for ML research.
great work , chief
Thats a little slower than a $200 8GB RX7600 and a LOT slower than a similarly priced 8GB RTX3050...
good thing is you can make it up to 64GB VRam which you can run huge images, bad thing is it's too slow..
Nice idea, what is the name of motherboard you are using? thank you
thanks! I think most AM4 motherboard should work. I tested on MSI B450-A PRO MAX
Buy a secondhand 3060 12GB, it's cheaper than this, much faster, and will work without issues with anything, since it's Nvidia
Wince. That is. morbidly slow. would have been better to just slap an old 1070 in a workstation. would have been cheaper too.
is this work for LLaMa 2 - 13B?
No, 13b is too big for a machine like this. You would struggle to get a 7B to run faster than molasses.
Has this been tested with LLMs? Where is the best place to find more people trying this?
yes, it works but inferencing is slow
@@tech-practice9805
Do have any examples for the speed? Benchmarks? Token per seconds? Did you try it with a 13B or bigger one? 16GB or 64GB? How does it compare to CPU inference? Maybe make another video about it.
Does SDXL work on CPU ?
All AI tasks will work on CPU. (In principal at least... Somebody still has to write a script for it.). You just have to be willing to wait. Even As fast as modern CPU cores are, there's still at most a dozen or two available. GPU'S have thousands. That's why on CPU an image generation takes three minutes, whereas on GPU it's closer to three seconds.
@@DerrangedGadgeteer with sdxl on cpu it works with 32gb ram it takes 60s with sd1.5 and 90s with sdxl for a 512px image
@@Techonsapevole That's pretty dang quick for CPU inference. What CPU did you use to get those times? I'm genuinely curious.
I've been an advocate for repurposing old server GPU'S for hobbyist use. Nvidia Tesla M40'S came with 12 or 24GB of Vram, they can crank out Stable Diffusion images at about 12s per image, and they're getting CHEAP since data centers can't afford to feed whole racks of them at 250W each. That said, if power efficiency is your goal, then that's not gonna be the way.
More VRAM is not always better :)
Your render from 11:00 took you 97 seconds. While my GTX 1080 with "only" 8GB of vram did the same render in 24 seconds.
This video is pretty misleading as someone might assume that with 16 gb of vram he/she can do high res renders / batch renders / local lora training. Technically yes. But he/she would have to spend weeks waiting for this APU to do the job.
Yeah like, I think this is rad as hell because of the way the GPU market is right now, it's awesome that people can do this, but my 3090ti does that workload in literally 1 second. ONE. I'm usually doing 50-150 iterations, sometimes more, and at higher resolutions with high res fix on top. I suppose if I was coming from having nothing and wanting to gain admission at all, this would be cool, but if I had to go from this card to an APU like this, I'd likely just quit, tbh.
I am glad you can afford to high price and powerful cards!
Few observations from your video:
1) you have a lot of "noise" on CPU activity side. Even before you actually started inference, there were like 30-40% CPU used, just visually from Gnome System Monitor. This is no good. You might be running some extra heavy shit on a background which is interfering with your test. OBS?
2) before jumping to a conclusion like "hey just buy Ryzen APU and allocate more RAM to be used by iGPU", you should have made the same inference test on CPU only. I do own couple of Ryzen powered machines and the least powerful 3400g is having pretty much same figures (2 minutes for one image generation in SD 1.5 via a111) using CPU only. And my other beefy 7940HS is great at inferencing Mixtral 8x7B 4 bits quantized, giving me like 10-15 tokens per second in llama.cpp gguf.
I’m using stable diffusion web ui , but can only select my cpu (4600G), I can’t select igpu (vega7) , can you help me?
Hi, i'm using Ryzen 4650G CPU (same Vega 7 iGP as 4600G), with Ubuntu 22.04 and ROCm 5.4.2, HSA_OVERRIDE_GFX_VERSION=9.0.0 . all installation is successfull, but all generated image are black. do you know what's wrong with it? i'm also trying ROCm 5.6, but it gives me error about HIP kernel and checkpoint model did'nt loaded successfully
which stable diffusion program were you using? it may because of half precision
@@tech-practice9805 i am using A1111 webui. as you said in other comment, that torch 2+ROCm5.4.2 is give black image result. it’s same as this problem. do you successfully running A1111 SD webui with torch 2 + ROCm 5.4.2 / 5.6?
@@ZulfikarAjiKusworo I will upload a new video soon which seems working with torch 2
@@tech-practice9805 nice. i’ll be wait
can you game with this?
Yes, I can run some benchmarking for it
@@tech-practice9805 I would absolutely LOVE that
What we do with this AI application ? earning money ?
anything AI can do! including earning money!
You aren't really allowed to use AI to make money. Unless its your own trained model. I'm sure thats not stopping anybody though.
what is the motherboard and its bios version ?
I will upload details soon. stay tuned! thanks!
@@tech-practice9805soon?
dude the info is good and all but you need to consolidate the info in a single video, like it is right now is a huge mess D:
Sorry about that
How to install those AI models?
I will upload a detailed tutorial. please subscribe to stay tuned!
3400G?
It can work. But for old generations, the motherboard bios may not support it. So it depends on the brand of motherboard and need to test
@@tech-practice9805 awesome. I have one idling around
16 GB isn't really enough unless you are okay with taking those HUGE accuracy losses and slow replies... Just isnt that good. I have 64GB and still need way more.. You want at least 128 for GOOD LLM fun. You REALLY want like a quadro or a pascal40 but those are too hard to get working on anything but very specific hardware. Artificial lockouts against workstations...Some people get them to work in certain gaming boards but then you are still limited by the maximum memory of gaming borads at usually 64GB... They have intentionally crippled ram on new systems also. soldered in, or capping new boards at 32 GB... they don't really want you messing with these LLM's in any REAL capacity... Not unless you got money. Or corpo/militech/government funding.
waste of money and time. you're way better off buying modded 3060/3070 with 12GB/16GB VRAM from taobao for around 200 bucks
As mentioned in the title: "democratize AI". Assuming lots of people cannot afford discrete GPUs.
Waste of time and money, just buy a $95 4600G.
@@jordanhofer7376nah, i use 3090ti
No, you can now run Stable Diffusion on Ryzen laptops.. Can now run SD on machines that are not supported for AI.
@@lightningrooodScrew those comsumer level gaming GPUs, use Nvidia RTX A4000/Teslas etc. for better Stable Diffusion memory handling.