SQUEAKY WHEEL GETS THE GREASE: gigabyte aorus elite ax working now with f30 bios, see wendells tweet. Go team!! x.com/tekwendell/status/1806160769723076873
Hi, I have a question about the 128 gb with unregistered udimms in ddr5 (without or with ecc, would like without for my system, Intel...) Where can you get 64gb single stick ddr5 Memory to get 128gb with only 2 dimms? Had 4 32gb sticks and couldn't go over 4800 mt/s, with 2 sticks I can use 5600 mt/s and also 6000 mt/s without problems, 6000 mt/s only with xmp (so would only use 5600 on 128 gb with 2 sticks, xmp on ddr5 is just a mess mess, but since I can't use my 128 gb, I want at least the speed from 2 sticks for 64 gb) Have seen no release news, only working on it news from crucial, at least in Germany and looking at English press I see through different channels and Google and CZcams videos... Is it available as udimms right now or where you talking about the upcoming things not released yet or about rdimms? Can't find 64 dimms, Google only shows me 2x32gb sticks udimm ddr5 non ecc or udimm ecc, only registered ecc 64 gb ddr5.
I hope that complete ECC memory support will someday become a standard feature everywhere. Reminder: Regular DDR5’s On-Die ECC is just a tool to be able to use lower quality DRAM ICs that would otherwise have been discarded, it has nothing to do with the complete ECC chain checking the data between the DIMMs and the CPU’s memory controller.
I have a theory that ECC is kept for Workstation/HEDT class and up, to cause more people to go for higher-end hardware. Full ECC is the single biggest chunk of differentiation. PCIe lanes are a big point too, but since you have some PCIe lanes already, more becomes moar, while full ECC is a clear yes/no.
@@coolhand4598nah it's just a cost thing. Why would the consumer care to pay for ECC when they are running windows. Windows breaks itself without any ram error implication
> "Regular DDR5’s On-Die ECC is just a tool to be able to use lower quality DRAM ICs that would otherwise have been discarded" This is just wrong, on-die ECC functionality does not come free and it's much cheaper to just not have it and bin the DRAM chips to lower frequencies on modules. On-die ECC is required because of RAM densities reached with DDR5, the smaller the cells the easier it is to flip them accidentally
@@SquintyGears Are you aware that servers run Windows too? By that logic "eh it will break itself anyway, who cares". Same in hard drives, why have the ECC logic in those (which they have btw), Windows will break on its own anyway
When I heard that part I was like: oh cool, so Cyclic Redundancy Check (CRC) for both RD/WR data is mandatory now? Judging by your comment it isn't? Or did you just mean that it's not end to end protected?
Registered ECC should be the standard for all computing today. The fact that JEDEC member companies still view security and data integrity as premium features for only certain classes of consumers is a total ethical failure.
After 15+ years of pc building I still don't fully understand buffered memory or why even after all these years of DDR improvements the benefit of that additional register isn't simply part of the specification. I understand the latency benefits being unbuffered and why this can sometimes be preferable but why is this driver still necessary for such significant increases in capacity per dimm? On top of this why are RDIMMs no longer pin compatible with UDIMMs? It just seems like we're going backwards. I think there's input voltage differences with DDR5 for some reason but it's all still dropped down by the PMIC. Perhaps this just ignorance of the situation but feel like at this point we could simply have one specification that supports all primary configurations, with some limitations obviously due to what the board and specific CPU can technically handle.
@@JJFX- there are two ways that RDIMMs reach those high capacities. The first is to use x4 chips (4 data bits) instead of the x8 and x16 chips used in UDIMMs. This will double the number of chips on the command and address (CA) bus. All those connections soak up signal power and add stubs to the bus lines, reducing signal integrity. DDR5 desktop PCs already sacrifice some performance to support two DIMMs per channel, so the memory controllers would have to be made twice as powerful, which is not cost effective for a rare use case. The other way to increase capacity is to stack the dice in each IC package, they've gone up to eight dice in one package. There's no way an integrated memory controller can directly drive all those chips. The register delays the commands by one clock cycle, a fraction of a nanosecond. This isn't really noticeable when accesses still take ~60 ns. Including a register on low-capacity UDIMMs which can have as few as four DRAM ICs is also not cost effective. The register would be a large fraction of the total costs. The combination of x4 chips and stacked memory means the DIMMs can use a lot of power, hence 12V for RDIMM v. 5V for UDIMM, and the incompatible connectors.
13:10 Has anyone seen 64 GB per DIMM unbuffered memory in the wild, yet? It’s been half a year since motherboard manufacturers qualified these sticks (Kingston modules with Micron DRAM ICs).
I have asked him under his comment, comment disapiered without an answer. My standalone comment is still there. Have only seen the announcement news. If you find out something, please let me know. Have a Intel system, need unbuffered unregistered udimm with 64 gb sticks for my system with 128gb on intel, so without ecc, but also interested for servers for home. Had 4 sticks 32gb, but couldnt go over 4800, 5600 with xmp but crashes sometimes. Is a gaming and work system. With 2 sticks I can go to 5600 without xmp, no crash. Now run 2 32gb sticks in 6000 with xmp, doesnt crash but I don't like that I have xmp, but if I can't have 128gb I want that speed. so with 2 sticks a 64gb I would go no xmp and 5600 because that's enough speed and enough memory for bigger Projekts (learning coding, 3d printing and cad, and want to learn video editing) so please let me know if you see something, best wishes from Germany.
@@kleingezockt5923 What exactly do you mean by with&without xmp, you know it's just a setting profile, right? It's just a preset, like the normal JEDEC SPD settings but faster&tighter timings usually. Nothing more. Like in the old DDR days you could have a stick with JEDEC SPD 2.5-3-3-7 for example and you'd have a XMP that is intel validated for some CPU to instead run 2-2-2-7 or similar. There is nothing else to XMP other than a profile of settings that the manufacturer claims to run on certain Intel processors with tighter&faster settings than the JEDEC SPD validated settings. So you can check what settings XMP sets usually in BIOS, and try to back them off a little, like a clock or more for row refresh or something if it looks too optimistic, and see if it holds without crashes, then use that profile as a manual entry if it works. XMP settings routinely only work with one DIMM per channel, if you have more that is likely the issue, also typically work only in the closest slot to the MMU in the channel, and so on. But there is no difference between the settings manually set or set by XMP profile, if the BIOS works correctly, so XMP is nothing but a profile there to save time from setting the same settings manually, you might as well have the XMP values on paper instead of in the BIOS, makes no difference.
@@kleingezockt5923 YT often does that whenever a valid question is asked or comprehensive answer given. YT only wants half-baked questions and answers. No, I'm not joking.
board manufacturers can inject bloatware into a windows install and put a web browser into a bios, but they can't fix critical bugs on advertised features
You may be the first to directly say: these are renamed ryzens. While it seems a bit obvious, it has been surprising how many either just ignore that or tip toe their way around the issue. See tech tech potato, geerling, etc... Also, the low end epyc 8004, though a bit of a pain to get a hold of, isn't a massive amount more than a ryzen/epyc 4004 setup, except it has all the pcie lanes and other server must haves you could want. Also lower power depending on the chip.
No kidding on Siena being a pain to get a hold of, the CPU's aren't that hard to find but the motherboards are practically impossible. I'm holding out hope that by the time I rebuild my NAS I'll be able to actually find one to buy.
I have an eye on Siena as well. In my opinion it's a good option as you get high core count and much PCIe connectivity at a relatively low price and TDP. ASRock Rack has an ATX board that's pretty interesting that starts to become available in some stores. I'm currently running an ASRock Rack W680 Server board with an i5-12600K with ECC UDIMM, so low core count but high clock speeds and a Siena platform with high core count but lower clock speeds would be a nice addition. I also don't get the point of the new Xeon E series. You can get a 12th/13th/14th Gen Core processor and a W680 motherboard (that has verified ECC support!), disable the efficiency cores in the BIOS and you basically have a Xeon E (but with an iGPU!).
I love that all of these combinations are possible in 2024. However the reality of the ECC UDIMM market is that any savings you make, you pay for in ridiculous sticker shock and compromised halfway features like only half your DIMM slots populated, etc. I went with a used Milan CPU and a new RomeD8-2T mobo last year, and I'm still pretty tickled with it. I got 256GB of 3200 DDR4 RDIMMs for $381. I can barely get 64GB of DDR5 ECC UDIMMS for that price...
The 2 DPC issues are affecting all current Zen 4 systems, from Epyc Genoa, Ryzen Raphael to the AM5 Phoenix APUs, the memory frequency and timings have to be significantly worsened for it to work automatically on a system with default BIOS settings, it doesn’t have anything to do with ECC memory. Wendell mentioned it in this video since people interested in ECC memory probably also want to know how much total memory a specific platform can use. The situation is unlikely to get better with Zen 5 since the CPUs’ IO Die that connects to memory is the same as Zen 4’s :(
@@abavariannormiepleb9470 That's true, and I was aware of that. However specifically with Epyc I imagine there's not too much butthurt since Genoa has 12 channel memory. I don't know if I've ever even seen a 24 dimm single cpu mobo before...
I got the X570D4i second hand some years ago, while the firmware and bmc are kinda buggy, I've never had better customer support for something I don't even have the receipt for. They've walked me through testing my ECC SODIMMs (even sending them to Taiwan for me) and talking to kingston to add a note that some specific model didn't work on Zen 2. I highly recommend them! Only sad thing is that the APU models (useful for lower power consumption) need to be the PRO version, otherwise the Unregistered DIMMs don't work...
Why in 2024 is all memory not ECC? I mean its not that much extra cost to have un buffered ECC. EDIT: Honestly if you are doing anything more advanced than throwing a GPU and RGB in use ASRock. I dont honestly get the hate, every time i have contacted them for strange none standard requests they have been more than happy to help. Contrast that with MSI, all i asked them was for the location and model number of the fan controller on the PCB on my X99 motherboard and they would not even try to ask the back office for a simple service request. I know its out of support , i know it was me that exploded the fan controller but just please at least try to ask dev for the info, how will this hurt you?
DDR5 doubles the cost to implement ECC. With DDR4 and prior, regular dimms had 8 chips across a single 128 bit channel per dimm, and the ECC parity bits came from an additional memory chip. DDR5 is split into 2x 32 bit channels, so each 4 chip channel will need an additional chip for ECC parity. That would theoretically increase the price by 25% relative to non ECC, 10 vs 8 chips. Also, it won't run anywhere nearly the clocks of consumer ram, as it takes dialing things back a bit to get the stability which using ECC memory implies.
To be able to charge more for it. Also speed it takes time for the CPU to do error checks on the data retrieved from the RAM which will slow down how fast CPUs can be. Right now you really do not have a lot of applications that that small slow down will be noticeable to a non-server non-enterprise user. Gamers might in CPU limited games but as most right now are GPU limited not CPU limited that shouldn't be a reason. The other most likely reason is tradition of having UDIMM and RDIMM DRAM. Right now with how fast the CPU's memory controllers can support it will not really matter though RDIMMs do cap out at a slower speed before throwing errors that will require a re issue of the same data from the RDIMM. You have people who want their RAM to go as fast as possiable and if we are all on ECC RDIMMs well they are made to be correct every time it gets passed to the rest of the CPU not fast while non-ECC UDIMMs are made to be fast while having 98% ~ 99% correct data transfer is acceptable. For consumers even if you have 2% errors which is bad and most likely a sign it is failing it will not be seen. As most software is programmed to handle bad data and the process will be done again. For consumers that will be quick for servers while that might have taken a second and that second can add up or if it isn't caught it can lead to a bad output.
@@yumri4 The ECC is checked in hardware, it doesn't slow the CPU down, as long as there's no error, it is just lower clocks for stability, so the ECC memory is a little slower and has looser timings. Only if there's an error, the CPU will take time to correct it, and only single bit errors are correctable.
It's not that useful for consumers. I have unbuffered ecc on am4. Guess how many corrected errors I had in last 2 years? None. And one uncorrectable. I tested it, it works for sure and corrects when injected btw. If your pc is stable and thoroughly tested with the likes of y-cruncher, cost to benefit is too low for us. But I'm always open to ideas
I was running ECC on my DESKTOP on AM3, I am unning ECC on my DESKTOP on AM4. Because I care about my personal data and ECC+ZFS make sure that whatever I am saving is ending on the drive correctly. Which can not be guaranteed without ECC. So I really really want ECC on AM5 DESKTOP boards.
> Yes, you can get 64 gig DIMMs No you can't, they don't exist anywhere for sale and no manufacturer lists them anywhere. I'm aware they exist in the lab but they're not for sale.
i've seen them a few times on newegg. they're rare, and most users likely get them via custom contract. i've even heard of 128gb dimms, but the bandwidth of those is so poor, it's a waste of resources outside extremely niche use cases that need a ton of memory but not too much bandwidth.
@@stefannilsson2406 can you link? I can't find them on the usual suspects: amazon, newegg, microcenter, bh... Not even google, nor going directly to corsair, zskill, kingston....
One other shortcoming of on-die ECC is that it is only single-error correcting (SEC) and not dual-error detecting (not DED). For performance reasons a Hamming code is used, with 8 check bits protecting 128 information bits. Nine bits are required to detect all the double-bit errors, so approximately half of the double errors go undetected. This is why it's critical to have host support that scrubs the memory and corrects single errors before they accumulate and become undetectable double errors.
I have no regrets about going AM4 for my home server build (35W 5650GE Pro; 64GiB ECC; ITX server board with IPMI; 8 3.84TB SATA SSDs and an Optane boot drive). The small AM5 server board options seem much worse?
More same: Xeon E3-1275L in a Supermicro MBD-X10SL7-F-O with 16G of ECC and a lot of hard drives. It's almost 10 years old now and still works flawlessly. I would like to replace it but I wish the CPU manufacturers would realize not every server needs 64 cores and 256GB of RAM.
I'm on haswell on my nwwest machine so not that far... oldest is a dual socket LGA771. I have to find something to do with that chassi, it is basically collecting dust. Maybe a custom ARM cluster or something.
@@douglasbollinger8678 what's the problem in finding lower end Xeons or AM4 cpus? You are already using a low end xeon (i.e. same socket as consumer CPUs of the same gen, and not compatible with registered RAM)
oh, i've seen a ton of places lately offering 5000-series and 7000-series cpus on dedicated servers in colos. their cost efficiency in eco mode is extremely high in the server market, and their higher single-threaded performance has a growing niche in servers and workstations running compute-bound workflows that don't parallelize as well. datapacket has them almost perpetually in stock.
I'd only gone for AM5 two months ago, trying to surf one of those slightly more budget friendly waves shortly before the next big one. And boy was I knocked off the board by just how bad 2DPC is working with DDR5! The original IBM PC had parity RAM, never quite lost that primal fear of a bit flip since, even if I never observed any on the 64k it launched with. At a billion bits (128GB) I just feel better spending that little extra to ban and eliminate it from that large list of things that can go wrong in computers that require binary precision all the time. Only to find that what was trivial on AM4 with DDR4-3200 ECC was impossible to make work with DDR5 on AM5: I had completely missed months if not more than a year of tragedies from people who had tried and floundered. In the end I went with 96GB of DDR5-5600 1DPC ECC for a 128GB DDR5-5600 2DPC price, which isn't how I imagined progress, but at least worked just fine even at DDR5-6000 (ECC). 7950X3D finally gets me with an RTX 4090 where 5950X didn't quite: 4k gaming that just works without having to spend more time on tuning the game than gaming itself. I swear: no upgrades for years now, because you cheated on delivering real improvements on every angle and that's just not nice. And 240 FPS make no sense when my reaction time has dropped to half my age.
All cool and fancy but if boards lack PCIE split by default having a single x16 pcie5 in almost useless. IMHO at this point all those fast gen pcie is less useful than having multiple lower gen pcie slots unless manufaturers start to update the PHY on the chips but I'm still waiting to see gen 4 NICs. On my main Pc I went with the MSI Carbon B650 also due to the fact that the x16 can be split into 1x8+2x4(ngff) so when the mainboard will be used (in future) as one of my server I have the lanes to put cards in it (via adapter or whatever). I have the 7900X as cPU as the 7950X was unobtanium or priced a lot more than MSRP (scalpers?) Right now I have 2 NICs, 2 GPUs and 1 HBA in my server. And I still have lost some lanes as those slots are all x8 whiler some cards are x4 or x2. Unless you go with an NVME only server and you plug a bifurcation card or you only need compute the lanes as they are are basically useless.
I've done a fairly extensive survey of the AM5 mobo market and I'm dismayed with how bad PCIe lane allocation is. I like to mess around with compute cards and fast networking but not enough to justify buying Threadripper and I don't want the poor performance of an X99 system. All I want is well spaced x8x8x4 with PCIe 5.0 for the top slot without spending huge money on a board. If I relax the PCIe 5 requirement there's ONE board on the market that does that, the Proart B650. Kind of ironically, the Proart X670E has a worse third slot, being only x2. My 10 year old Z87 system does x8x4x4 using just the 16 CPU lanes and without being expensive. Now we have 24 CPU lanes but far worse allocation unless you spend high end GPU money on the mobo. The death of SLI gaming is probably the cause of this.
My question is when will 64GB DDR4 ECC UDIMMs be available. I have 120GB of ram in use so it's either use 2x 64GB DIMMS or 4x 48GB DIMMs clocked waaaay down to 3600MT/s.
@level1techs I've scoured Computex news for data about PCIe layouts on the new AM5 boards and I couldn't find anything written about it. I've pixel peeped the photos to see how many electrical connectors each slot has and only saw the usual x16 CPU x4 chipset rubbish, apart from the X870E Taichi Lite, which at least has x8 x8, but only two slots. Could you please tell me what you saw in the way of better PCIe lane assignments, as mentioned in the video?
I will not be buying another PC without ECC. It's too risky with the amount of RAM these days. Memory corruption is very real and the more RAM you have the higher chance for a bit flip to happen.
Wendell, of all these boards you showed off, which ones are best for IOMMU Vfio? I was considering the x670e ProArt, but if another board is better, I'd consider it... Also, you mentioned that the PCIe configurations were better on the new x870 boards - - - In what way are they better? Could you elaborate a little more on that? Thanks for all your amazing work!
Could I run a mullet computer? Business in the front (3200 ECC for CAD & sims), and party in the back (5800x3D for games)? My b550 vision dp supports ECC and I'm thinking about moving away from my b-die 16gb (3600 CL14) setup to a 32gb ecc without sacrificing too much performance in my single player games
I was literally just thinking of acquiring that ASRock Rack board, I guess I will need to see the results of other AM5 boards and if they might be a better option for my price range.
love my cheap R7 2700 AM4 media stream server. 4x8GB DDR4 3600 c16. 5x 20TB HDD, RAID10 i believe, or maybe RAID5. (striped mirror) not the most expensive PC, but works awesome. and it is made of older gaming setups I recently had updated.
Hello, can you do one video and talk about server memory in medium level understanding? I’m really confused about ecc. E series r series a b c d of r0a …registered unregistered buffered unbuffered my goodness! Sigh… Thanks..
Great video! I was unclear from what you said if using only two sticks of 64 for a total of 128 is safe to run stable. This is for After Effects, Blender & C4D. Thank you for all your amazing work.
The Proart boards have some very nice features. The only one that matters is the 8x/8x pci-e slot split but the 65w charging port on the front USB is just... *chef's kiss* My god this thing was frustrating to deal with for the first year tho...
Don't forget Kingston for ECC UDIMMS. I have 2 KSM48E40BD8KM-32HM sticks in my gaming machine (also with a heavy overclock on them. ECC is wonderful for overclocking). Mobo is the Asrock x670e pro rs. Not 100% convinced on the error reporting to the kernel on this board though.
Woaaah. It's the 3rd mention now of Asrock's D4U boards in as many months by Wendell on Level1 Techs. X470 D4U [2:15] and B650 D4U 2L-2T [7:02] are great. I wanted to get X570 D4U 2L-2T 👍 but never did. Kindest regards, neighbours and friends.
Uh I'm not convinced this is a matter of first and second class citizen. I guess its a little bit similar to cars. First new engineering happens in racing and then trickles down into premium vehicles to eventually arrive in the mainstream when costs are down
@@Mooooov0815 Partially this and partially the fact that hyperscalers buy significantly more volume. I might buy a new system every few years but hyperscalers are buying in the thousands. DDR5 and PCIE5 (and 6-7 to follow) were/are really all about the datacenter/hyperscalers/enterprise and it trickles down to consumer stuff a later on.
It makes sense, datacentres, enterprises and hyperscalers NEED the new technology and NOW, and are willing to pay for that as well as the support. Home servers are just a tiny niche. People within that niche also will just buy enterprise gear on the second-hand market anyway, so most corporations have no reason to accommodate the needs of that niche.
I love both the Taichi and Steel Legend series' products. I've had two Taichi mobos, currently running Steel Legend mobo and GPU. All of them are/were nothing but rock solid. 👍
15:14 It has long been the case that unbuffered ECC memory can work without the ECC function on systems which don't support it. Being able to POST and function normally without ECC is a feature, not a bug.
Wikipedia definition: *Registered memory (also called buffered memory) is computer memory that has a register between the DRAM modules and the system's memory controller. A registered memory module places less electrical load on a memory controller than an unregistered one. Registered memory allows a computer system to remain stable with more memory modules than it would have otherwise.*
Whenever I build computers my number one concern is 24/7 reliability and it is remarkable to me how hard it is to find reviews of MB stability. They all focus on overclocking and other things. I have used Gigabyte for the past dozen or more builds. Is Asrock known for rock solid boards and BIOS????
I dunno if I should risk it. Epyc in AM5 is enticing for what I plan to do with a game host server, but with how sketchy it is I think I might stick with Intel. At least with W680 I know the ECC is gonna work.
well part of the point of this video is that now with epyc launching and having specific "server class" AM5 board, you can count on AM5 ecc (even on non epyc). Unlike last year where it was :dumpsterfire:
I fear AMD is going to have to have a track record before I can buy into their lineup at the low end. As you noted throughout, it has been a dumpster fire for years now. I'm much more interested in Siena from AMD... if only I could find a place to buy the CPU at MSRP.
Which taichi lite can you have 8 + 8 lines for the PCIe , because the b650e taichi lite Version says CPU: • 1 x PCIe 5.0 x16 Slot (PCIE1), supports x16 mode* Chipset: • 1 x PCIe 4.0 x16 Slot (PCIE2), supports x4 mode* You can doing on the INTEL version Z790 Taichi Lite , but that would not be on AM5 CPU: • 2 x PCIe 5.0 x16 Slots (PCIE1 and PCIE2), support x16 or x8/ x8 modes* Chipset: • 1 x PCIe 4.0 x16 Slot (PCIE3), supports x4 mode* it's also on the X670e taich but that is not a $200 , but a $500 motherboard
Is it possible to check that ecc is working? Is there a way to introduce memory error to test this? Should some errors be expected at all? Mine edac-util says "No errors to report."
Where can you get 64gb single stick ddr5 Memory? Have seen no release news, only working on it news. At least in Germany and looking at English press I see through different channels and Google and CZcams videos...
I can confirm Asrock PG Lightening X670E has ECC as an option to enable in the BIOS, and wmic reports the data with and physical with as 64 and 72 so the bus ECC is active. This board is also at 1.1.7.0 AGESA as of May BIOS rev 3.01 so should have the same stuff as the Taichi Light at a lower price point. Also I have tested with the higher density ram modules and have 1 build with 96GB 2 dim, and one with 48GB 2 dimm up and running with ECC (96gb kingston and 48GB NEMIX generic). Both of these we built as workstations for Solidworks with 7700X... since solid works scales poorly with cores this is about the fastest practical build on a sane budget. Also one with W7800 and the other with W7700 GPU.
Hi Wendel, want to build an ATX AM5 EPYC server for a Dentist office. What to give them biggest bang for $. What are your thoughts on how I should proceed? Thank you.
Thank you for this video!... I worried about this myself..I just started acquiring parts for my next build with AMD 7950x and I went with the x670e Asus ProArt since it has a lot of features I want HOWEVER... I seem to be coming across a lot go warnings regarding the Ram and I wanted to go 128gb. Asus is telling me its best to go with a 4 stick kit (kingston fury beast 4x32gb) but they only list XMP options as supported. Is this still wise to go with to keep things stable? I also have 2 - 2x32gb Gskill trident Z5 Neo ram but if youre saying running two sticks is more stable which should I go with the 4x32gb kingston xmp sticks or just run 2 of the 32gb gskill and only run a 64gb ram system..? any insight would be greatly appreciated thank you
So for ECC in AM5 ITX that only leaves me with B650E from Asus and Asrock right? I was really hoping for more than 2 sata ports, but I guess I can do m2 breakout to sata. I’m trying to build a home media server / NAS in ITX, want 4 sata, ecc, and good media decode. If I go Ryzen then I think a low profile intel arc a310 is my best bet for media decoding, which means I won’t have a free pcie slot. Intel seems out due to lack of ecc support in desktop, and where I live I can’t easily source second hand gear and especially not second hand server gear, so I’m going all new. I’ll just run a bunch of VMs to try help justify the cost to myself later on.
If only AMD would fix the horribly high idle power consumption (due to chiplet design inefficiencies introduced in 7000 series) on AM5. It's up to 30W more than an Intel 1700 or AM4 CPU. For a home server that can actually add up.
Are you able to downclock and undervolt the AM5 EPYC chips? Looking at building a low power OpenWRT system with the PCIe lanes to support some Wi-Fi modules and NVMe drives but I need to be able to lower the clocks and the voltage to drastically cut the system power draw. If not then I'll just have to try and get a hold of one of the Ryzen PRO chips which have shown to be more expensive than these EPYC desktop chips.
I didn't quite catch it....should we use registered or unregistered ECC sticks on AM5 boards? And will they actually post error logs? Also, 2 sticks or 4 sticks on AM5 boards?
I am in desperate need of an "ECC for dummies" video/article. About to build first NAS, and I want to focus on maximum data integrity. But most of this video was greek to me.
At this point I think Asrock needs a new flagship board called the Enigma. It exceeds expectations where you'd assume such a company would cut corners while dropping the ball in areas that boggles the mind. I love my AM5 boards and they've been reliable as hell but they make some odd layout decisions and my lord do they need to up their software game. I swear there's like one engineer who doubles as the part time software designer.
I'm updating my windows server to the am5 platform. Can you please compare the epyc 4004 series to the new Ryzen 9000 series. I was particularly interested in the 65w tdp bracket.
Some dust bridged a trace in my DDR3 8350 server, taking out the whole column and I didn't even notice until I checked the logs three months later. It would have crashed without ECC. I gave up on non-ECC systems after getting bus errors while running gcc or memory-intensive simulation codes. Such errors never were an issue on the ECC systems
When I worked in a datacenter we would see ECC error events every so often but we had one customer with a NAS that went from fine to throwing errors all over the place. In the end (I believe it was) Dell ended up replacing the hardware so I never got to troubleshoot what the exact problem was but the customers data on the NAS was fine so in the end ECC did it's job.
People experience this as extra instability they dont attribute to memory errors but people who run ecc see that these errors arent anywhere near as uncommon as one might think, and they might even be worse for some without them knowing. ECC keeps you sane knowing it is or isnt the ram and stops you chasing down random things. It also means no random green glitch in your video archive or lost data in documents.
No, errors are not corrected automatically by the chips. They have to be told to rewrite the corrected data. This command may be issued by the memory controller, in which case it should be transparent to the user and operating system.
@@shanent5793 Now this problem with not being able to monitor the ECC, does it affect the ability of the memory controller to issue correction commands? Or are they separate issues?
not much, 36 bits is enough to encode 32 bits and provide ECC. lower capacity modules use 4 x8 chips for the data, plus an extra identical chip to store ECC bits. The chips all have to match, so the extra chip is also x8. UDIMMs only have four pins to connect the check bits (per sub-channel) so the other four bits are wasted. RDIMMs have eight check bit pins per channel so they can connect all eight bits, making a 40 bit sub-channel. Two channels per DIMM gives us eighty or 72 bits total. High capacity DIMMs use x4 chips, so it's possible to have 36 bit ECC without wasting half a chip. It's up to the host computer to make use of the extra bits, so it's conceivable that a design utilizes the extra four bits but 36/72 is sufficient for SECDED
In my honest opinion: having a board with only two ram slots, if you usually always had 4 using 2 of them - and now are using a board with 2 and use 2 - is the definition "of 5he biggest error you could make". Bro electricity also only works bc if the empty slots. Going from 4 slot: 2 on and 2 off (where you got 50% split into (25% ( 0 ) 25% ( 0 )) to now (50% + 50%) is the definition "of not having the system put the 4 (chinese death) and (50% (if it's about truth values a 50% true and 50% false statement can never be logically understood, like 20 heads coinflips in a row first try) into a moebiusstrip. In my opinion a pc is a logic machine. When you do "ram - no ram - ram - no ram" you do circle logic the circle logic" you split a 50% - you in the hardcore architecture say "power symbol" - interruption of a circle (that is a 50% statement). If you go 2 ram slots and fill both of them up, your like a 100% statement - if that's your life - your life is over. You need to leave out that which is defined as your death. Because it's like 2x 50% is 50% in an equals sign or 2 non logic statements on both sides of a non logical differential diagnosis. AM5 is critical... Not in a good way. Poisonous tree. It's like there is a set of instructions of what a motherboard in my opinion should be and that's 4ram slots and atx. If it doesn't have that - correct that error otherwise you building that thing are responsible for torturing people. everything smaller than atx... I don't like amputation and "small form factors" - in comparison they - are like persons who are/have been ripped off and it is like you now under them one of them. I'd honestly even consider it a problem to have these laying around - your error correction is not working... Thats not what it is about... Well maybe it is for me OK!? Only 2 slots instead of 4 on a board, even if your only using two? That's not what I consider a pc to be and I don't want that bs.
Are rack motherboards a suitable replacement for regular gaming motherboards? I have a mixed use-case that requires both ECC and gaming (it's complicated). What would I be missing out on if I go with a rack motherboard like the ASRock B650?
No RGB, so most should stop reading right here. The boot time is much longer, taking several minutes. It's not just the memory training, you flip the switch and wait for the BMC to boot, then it powers up, then does self tests for a few minutes and possibly probes the network looking for a configuration server. If you want to experiment with RAM timings, there's no auto recovery so you have to pull the battery and drain the CMOS, then re-enter all the CMOS settings. It might be possible to store the CMOS on a server and edit the configuration file which would actually be convenient but I haven't tried it yet. At least there's a diagnostic display, if it says "C5" then the memory training has certainly failed and you can stop waiting. The second page of the BIOS which leads to most of the settings is excruciatingly slow to enter. The CPU socket is rotated a quarter turn CCW, putting the RAM across the top of the board, perpendicular to and on the side adjacent to the IO shield area. This will rotate the CPU cooler, though you can get brackets from Noctua to rotate their big cooler back. It still interferes with the x4 socket, so standard cards won't fit, I had to use the lowest profile cable adapter I could find. Probably not a big deal with smaller coolers. No built in audio, so use the speakers in the monitor or plug headphones into a game controller. USB is only 5Gbit. There's really no substitute for the BMC, which can be accessed through any current web browser. There's also a real serial port, which can be used to access the BIOS if needed. The fans are also controlled by an app inside the BMC, though you have to read the ASROCK help pages to make sense of it. Genuine Intel gigabit. The two 10GBase-T ports are respectable Broadcom interfaces, unlike the Marvell or Aquantia junk slapped onto other boards. Did I mention no RGB?
@@shanent5793 Wow thanks for the detailed response. No RGB is a feature for me, also no shitty Killer networking like on ASRock's consumer boards is a good one. No audio is a good thing as that should be handled by an external DAC. Only headache would be the layout with regards to cooling.
@@TRK--xk7bb back in the '90s I had a board with an on-board vacuum tube preamp, but otherwise I go external. The cooling could actually be superior because it flows front to back but you still have to be careful with the huge tower coolers. I should mention that I use mini-sas cables to split up the x16 slot so the GPU isn't actually isn't in it's place, if it were it would smother a tower cooler if the fans are facing up and down in the default orientation. The VRM also looks absolutely puny, but there's no problem turning up the PBO with a 7950X3D. Not sure how it would handle a 7950X with all the stops pulled
DDR5 should have enforced full end-to-end ECC and 1 DIMM per channel. JEDEC seems to think consumers don't care about the integrity of their data and stability of their machines.
just do a test One to one... with ... an AMD 5600X DDR4 3600 (1800Fclk) VS an AMD 7600X DDR5 (ANY SPEED) With (1800Fclk) 1800 Fclk(RECOMMENDED SINCE IT CAN DO 2000 MHZ/MT's ( or 1200 if needed) and see which had better speed and latency
With my ASROCK rack board and the January bios, it still defaults to 3600, but it will boot reliably with it set to 4200 and no other changes. 4800 needed some bus termination tweaks and occasionally fails to train, but once it's running it doesn't fail the usual stress tests. I have crashed it when I shut down an enormous, bloated, in-house Java application, so I leave it at 4200
SQUEAKY WHEEL GETS THE GREASE: gigabyte aorus elite ax working now with f30 bios, see wendells tweet. Go team!!
x.com/tekwendell/status/1806160769723076873
Hi, I have a question about the 128 gb with unregistered udimms in ddr5 (without or with ecc, would like without for my system, Intel...) Where can you get 64gb single stick ddr5 Memory to get 128gb with only 2 dimms? Had 4 32gb sticks and couldn't go over 4800 mt/s, with 2 sticks I can use 5600 mt/s and also 6000 mt/s without problems, 6000 mt/s only with xmp (so would only use 5600 on 128 gb with 2 sticks, xmp on ddr5 is just a mess mess, but since I can't use my 128 gb, I want at least the speed from 2 sticks for 64 gb) Have seen no release news, only working on it news from crucial, at least in Germany and looking at English press I see through different channels and Google and CZcams videos... Is it available as udimms right now or where you talking about the upcoming things not released yet or about rdimms? Can't find 64 dimms, Google only shows me 2x32gb sticks udimm ddr5 non ecc or udimm ecc, only registered ecc 64 gb ddr5.
ECC CAMM2 RAM will be interesting!
I hope that complete ECC memory support will someday become a standard feature everywhere.
Reminder: Regular DDR5’s On-Die ECC is just a tool to be able to use lower quality DRAM ICs that would otherwise have been discarded, it has nothing to do with the complete ECC chain checking the data between the DIMMs and the CPU’s memory controller.
ECC should of been standard 10 years ago but thank Intel
I have a theory that ECC is kept for Workstation/HEDT class and up, to cause more people to go for higher-end hardware. Full ECC is the single biggest chunk of differentiation.
PCIe lanes are a big point too, but since you have some PCIe lanes already, more becomes moar, while full ECC is a clear yes/no.
@@coolhand4598nah it's just a cost thing. Why would the consumer care to pay for ECC when they are running windows. Windows breaks itself without any ram error implication
> "Regular DDR5’s On-Die ECC is just a tool to be able to use lower quality DRAM ICs that would otherwise have been discarded"
This is just wrong, on-die ECC functionality does not come free and it's much cheaper to just not have it and bin the DRAM chips to lower frequencies on modules.
On-die ECC is required because of RAM densities reached with DDR5, the smaller the cells the easier it is to flip them accidentally
@@SquintyGears Are you aware that servers run Windows too? By that logic "eh it will break itself anyway, who cares". Same in hard drives, why have the ECC logic in those (which they have btw), Windows will break on its own anyway
Slight error at 6:05: DDR5 on-die ECC has *no* ability to correct in-flight errors, only those that happen inside the memory chip.
When I heard that part I was like: oh cool, so Cyclic Redundancy Check (CRC) for both RD/WR data is mandatory now? Judging by your comment it isn't?
Or did you just mean that it's not end to end protected?
Registered ECC should be the standard for all computing today. The fact that JEDEC member companies still view security and data integrity as premium features for only certain classes of consumers is a total ethical failure.
After 15+ years of pc building I still don't fully understand buffered memory or why even after all these years of DDR improvements the benefit of that additional register isn't simply part of the specification. I understand the latency benefits being unbuffered and why this can sometimes be preferable but why is this driver still necessary for such significant increases in capacity per dimm?
On top of this why are RDIMMs no longer pin compatible with UDIMMs? It just seems like we're going backwards. I think there's input voltage differences with DDR5 for some reason but it's all still dropped down by the PMIC.
Perhaps this just ignorance of the situation but feel like at this point we could simply have one specification that supports all primary configurations, with some limitations obviously due to what the board and specific CPU can technically handle.
@@JJFX- there are two ways that RDIMMs reach those high capacities. The first is to use x4 chips (4 data bits) instead of the x8 and x16 chips used in UDIMMs. This will double the number of chips on the command and address (CA) bus. All those connections soak up signal power and add stubs to the bus lines, reducing signal integrity. DDR5 desktop PCs already sacrifice some performance to support two DIMMs per channel, so the memory controllers would have to be made twice as powerful, which is not cost effective for a rare use case.
The other way to increase capacity is to stack the dice in each IC package, they've gone up to eight dice in one package. There's no way an integrated memory controller can directly drive all those chips.
The register delays the commands by one clock cycle, a fraction of a nanosecond. This isn't really noticeable when accesses still take ~60 ns.
Including a register on low-capacity UDIMMs which can have as few as four DRAM ICs is also not cost effective. The register would be a large fraction of the total costs.
The combination of x4 chips and stacked memory means the DIMMs can use a lot of power, hence 12V for RDIMM v. 5V for UDIMM, and the incompatible connectors.
Registers are unnecessary for low capacity systems. They're only needed for servers with a huge amount of chips being driven by a memory controller.
@@shanent5793 “640K ought to be enough for anybody.”
@@shanent5793 “640K ought to be enough for anybody.”
13:10 Has anyone seen 64 GB per DIMM unbuffered memory in the wild, yet?
It’s been half a year since motherboard manufacturers qualified these sticks (Kingston modules with Micron DRAM ICs).
I have asked him under his comment, comment disapiered without an answer. My standalone comment is still there. Have only seen the announcement news. If you find out something, please let me know. Have a Intel system, need unbuffered unregistered udimm with 64 gb sticks for my system with 128gb on intel, so without ecc, but also interested for servers for home. Had 4 sticks 32gb, but couldnt go over 4800, 5600 with xmp but crashes sometimes. Is a gaming and work system. With 2 sticks I can go to 5600 without xmp, no crash. Now run 2 32gb sticks in 6000 with xmp, doesnt crash but I don't like that I have xmp, but if I can't have 128gb I want that speed. so with 2 sticks a 64gb I would go no xmp and 5600 because that's enough speed and enough memory for bigger Projekts (learning coding, 3d printing and cad, and want to learn video editing) so please let me know if you see something, best wishes from Germany.
@@kleingezockt5923 What exactly do you mean by with&without xmp, you know it's just a setting profile, right? It's just a preset, like the normal JEDEC SPD settings but faster&tighter timings usually. Nothing more. Like in the old DDR days you could have a stick with JEDEC SPD 2.5-3-3-7 for example and you'd have a XMP that is intel validated for some CPU to instead run 2-2-2-7 or similar. There is nothing else to XMP other than a profile of settings that the manufacturer claims to run on certain Intel processors with tighter&faster settings than the JEDEC SPD validated settings. So you can check what settings XMP sets usually in BIOS, and try to back them off a little, like a clock or more for row refresh or something if it looks too optimistic, and see if it holds without crashes, then use that profile as a manual entry if it works. XMP settings routinely only work with one DIMM per channel, if you have more that is likely the issue, also typically work only in the closest slot to the MMU in the channel, and so on.
But there is no difference between the settings manually set or set by XMP profile, if the BIOS works correctly, so XMP is nothing but a profile there to save time from setting the same settings manually, you might as well have the XMP values on paper instead of in the BIOS, makes no difference.
@@kleingezockt5923 YT often does that whenever a valid question is asked or comprehensive answer given. YT only wants half-baked questions and answers.
No, I'm not joking.
board manufacturers can inject bloatware into a windows install and put a web browser into a bios, but they can't fix critical bugs on advertised features
You may be the first to directly say: these are renamed ryzens. While it seems a bit obvious, it has been surprising how many either just ignore that or tip toe their way around the issue. See tech tech potato, geerling, etc... Also, the low end epyc 8004, though a bit of a pain to get a hold of, isn't a massive amount more than a ryzen/epyc 4004 setup, except it has all the pcie lanes and other server must haves you could want. Also lower power depending on the chip.
No kidding on Siena being a pain to get a hold of, the CPU's aren't that hard to find but the motherboards are practically impossible. I'm holding out hope that by the time I rebuild my NAS I'll be able to actually find one to buy.
I have an eye on Siena as well. In my opinion it's a good option as you get high core count and much PCIe connectivity at a relatively low price and TDP. ASRock Rack has an ATX board that's pretty interesting that starts to become available in some stores. I'm currently running an ASRock Rack W680 Server board with an i5-12600K with ECC UDIMM, so low core count but high clock speeds and a Siena platform with high core count but lower clock speeds would be a nice addition.
I also don't get the point of the new Xeon E series. You can get a 12th/13th/14th Gen Core processor and a W680 motherboard (that has verified ECC support!), disable the efficiency cores in the BIOS and you basically have a Xeon E (but with an iGPU!).
I love that all of these combinations are possible in 2024. However the reality of the ECC UDIMM market is that any savings you make, you pay for in ridiculous sticker shock and compromised halfway features like only half your DIMM slots populated, etc. I went with a used Milan CPU and a new RomeD8-2T mobo last year, and I'm still pretty tickled with it. I got 256GB of 3200 DDR4 RDIMMs for $381. I can barely get 64GB of DDR5 ECC UDIMMS for that price...
The 2 DPC issues are affecting all current Zen 4 systems, from Epyc Genoa, Ryzen Raphael to the AM5 Phoenix APUs, the memory frequency and timings have to be significantly worsened for it to work automatically on a system with default BIOS settings, it doesn’t have anything to do with ECC memory. Wendell mentioned it in this video since people interested in ECC memory probably also want to know how much total memory a specific platform can use.
The situation is unlikely to get better with Zen 5 since the CPUs’ IO Die that connects to memory is the same as Zen 4’s :(
That pc must be punching B holes.
@@abavariannormiepleb9470 That's true, and I was aware of that. However specifically with Epyc I imagine there's not too much butthurt since Genoa has 12 channel memory. I don't know if I've ever even seen a 24 dimm single cpu mobo before...
I got the X570D4i second hand some years ago, while the firmware and bmc are kinda buggy, I've never had better customer support for something I don't even have the receipt for. They've walked me through testing my ECC SODIMMs (even sending them to Taiwan for me) and talking to kingston to add a note that some specific model didn't work on Zen 2. I highly recommend them! Only sad thing is that the APU models (useful for lower power consumption) need to be the PRO version, otherwise the Unregistered DIMMs don't work...
"Newegg lists..." That was a really subtle application of concentrated heat on a combustible surface.
15:01 imho that's not enabled, I get that with non-pro APUs on AM4 too.
When ECC is actually enabled it should show the DIMMs in dmesg, like this where there are 2x 16gb sticks and 2x 8gb sticks mounted
sudo dmesg | grep EDAC
[ 0.462287] EDAC MC: Ver: 3.0.0
[ 181.059288] EDAC MC0: Giving out device to module amd64_edac controller F17h_M10h: DEV 0000:00:18.3 (INTERRUPT)
[ 181.059295] EDAC amd64: F17h_M10h detected (node 0).
[ 181.059300] EDAC MC: UMC0 chip selects:
[ 181.059302] EDAC amd64: MC: 0: 8192MB 1: 8192MB
[ 181.059304] EDAC amd64: MC: 2: 4096MB 3: 4096MB
[ 181.059310] EDAC MC: UMC1 chip selects:
[ 181.059311] EDAC amd64: MC: 0: 8192MB 1: 8192MB
[ 181.059313] EDAC amd64: MC: 2: 4096MB 3: 4096MB
Why in 2024 is all memory not ECC? I mean its not that much extra cost to have un buffered ECC.
EDIT: Honestly if you are doing anything more advanced than throwing a GPU and RGB in use ASRock. I dont honestly get the hate, every time i have contacted them for strange none standard requests they have been more than happy to help. Contrast that with MSI, all i asked them was for the location and model number of the fan controller on the PCB on my X99 motherboard and they would not even try to ask the back office for a simple service request. I know its out of support , i know it was me that exploded the fan controller but just please at least try to ask dev for the info, how will this hurt you?
DDR5 doubles the cost to implement ECC. With DDR4 and prior, regular dimms had 8 chips across a single 128 bit channel per dimm, and the ECC parity bits came from an additional memory chip. DDR5 is split into 2x 32 bit channels, so each 4 chip channel will need an additional chip for ECC parity. That would theoretically increase the price by 25% relative to non ECC, 10 vs 8 chips.
Also, it won't run anywhere nearly the clocks of consumer ram, as it takes dialing things back a bit to get the stability which using ECC memory implies.
To be able to charge more for it. Also speed it takes time for the CPU to do error checks on the data retrieved from the RAM which will slow down how fast CPUs can be. Right now you really do not have a lot of applications that that small slow down will be noticeable to a non-server non-enterprise user. Gamers might in CPU limited games but as most right now are GPU limited not CPU limited that shouldn't be a reason.
The other most likely reason is tradition of having UDIMM and RDIMM DRAM. Right now with how fast the CPU's memory controllers can support it will not really matter though RDIMMs do cap out at a slower speed before throwing errors that will require a re issue of the same data from the RDIMM. You have people who want their RAM to go as fast as possiable and if we are all on ECC RDIMMs well they are made to be correct every time it gets passed to the rest of the CPU not fast while non-ECC UDIMMs are made to be fast while having 98% ~ 99% correct data transfer is acceptable. For consumers even if you have 2% errors which is bad and most likely a sign it is failing it will not be seen. As most software is programmed to handle bad data and the process will be done again. For consumers that will be quick for servers while that might have taken a second and that second can add up or if it isn't caught it can lead to a bad output.
@@yumri4 The ECC is checked in hardware, it doesn't slow the CPU down, as long as there's no error, it is just lower clocks for stability, so the ECC memory is a little slower and has looser timings. Only if there's an error, the CPU will take time to correct it, and only single bit errors are correctable.
@@dgo4490the error checking function simultaneously indicates the which bit has an error, so correction doesn't take any more time
It's not that useful for consumers. I have unbuffered ecc on am4. Guess how many corrected errors I had in last 2 years? None. And one uncorrectable. I tested it, it works for sure and corrects when injected btw. If your pc is stable and thoroughly tested with the likes of y-cruncher, cost to benefit is too low for us. But I'm always open to ideas
I was running ECC on my DESKTOP on AM3, I am unning ECC on my DESKTOP on AM4.
Because I care about my personal data and ECC+ZFS make sure that whatever I am saving is ending on the drive correctly. Which can not be guaranteed without ECC.
So I really really want ECC on AM5 DESKTOP boards.
Thank you! I've been trying to figure this stuff out, and it had all been such scattered info I had given up.
love the classic OD Green motherboards! Glad to hear about AM5 and ECC. I'm still rocking DDR3 ECC on my old 4th gen Intel fileserver
> Yes, you can get 64 gig DIMMs
No you can't, they don't exist anywhere for sale and no manufacturer lists them anywhere. I'm aware they exist in the lab but they're not for sale.
i've seen them a few times on newegg. they're rare, and most users likely get them via custom contract.
i've even heard of 128gb dimms, but the bandwidth of those is so poor, it's a waste of resources outside extremely niche use cases that need a ton of memory but not too much bandwidth.
I can find them. They are quite expensive though. 300+ $/dimm.
@@stefannilsson2406 can you link? I can't find them on the usual suspects: amazon, newegg, microcenter, bh... Not even google, nor going directly to corsair, zskill, kingston....
I have seen 48 gigs tho
Hey... Are you the LLM finetuner?
If so, good going.
One other shortcoming of on-die ECC is that it is only single-error correcting (SEC) and not dual-error detecting (not DED).
For performance reasons a Hamming code is used, with 8 check bits protecting 128 information bits. Nine bits are required to detect all the double-bit errors, so approximately half of the double errors go undetected. This is why it's critical to have host support that scrubs the memory and corrects single errors before they accumulate and become undetectable double errors.
So in short ECC on a Ryzen platform is still hit or miss. Disappointing
Just like punching B holes. Sometimes it doesn't work out in your favor.
There are so many other Ryzen platforms its bulletproof on, its just AM5... for now, on some mobos in particular.
I have no regrets about going AM4 for my home server build (35W 5650GE Pro; 64GiB ECC; ITX server board with IPMI; 8 3.84TB SATA SSDs and an Optane boot drive). The small AM5 server board options seem much worse?
>AM4 works for home servers
Me still using a Sandybridge i5 for my home server in 2024
Same. 2500k running Proxmox with everything virtualized on one machine.
More same: Xeon E3-1275L in a Supermicro MBD-X10SL7-F-O with 16G of ECC and a lot of hard drives. It's almost 10 years old now and still works flawlessly. I would like to replace it but I wish the CPU manufacturers would realize not every server needs 64 cores and 256GB of RAM.
I'm on haswell on my nwwest machine so not that far... oldest is a dual socket LGA771.
I have to find something to do with that chassi, it is basically collecting dust.
Maybe a custom ARM cluster or something.
@@douglasbollinger8678 what's the problem in finding lower end Xeons or AM4 cpus? You are already using a low end xeon (i.e. same socket as consumer CPUs of the same gen, and not compatible with registered RAM)
Brothers in arms
oh, i've seen a ton of places lately offering 5000-series and 7000-series cpus on dedicated servers in colos. their cost efficiency in eco mode is extremely high in the server market, and their higher single-threaded performance has a growing niche in servers and workstations running compute-bound workflows that don't parallelize as well.
datapacket has them almost perpetually in stock.
Memory on AM5 is such a pain in the ass.
Zen 1 had memory issues too, from all the leaks Zen 5 seems to remedy many of the memory issues from Zen 4
I'd only gone for AM5 two months ago, trying to surf one of those slightly more budget friendly waves shortly before the next big one.
And boy was I knocked off the board by just how bad 2DPC is working with DDR5!
The original IBM PC had parity RAM, never quite lost that primal fear of a bit flip since, even if I never observed any on the 64k it launched with.
At a billion bits (128GB) I just feel better spending that little extra to ban and eliminate it from that large list of things that can go wrong in computers that require binary precision all the time.
Only to find that what was trivial on AM4 with DDR4-3200 ECC was impossible to make work with DDR5 on AM5: I had completely missed months if not more than a year of tragedies from people who had tried and floundered.
In the end I went with 96GB of DDR5-5600 1DPC ECC for a 128GB DDR5-5600 2DPC price, which isn't how I imagined progress, but at least worked just fine even at DDR5-6000 (ECC).
7950X3D finally gets me with an RTX 4090 where 5950X didn't quite: 4k gaming that just works without having to spend more time on tuning the game than gaming itself.
I swear: no upgrades for years now, because you cheated on delivering real improvements on every angle and that's just not nice. And 240 FPS make no sense when my reaction time has dropped to half my age.
All cool and fancy but if boards lack PCIE split by default having a single x16 pcie5 in almost useless.
IMHO at this point all those fast gen pcie is less useful than having multiple lower gen pcie slots unless manufaturers
start to update the PHY on the chips but I'm still waiting to see gen 4 NICs.
On my main Pc I went with the MSI Carbon B650 also due to the fact that the x16 can be split into 1x8+2x4(ngff) so when the mainboard will be used (in future)
as one of my server I have the lanes to put cards in it (via adapter or whatever). I have the 7900X as cPU as the 7950X was unobtanium or priced a lot more than MSRP (scalpers?)
Right now I have 2 NICs, 2 GPUs and 1 HBA in my server. And I still have lost some lanes as those slots are all x8 whiler some cards are x4 or x2.
Unless you go with an NVME only server and you plug a bifurcation card or you only need compute the lanes as they are are basically useless.
I've done a fairly extensive survey of the AM5 mobo market and I'm dismayed with how bad PCIe lane allocation is. I like to mess around with compute cards and fast networking but not enough to justify buying Threadripper and I don't want the poor performance of an X99 system. All I want is well spaced x8x8x4 with PCIe 5.0 for the top slot without spending huge money on a board. If I relax the PCIe 5 requirement there's ONE board on the market that does that, the Proart B650. Kind of ironically, the Proart X670E has a worse third slot, being only x2.
My 10 year old Z87 system does x8x4x4 using just the 16 CPU lanes and without being expensive. Now we have 24 CPU lanes but far worse allocation unless you spend high end GPU money on the mobo. The death of SLI gaming is probably the cause of this.
I got lucky with an asus a620 pro a 7700 and a single ecc ddr5 dim from the qvl. Ecc enabled on first try after a big bios update
09:32 Attention editor! The motherboards you've inserted here are both Intel boards, not AMD AM5, which is what this video is discussing.
My question is when will 64GB DDR4 ECC UDIMMs be available. I have 120GB of ram in use so it's either use 2x 64GB DIMMS or 4x 48GB DIMMs clocked waaaay down to 3600MT/s.
Would love a Deskmini with ECC and IPMI
@level1techs I've scoured Computex news for data about PCIe layouts on the new AM5 boards and I couldn't find anything written about it. I've pixel peeped the photos to see how many electrical connectors each slot has and only saw the usual x16 CPU x4 chipset rubbish, apart from the X870E Taichi Lite, which at least has x8 x8, but only two slots.
Could you please tell me what you saw in the way of better PCIe lane assignments, as mentioned in the video?
I will not be buying another PC without ECC. It's too risky with the amount of RAM these days. Memory corruption is very real and the more RAM you have the higher chance for a bit flip to happen.
I didn't understand a word of this and I loved it.
Wendell, of all these boards you showed off, which ones are best for IOMMU Vfio? I was considering the x670e ProArt, but if another board is better, I'd consider it...
Also, you mentioned that the PCIe configurations were better on the new x870 boards - - - In what way are they better? Could you elaborate a little more on that? Thanks for all your amazing work!
thats a good choice for iommu tbh. newer boards seem to have 1 more 4 or 8 lane slot.
Could I run a mullet computer? Business in the front (3200 ECC for CAD & sims), and party in the back (5800x3D for games)?
My b550 vision dp supports ECC and I'm thinking about moving away from my b-die 16gb (3600 CL14) setup to a 32gb ecc without sacrificing too much performance in my single player games
I suppose, the prices were lower 1 month ago? The Taichi 650 lite costs 297 EUR (official price: 373 EUR) here in Germany ...
I was literally just thinking of acquiring that ASRock Rack board, I guess I will need to see the results of other AM5 boards and if they might be a better option for my price range.
love my cheap R7 2700 AM4 media stream server. 4x8GB DDR4 3600 c16. 5x 20TB HDD, RAID10 i believe, or maybe RAID5. (striped mirror)
not the most expensive PC, but works awesome. and it is made of older gaming setups I recently had updated.
I wonder if the new CUDIMMs will have ECC and help with running two sticks per channel
Hello, can you do one video and talk about server memory in medium level understanding? I’m really confused about ecc. E series r series a b c d of r0a …registered unregistered buffered unbuffered my goodness!
Sigh…
Thanks..
I always wondered why we could not just use any type of memory we want if wanted to use ECC kits we should just be able to.
YES, looking forward to the info on the new chipset boards incoming! THANK YOU!
Great video! I was unclear from what you said if using only two sticks of 64 for a total of 128 is safe to run stable. This is for After Effects, Blender & C4D. Thank you for all your amazing work.
The Proart boards have some very nice features.
The only one that matters is the 8x/8x pci-e slot split but the 65w charging port on the front USB is just... *chef's kiss*
My god this thing was frustrating to deal with for the first year tho...
Yo 65W charging port? I wonder if you could split that into charging hub
Don't forget Kingston for ECC UDIMMS. I have 2 KSM48E40BD8KM-32HM sticks in my gaming machine (also with a heavy overclock on them. ECC is wonderful for overclocking). Mobo is the Asrock x670e pro rs. Not 100% convinced on the error reporting to the kernel on this board though.
Woaaah. It's the 3rd mention now of Asrock's D4U boards in as many months by Wendell on Level1 Techs. X470 D4U [2:15] and B650 D4U 2L-2T [7:02] are great. I wanted to get X570 D4U 2L-2T 👍 but never did.
Kindest regards, neighbours and friends.
Us consumers will forever be second class citizens to hyperscalers and the like…
Uh I'm not convinced this is a matter of first and second class citizen. I guess its a little bit similar to cars. First new engineering happens in racing and then trickles down into premium vehicles to eventually arrive in the mainstream when costs are down
That’s capitalism for you. More money, more votes.
@@Mooooov0815 Partially this and partially the fact that hyperscalers buy significantly more volume. I might buy a new system every few years but hyperscalers are buying in the thousands. DDR5 and PCIE5 (and 6-7 to follow) were/are really all about the datacenter/hyperscalers/enterprise and it trickles down to consumer stuff a later on.
Obviously, hyperscalers spend so much more money on hardware than homelab-nerds, it's totally understandable to treat them with better support.
It makes sense, datacentres, enterprises and hyperscalers NEED the new technology and NOW, and are willing to pay for that as well as the support.
Home servers are just a tiny niche. People within that niche also will just buy enterprise gear on the second-hand market anyway, so most corporations have no reason to accommodate the needs of that niche.
I love both the Taichi and Steel Legend series' products. I've had two Taichi mobos, currently running Steel Legend mobo and GPU.
All of them are/were nothing but rock solid. 👍
Never considered switching from a ROG Board to a ProArt... Guess I am getting older xD Such nice features!
Thank you so much for this video since I am actually at the moment choosing parts for home server based on AM5
15:14 It has long been the case that unbuffered ECC memory can work without the ECC function on systems which don't support it. Being able to POST and function normally without ECC is a feature, not a bug.
Whats the difference between registered and unregistered? I have unregistered ECC in my NAS...
registered = allows higher density ram by having extra registers on ram and thus having less stress on memory controller of cpu.
Wikipedia definition:
*Registered memory (also called buffered memory) is computer memory that has a register between the DRAM modules and the system's memory controller. A registered memory module places less electrical load on a memory controller than an unregistered one. Registered memory allows a computer system to remain stable with more memory modules than it would have otherwise.*
Whenever I build computers my number one concern is 24/7 reliability and it is remarkable to me how hard it is to find reviews of MB stability. They all focus on overclocking and other things. I have used Gigabyte for the past dozen or more builds. Is Asrock known for rock solid boards and BIOS????
that asrock rack am5 board can be had for $310 sans 10G and IPMI/BMS
onboard 10g doesn't sound as good as an addon card where you're assured that it's getting its own lanes.
Im eyeing with H13SAE-MF supermicro motherboard seems like a cool platform to me. Good lane distribution etc
I also went with that over the ASRock Rack option due to better layout and the ability to fit a 22110 SSD in the M.2 slot.
I never dealt with server hardware so half of this video had me just smiling and nodding my head not sure about what is beind said :-)
I want to see some X870 with DDR5-CAMM2 ECC in action.
Doesn't Registered ECC have higher supply voltage, and different socket?...
I dunno if I should risk it. Epyc in AM5 is enticing for what I plan to do with a game host server, but with how sketchy it is I think I might stick with Intel. At least with W680 I know the ECC is gonna work.
well part of the point of this video is that now with epyc launching and having specific "server class" AM5 board, you can count on AM5 ecc (even on non epyc). Unlike last year where it was :dumpsterfire:
I fear AMD is going to have to have a track record before I can buy into their lineup at the low end. As you noted throughout, it has been a dumpster fire for years now. I'm much more interested in Siena from AMD... if only I could find a place to buy the CPU at MSRP.
I'm so glad that I had bought my 7950x at launch, because I doubt I would have had another chance long after that
Which taichi lite can you have 8 + 8 lines for the PCIe , because the b650e taichi lite Version says
CPU:
• 1 x PCIe 5.0 x16 Slot (PCIE1), supports x16 mode*
Chipset:
• 1 x PCIe 4.0 x16 Slot (PCIE2), supports x4 mode*
You can doing on the INTEL version Z790 Taichi Lite , but that would not be on AM5
CPU:
• 2 x PCIe 5.0 x16 Slots (PCIE1 and PCIE2), support x16 or x8/
x8 modes*
Chipset:
• 1 x PCIe 4.0 x16 Slot (PCIE3), supports x4 mode*
it's also on the X670e taich but that is not a $200 , but a $500 motherboard
Is it possible to check that ecc is working? Is there a way to introduce memory error to test this? Should some errors be expected at all? Mine edac-util says "No errors to report."
As long as they don’t give me more than 20 lanes, I don’t care about ECC or IPMI or whatever.
Where can you get 64gb single stick ddr5 Memory? Have seen no release news, only working on it news. At least in Germany and looking at English press I see through different channels and Google and CZcams videos...
Test CZcams is buggy
Kingston seems to have a 5600Mhz 64GB single stick ECC RAM, though it costs like €300
I can confirm Asrock PG Lightening X670E has ECC as an option to enable in the BIOS, and wmic reports the data with and physical with as 64 and 72 so the bus ECC is active. This board is also at 1.1.7.0 AGESA as of May BIOS rev 3.01 so should have the same stuff as the Taichi Light at a lower price point. Also I have tested with the higher density ram modules and have 1 build with 96GB 2 dim, and one with 48GB 2 dimm up and running with ECC (96gb kingston and 48GB NEMIX generic). Both of these we built as workstations for Solidworks with 7700X... since solid works scales poorly with cores this is about the fastest practical build on a sane budget. Also one with W7800 and the other with W7700 GPU.
Where did you order the ECC RAM? I got the same mobo with 7900 non X
Hi Wendel, want to build an ATX AM5 EPYC server for a Dentist office. What to give them biggest bang for $. What are your thoughts on how I should proceed?
Thank you.
Thank you for this video!... I worried about this myself..I just started acquiring parts for my next build with AMD 7950x and I went with the x670e Asus ProArt since it has a lot of features I want HOWEVER... I seem to be coming across a lot go warnings regarding the Ram and I wanted to go 128gb. Asus is telling me its best to go with a 4 stick kit (kingston fury beast 4x32gb) but they only list XMP options as supported. Is this still wise to go with to keep things stable? I also have 2 - 2x32gb Gskill trident Z5 Neo ram but if youre saying running two sticks is more stable which should I go with the 4x32gb kingston xmp sticks or just run 2 of the 32gb gskill and only run a 64gb ram system..? any insight would be greatly appreciated thank you
do those epyc 4000 chips boot on consumer am5?
So for ECC in AM5 ITX that only leaves me with B650E from Asus and Asrock right?
I was really hoping for more than 2 sata ports, but I guess I can do m2 breakout to sata.
I’m trying to build a home media server / NAS in ITX, want 4 sata, ecc, and good media decode.
If I go Ryzen then I think a low profile intel arc a310 is my best bet for media decoding, which means I won’t have a free pcie slot.
Intel seems out due to lack of ecc support in desktop, and where I live I can’t easily source second hand gear and especially not second hand server gear, so I’m going all new. I’ll just run a bunch of VMs to try help justify the cost to myself later on.
Great video!
If only AMD would fix the horribly high idle power consumption (due to chiplet design inefficiencies introduced in 7000 series) on AM5. It's up to 30W more than an Intel 1700 or AM4 CPU. For a home server that can actually add up.
I have here a 13600T 175usd (used) on the wall with 4 ssd 4 mem and a 4060ti can idle about 35w - 45w light loads
Are you able to downclock and undervolt the AM5 EPYC chips?
Looking at building a low power OpenWRT system with the PCIe lanes to support some Wi-Fi modules and NVMe drives but I need to be able to lower the clocks and the voltage to drastically cut the system power draw.
If not then I'll just have to try and get a hold of one of the Ryzen PRO chips which have shown to be more expensive than these EPYC desktop chips.
I didn't quite catch it....should we use registered or unregistered ECC sticks on AM5 boards? And will they actually post error logs?
Also, 2 sticks or 4 sticks on AM5 boards?
He was very clear about never using 4 sticks, and both will work, and posting error logs depends on motherboard bios support.
Oof, I am stuck with an Aorus B board. Hopefully with the 9000 support, it will be more functional.
what model in particular? F30 on the Aorus X670 fixes the ECC problem.
@@Level1TechsMy fault it is a B650 Gaming X AX.
Which of those client boards can be managed through a serial console? Do they even have serial ports?
What about ASRock X670E PG Lightning? It's the cheapest X670E I could find in Thailand.
Is a memtest86 enough to tell you if ECC work? I've tried this and it's showing as enabled and my system shows 72bits too with dmidecode
That timing though. 😂 pardon the pun but seriously I just used ECC on my AM5 build. Thanks Asus.
I am in desperate need of an "ECC for dummies" video/article. About to build first NAS, and I want to focus on maximum data integrity. But most of this video was greek to me.
I would love is something like ms01 would show up that has amd cpu and support for ECC ... that would be just perfect workstation / SOHO server
My NAS has a 5950x. 🤷🏻♂️
every application needs to be compiled to shaders instead of x86 or arm, yep you can run applications even on cpu gpu computer shaders.
At this point I think Asrock needs a new flagship board called the Enigma. It exceeds expectations where you'd assume such a company would cut corners while dropping the ball in areas that boggles the mind. I love my AM5 boards and they've been reliable as hell but they make some odd layout decisions and my lord do they need to up their software game. I swear there's like one engineer who doubles as the part time software designer.
I use new supermicro mobo for AM5, works for me.
Is it the H13SAE-MF?
@@justinth83 yes
Good to know it's working, I have 3 on order for a proxmox HA cluster! 2x32GB ECC in each.
Why Ubuntu? were you running kernel 6.9.6?
Does it matter that you are using new hardware on Ubuntu with Kernel 6.5?
I'm updating my windows server to the am5 platform. Can you please compare the epyc 4004 series to the new Ryzen 9000 series. I was particularly interested in the 65w tdp bracket.
One of these days 128gb or even 256gb over 4 sticks of ddr5 at 6000mtps will run out of the box on am5. One of these days
Wonderful :)
Has anyone encountered real world events where ECC has kicked in? What's your story?
Some dust bridged a trace in my DDR3 8350 server, taking out the whole column and I didn't even notice until I checked the logs three months later. It would have crashed without ECC.
I gave up on non-ECC systems after getting bus errors while running gcc or memory-intensive simulation codes. Such errors never were an issue on the ECC systems
When I worked in a datacenter we would see ECC error events every so often but we had one customer with a NAS that went from fine to throwing errors all over the place. In the end (I believe it was) Dell ended up replacing the hardware so I never got to troubleshoot what the exact problem was but the customers data on the NAS was fine so in the end ECC did it's job.
In general you'll see a few ECC correcteds in any server at least a few times a year, not atypical to have like 6-8 events a year, really. correcteds.
People experience this as extra instability they dont attribute to memory errors but people who run ecc see that these errors arent anywhere near as uncommon as one might think, and they might even be worse for some without them knowing.
ECC keeps you sane knowing it is or isnt the ram and stops you chasing down random things. It also means no random green glitch in your video archive or lost data in documents.
Most randoms blue screens, and many of the problems that get solved with a reboot, are related to memory errors
Can i get ECC memory for a normal gaming PC? like a 5800x? do i need to check if the mobo supports it i supposed but is ECC more expensive ?
unregistered, and the mobo has to support ECC mode, but Zen 3 CPUs do support it
Even without system-based monitoring support, does the internal ECC of DDR5 correct errors automatically? At least stop cosmic ray incursions.
No, errors are not corrected automatically by the chips. They have to be told to rewrite the corrected data. This command may be issued by the memory controller, in which case it should be transparent to the user and operating system.
@@shanent5793 Now this problem with not being able to monitor the ECC, does it affect the ability of the memory controller to issue correction commands? Or are they separate issues?
Would be interested in what the technical difference between 72bit vs 80bit ECC is.
8 bits extra for corrections
not much, 36 bits is enough to encode 32 bits and provide ECC. lower capacity modules use 4 x8 chips for the data, plus an extra identical chip to store ECC bits. The chips all have to match, so the extra chip is also x8. UDIMMs only have four pins to connect the check bits (per sub-channel) so the other four bits are wasted.
RDIMMs have eight check bit pins per channel so they can connect all eight bits, making a 40 bit sub-channel. Two channels per DIMM gives us eighty or 72 bits total.
High capacity DIMMs use x4 chips, so it's possible to have 36 bit ECC without wasting half a chip.
It's up to the host computer to make use of the extra bits, so it's conceivable that a design utilizes the extra four bits but 36/72 is sufficient for SECDED
ProArt motherboard? I thought ProArt was for decent monitors, not mother boards....
How often does a neuron in my brain flip :) I do prefer no errors.
In my honest opinion: having a board with only two ram slots, if you usually always had 4 using 2 of them - and now are using a board with 2 and use 2 - is the definition "of 5he biggest error you could make".
Bro electricity also only works bc if the empty slots.
Going from 4 slot: 2 on and 2 off (where you got 50% split into (25% ( 0 ) 25% ( 0 )) to now (50% + 50%) is the definition "of not having the system put the 4 (chinese death) and (50% (if it's about truth values a 50% true and 50% false statement can never be logically understood, like 20 heads coinflips in a row first try) into a moebiusstrip.
In my opinion a pc is a logic machine. When you do "ram - no ram - ram - no ram" you do circle logic the circle logic" you split a 50% - you in the hardcore architecture say "power symbol" - interruption of a circle (that is a 50% statement).
If you go 2 ram slots and fill both of them up, your like a 100% statement - if that's your life - your life is over.
You need to leave out that which is defined as your death.
Because it's like 2x 50% is 50% in an equals sign or 2 non logic statements on both sides of a non logical differential diagnosis.
AM5 is critical... Not in a good way. Poisonous tree.
It's like there is a set of instructions of what a motherboard in my opinion should be and that's 4ram slots and atx.
If it doesn't have that - correct that error otherwise you building that thing are responsible for torturing people.
everything smaller than atx... I don't like amputation and "small form factors" - in comparison they - are like persons who are/have been ripped off and it is like you now under them one of them.
I'd honestly even consider it a problem to have these laying around - your error correction is not working...
Thats not what it is about... Well maybe it is for me OK!?
Only 2 slots instead of 4 on a board, even if your only using two? That's not what I consider a pc to be and I don't want that bs.
Are rack motherboards a suitable replacement for regular gaming motherboards?
I have a mixed use-case that requires both ECC and gaming (it's complicated). What would I be missing out on if I go with a rack motherboard like the ASRock B650?
No RGB, so most should stop reading right here.
The boot time is much longer, taking several minutes. It's not just the memory training, you flip the switch and wait for the BMC to boot, then it powers up, then does self tests for a few minutes and possibly probes the network looking for a configuration server.
If you want to experiment with RAM timings, there's no auto recovery so you have to pull the battery and drain the CMOS, then re-enter all the CMOS settings. It might be possible to store the CMOS on a server and edit the configuration file which would actually be convenient but I haven't tried it yet. At least there's a diagnostic display, if it says "C5" then the memory training has certainly failed and you can stop waiting.
The second page of the BIOS which leads to most of the settings is excruciatingly slow to enter.
The CPU socket is rotated a quarter turn CCW, putting the RAM across the top of the board, perpendicular to and on the side adjacent to the IO shield area. This will rotate the CPU cooler, though you can get brackets from Noctua to rotate their big cooler back. It still interferes with the x4 socket, so standard cards won't fit, I had to use the lowest profile cable adapter I could find. Probably not a big deal with smaller coolers.
No built in audio, so use the speakers in the monitor or plug headphones into a game controller.
USB is only 5Gbit.
There's really no substitute for the BMC, which can be accessed through any current web browser. There's also a real serial port, which can be used to access the BIOS if needed. The fans are also controlled by an app inside the BMC, though you have to read the ASROCK help pages to make sense of it.
Genuine Intel gigabit. The two 10GBase-T ports are respectable Broadcom interfaces, unlike the Marvell or Aquantia junk slapped onto other boards.
Did I mention no RGB?
@@shanent5793 Wow thanks for the detailed response. No RGB is a feature for me, also no shitty Killer networking like on ASRock's consumer boards is a good one. No audio is a good thing as that should be handled by an external DAC. Only headache would be the layout with regards to cooling.
@@TRK--xk7bb back in the '90s I had a board with an on-board vacuum tube preamp, but otherwise I go external.
The cooling could actually be superior because it flows front to back but you still have to be careful with the huge tower coolers. I should mention that I use mini-sas cables to split up the x16 slot so the GPU isn't actually isn't in it's place, if it were it would smother a tower cooler if the fans are facing up and down in the default orientation.
The VRM also looks absolutely puny, but there's no problem turning up the PBO with a 7950X3D. Not sure how it would handle a 7950X with all the stops pulled
DDR5 should have enforced full end-to-end ECC and 1 DIMM per channel. JEDEC seems to think consumers don't care about the integrity of their data and stability of their machines.
just do a test One to one... with ...
an AMD 5600X DDR4 3600 (1800Fclk)
VS
an AMD 7600X DDR5 (ANY SPEED) With (1800Fclk)
1800 Fclk(RECOMMENDED SINCE IT CAN DO 2000 MHZ/MT's ( or 1200 if needed) and see which had better speed and latency
what about the Supermicro H13SAE-MF
is 3600mhz limit still there with 4 sticks?
With my ASROCK rack board and the January bios, it still defaults to 3600, but it will boot reliably with it set to 4200 and no other changes. 4800 needed some bus termination tweaks and occasionally fails to train, but once it's running it doesn't fail the usual stress tests. I have crashed it when I shut down an enormous, bloated, in-house Java application, so I leave it at 4200
Its probably fine for gaming but its not stable enough for ECC use
thats epic