Asrock Ryzen 9 XCP-NG Virtualization Server Build With 25GB
Vložit
- čas přidán 22. 05. 2024
- lawrence.video/xcp-ng
Kit Link with parts
kit.co/lawrencesystems/ryzen-...
Parts List:
Mellanox MCX512A-ACAT ConnectX®-5 0/25GbE Dual-Port SFP28
Qlogic FastLinQ QL41262H Dual 25GbE
Rosewill RSV-R4100U 4U Server Chassis
AMD RYZEN 9 5900X
Enermax ETS-N31 ll Compact Intel/AMD CPU Cooler
AsRock Rack X570D4U micro-ATX Server
EVGA 210-GQ-0750-V1 750 GQ, 80+ GOLD 750W, Semi Modular
G.Skill Ripjaws V Series 128GB Total
WD_BLACK 2TB SN850 NVMe Gen4
Graylog Video
• Open Source Logging: G...
45 Storinator Q30 Followup Review 2022
• 45 Storinator Q30 Foll...
Xenserver / XCP-NG 8.0 HA / High Availability Cluster Setup
• Xenserver / XCP-NG 8.0...
Connecting With Us
---------------------------------------------------
+ Hire Us For A Project: lawrencesystems.com/hire-us/
+ Tom Twitter 🐦 / tomlawrencetech
+ Our Web Site www.lawrencesystems.com/
+ Our Forums forums.lawrencesystems.com/
+ Instagram / lawrencesystems
+ Facebook / lawrencesystems
+ GitHub github.com/lawrencesystems/
+ Discord / discord
Lawrence Systems Shirts and Swag
---------------------------------------------------
►👕 lawrence.video/swag
AFFILIATES & REFERRAL LINKS
---------------------------------------------------
Amazon Affiliate Store
🛒 www.amazon.com/shop/lawrences...
UniFi Affiliate Link
🛒 store.ui.com?a_aid=LTS
All Of Our Affiliates that help us out and can get you discounts!
🛒 lawrencesystems.com/partners-...
Gear we use on Kit
🛒 kit.co/lawrencesystems
Use OfferCode LTSERVICES to get 10% off your order at
🛒 lawrence.video/techsupplydirect
Digital Ocean Offer Code
🛒 m.do.co/c/85de8d181725
HostiFi UniFi Cloud Hosting Service
🛒 hostifi.net/?via=lawrencesystems
Protect you privacy with a VPN from Private Internet Access
🛒 www.privateinternetaccess.com...
Patreon
💰 / lawrencesystems
⏱️ Timestamps ⏱️
00:00 XCP-NG Ryzen Build Case, Rack and Netwwork
02:08 25G SFP28 Network Adapters
04:27 AsRock Rack X570D4U IPMI
07:29 XCP-NG Monitoring Raid With Graylog
09:00 Storage and Migration
#XCPNG #Virtualization #Ryzen9 - Věda a technologie
Kit Link with parts
kit.co/lawrencesystems/ryzen-9-xcp-ng-virtualization-server-build-with-25gb
Parts List:
Mellanox MCX512A-ACAT ConnectX®-5 0/25GbE Dual-Port SFP28
Qlogic FastLinQ QL41262H Dual 25GbE
Rosewill RSV-R4100U 4U Server Chassis
AMD RYZEN 9 5900X
Enermax ETS-N31 ll Compact Intel/AMD CPU Cooler
AsRock Rack X570D4U micro-ATX Server
EVGA 210-GQ-0750-V1 750 GQ, 80+ GOLD 750W, Semi Modular
G.Skill Ripjaws V Series 128GB Total
WD_BLACK 2TB SN850 NVMe Gen4
Graylog Video
czcams.com/video/rtfj6W5X0YA/video.html
45 Storinator Q30 Followup Review 2022
czcams.com/video/MtWM19Ikmdo/video.html
Xenserver / XCP-NG 8.0 HA / High Availability Cluster Setup
czcams.com/video/jvhUY81pBw0/video.html
Tom, I know you like marktext... Check out obsidian though.. Marktext +++++ (incl. folder managerment etc)
@@JPEaglesandKatz Not open source, does not save as files that I can easily sync.
@@LAWRENCESYSTEMS It saves the files in .md format just like marktext.. and in standard folder format organisation. but yeh. doesn't appear to be opensource. Anyone, I just saw an marktext screenshot in your video so figured I'd give you a hint in case you didn't know.
Hi Tom, I find your channel invaluable to help me get an understanding how things work and set up stuff. Thanks!
Glad to help!
This makes total sense for a virtualization Setup. You've build a 48 thread cluster for less than the cost of one Threadripper. If you're doing it for VMs this is just smarter than cramming 32-64 cores into one device.
And it will be faster than a Threadripper too, that's the beauty of these Zen 3.
I've been running Ryzen systems on Hyper-V and Proxmox for quite a while now and I've never had any issues. I currently have 5950X, 2400G, and 3700U servers running now. Great stability, no crashes, I've been quite happy. I'll likely add another 5950X system when AM4 prices bottom out.
I’m glad your making the build for the more common geek. This is the build I want was looking at asrock as some people use them but still feel odd before this still would rather us asus,msi or super micro, tyan but glade I have spent many years not making a move but as rock will be my next server build as my asus Maximus extreme started killing ddr2 ram sticks. Sad it’s been far more responsive then Any other computer I have used till I ran out of ram to boot from.
thank you this video was helpful cant wait for you new build
WARNING for the Asrock X570D4U board. There is a bug in the BIOS/BMC that if you change the IPMI from DHCP to manual assigment, it corrupts the BMC firmware and the IPMI will no longer work. You will have to reload the BMC firmware though Freedos if that happens. So assign your static IP through your DHCP and not from the board itself.
wow thats a heck of a bug, nice looking out
Thank you for the warning!
I changed my IP with the previous stable BMC version and it is still set to manual after update.
Changed it in IPMI webgui > settings > network > network IP settings
0:11 seconds in and I'm already hooked 😂.
Very cool. I have a R5 3600 sitting in my X570D4U-2L2T right now. In the next 6 months my desktop will get an upgrade and the 5900x will go into it. AsRock's IPMI isn't great but the last update fixed a lot of things.
I was using 2 x 3900X for my lab a few years ago and they performed very well with Proxmox and Vmware 6.7. Sadly, at that time, I didn't get the same performance out of them from XCP-ng. I might revisit this eventually.
Edit: @10:30 For those saying it need a faster cpu, consider that the AMD Ryzen 5900X is faster than any Intel server CPU out there bar none. It "only" has 12-core/24-thread, but you will be able to run more VMs on the same hardware specs (Ram/Disk/Network) and acheived greater performance. It is just that fast and good.
I really like the kit specs! I am more comfortable with proxmox, but this should run relatively well for that hypervisor platform too. I wonder if you could water cool the CPU so you can get it in a 2U case. Greate build and impressed with asrock's IPMI platform!
Great video, can’t wait until we upgrade our core and distribution to 100 and 25g
Great build Lawrence, i'm planning to build a home server soon...Intel Xeon E5 2630 V3 LGA2011-v3, ASRock Rack Board with Dual Channel ECC DDR4,...will be running both Windows Server 2019 and RHEL 9.0....And yes Ryzen's single thread performance will be higher than these higher core Xeon CPUs...i really liked that you pointed out the CPU utilization was very less indicating that going for a Threadripper or EPYC CPU would be an overkill for this build...
" ... crashing really inhibits work getting done ... " - So true!
Someone at CZcams has a sense of humour! When he clicked on the Remote Control viewer, up popped an ad! Took a moment for my mind to register the switch in content!
The 18/20Gbps thing is likely due to the packet loss shown by iperf. As an experiment you could see what it does with flow control enabled, I have seen this exact thing with machine A sending untagged packets to switch B, which then forwards them as tagged (and thus slightly larger) to C, then B runs out of output buffer, and TCP in A then backs off a lot.
Oh wow how others stupidity made me a rich man.
The video recording function looks good. Does that allow you to set the system to record video files of KVM sessions for audit purposes?
Great setup. I have a similar one, with two nodes for the compute and a Synology providing the shared storage. All my network is gigabit, but hoping to make the jump to 2.5 Gbe next year.
Hey Mike, Im currently looking at Synology too for some shared storage - what model did you use and what has perf been like?
If you are not in a hurry and on a budget I bet you'll be able to roll out 5 or 10Gbps late next year for not too much $. I did this a couple years ago and have seen the prices falling quickly since then. If your concern is hitting higher rates over old cable, I had good luck hitting 10Gbps over CAT5e on Netgear and Ubiquiti 10Gbps equipment.
@@locusm It is a 720+ with the expansion unit. Performance is good with the SSDs as a read/write cache.
Very nice, I have a similar build, but I used the Noctua NH-D9L cooler so that I can change the fan orientation.
Noctua sent me the extra pieces for free.
I also went with ECC ram KSM32ED8/32ME
It's running in my old Antec 900 case, I really want to get a nice 4U case with 120mm fans and a lot of HDD slots.
Future upgrade perhaps.
That said, I went with Unraid as the host OS as I needed an all-in-one server for my home lab.
At work we run XCP-NG on threadripper systems with TrueNAS on older intel systems.
"Untested backups are just wishful thinking" - Well said, sir. Well said.
You always have great informative videos. Some a little too complex for me, a non IT Guy. I now know I need a a Chassis (not rack mount) Server and the server should have E1.S drives....maybe start with 6- 7 TB drives....dont now where to buy or build.
That live vm migration is amazing. I only use truenas scale as my hypervisor but this is something over I which would consider trying xcpng. Also I would love to see a video about high availability you mentioned!
czcams.com/video/jvhUY81pBw0/video.html
Would like to see a 2U build out of something like this and what kind of at wall power usage you are seeing.
i have lusted after that motherboard
For so long
i'm probably never gonna get it
but man i have i windowshopped that thing hard
For my home-lab, I run a Ryzen because it works just fine. In my office, I find that RAM is my bottleneck. With a couple dozen VM's running, I need more RAM than Ryzen can handle. I'm close to topped out at 128 GB of ram, which is why I went with a low-end Threadripper for my backup virtualization server and a high clock speed 7371 Epyc for the primary server. You are absolutely correct though, I never come close to topping out on the CPU's. Even when I ran my virtualization on 8350 chips, RAM was always my bottleneck.
Gavin Belson's signature 😅
I have the x470 version of that board and I had nothing but issues with it when I first bought it. Constant crashing no matter what hypervisor I tried and even tried several different kits of ram. I finally shelved this entire server for a year or 2 then gave it a another try this year. With a bios update and swapped cpu its finally been stable and I deployed it as my main hypervisor. Now I've been looking at investing in 5900x or a 5950x for it since its only running on a 2700x right now. My only issue now is this is my only AMD node in my cluster so I can't live migrate to and from it.
I want to build a file server with one of the new multi-NVME boards, bypass the limitations of physical drives, transfer data via a 10 gig or 25 gig to/from NVME's.
Any spec changes you would make for a production server? ECC, enterprise SSDs, etc.?
That blue LED in front is so bright it almost broke your camera.
I have looked through a lot of the comments, but not all of them, so sorry if I have missed it. Can you let us know what the power usage is for one of the servers on idle and under normal load with XCP-ng. Think this is an important element of designing systems following the cost of energy in recent years.
I have the x470 version of the same motherboard can you explain better how you setup the fan settings… also I don’t get any information about the processor installed nor the ram or the other components….how do you fixed that?
At 0:12 on the server rack on the right side, what is that phallic thing on the front/top of it? 🤣 Looks like a hilarious signature.
Gavin Belson Signature box :)
Hey Tom, have you tried setting a larger TCP Window size? Up to a point it can have quite a dramatic effect on performance.
Nope, don't really want to change that, too many other devices on that network.
@lawrence systems I'm curious how this stacks up to intel's own competing NIC (XXV710DA2) performance-wise, the Mellanox is much cheaper than Intel by almost 200 bucks.
I use an old dl380p gen 8 as a test environment for work. Something like this would be awesome. My system seems to be a little slow nowadays which is fine but eventually i'd love to upgrade to something a little faster with a little better efficiency. That being said, how is the efficiency for one of those ryzen systems? What kind of power draw are you looking at for that on average?
Typically Amd is the efficiency king. Most of their consumer products are about 2/3s to 1/2 of the power draw of a comparable Intel chip. At least at the consumer end which the 5900x that he is using is apart of.
I see that the ConnectX-4 24Gbe cards are much cheaper used than what you went with. For a home lab, is it worth 2x going for the newer 5th gen cards or stick with 4th?
The X-4 boards are just fine.
I’d be interested to know which VMs you’re running? Are the all Linux distros?
Mostly Linux, one windows server
I am running hyper-v hypervisor running 8-10 win10 vms that have external ip or USB connected instruments. They are running on a 200gb ram xeon silver. How would moving a vm handle the connections? I can't have connectivity loss. I am looking to upgrade to a faster system to be forward compatible. I have an msp but am learning as much as I can. Eventually looking to do an HA cluster
What UPS do you use?
What's the power consumption of the 5900X servers?
Hey Tom, how are these systems faring today with the 5900x? Would you say they still keep up and there's no reason to upgrade or are they showing its age for virtualization needs? I'm planning on purchasing a 5950x for my server build, also with 128GB RAM since it is considerably cheaper than going for 7000 series along with the platform change, and I'm pretty certain it would suffice for my needs but I'd like to know your thoughts on such a platform in 2023, thanks!
Still working great, no issues at all.
it is that a Pen... in the top of the rack door ?
What OS are you using? Been a while since I used a Linux OS, kind of want to dive back in.
POP_OS!
what is the black network used here for ?
I need to talk to my boss to get a test lab up and running
Hi Lawrence, any documentation with instructions step by step for deploy this lab from zero??
czcams.com/video/q-jKs62b6Co/video.html
the SSD is it SN770 or SN850 ?
mate what is the brand of the chasis u r using??
I need one for a E-ATX motherboard
He did pin a comment with list of parts in the comments section.
@@aimannorazman7959 thanks Aiman!
how would the AMD cpu in this build compare to say Intel Xeon Silver 4214R Processor ? would you recommend ryzen 9 cpus in a production server ?
AMD is very power efficient than Intel and the Ryzen9 seems to hold up really well for production loads.
@@LAWRENCESYSTEMS can you also suggest a 2u server chassis with hot swappable sata backplane along with redundant power supply.
somewhat off topic what's your opinion on the pbxact 25?
We have one, we like it, it works well.
As a young IT professional, I have a couple questions:
1. You made a comment the Rosewill cases were not on rails, which I’m ok with. But why was that decision made? And what are you using instead of sliding rails?
2. Platforms like XCP-ng are more supported on Server Hardware (Xeons, Epyc, etc). Have you had any issues running XCP-ng on consumer-level hardware?
3. I noticed on my Optiplex I’m using to evaluate XCP-ng that hyperthreading (SMT) was not enabled. I saw that for your install, it was. How do you recommend it to be?
Hyperthreading is generally something you’d want to leave on unless your specific circumstances benefit from turning it off
1) Why spend money on things that are not needed, like rails. We use these kit.co/lawrencesystems/homelab-rack-build/navepoint-adjustable
2) XCP-NG works on a wide range of hardware
3) I leave it on
If you wanna trigger people properly name the servers "Ruby" so it's technically "Ruby off Rails"
You didn't think ECC ram was worth it for these machines?
Nope
The unifi SFP switch only has 4 x 25gig ports. BUT i see 5 x 25gig dac cables..
No, only the Ryzen servers have 25G cables on one port. The other (orange cable) is 10G
@@JasonsLabVideos he put up a chart that clearly explains this. Did you watch the video?
Why not go with the X570D4U-2L2T that includes 2x 10GbE?
Because you can't find them anywhere close to MSRP.
Correct and we wanted 25gb
Are these servers compatible with the latest version of ESXi?
Not sure, I don't have time to test them with ESXI
AsRock Rack X570D4U with the new bios, version 1.50, I'm not able to clock memory to 3200MHz. they've added a new OC menu and if I change anythin, according IPMI VCCM goes up to 1,7V and I need to clear CMOS...
I have 4x Kingston Server Premier DIMM 32GB, DDR4-3200, CL22-22-22, ECC (KSM32ED8/32ME) which are also listed on Memory QVL
Bios 1.20 was fine
I didn't had 1.29b
Ours would not clock to 3200 either.
@@LAWRENCESYSTEMS Thanks! Also thanks for the additional information on the live stream. I went back to 1.20 and kept latest BMC, 3200MHz is working again. Also I'm in contact with Asrock Rack support.
Anything using the SATA interface would never saturate a 10+ gig network link.
Try iperf2 instead of iperf3. The latter has performance issues.
What’s the software used to demonstrate?
czcams.com/video/mpF1i9sfEJ0/video.html
Does xen not have SNMP to monitor the raid?
Not enabled by default and not something I have tested.
What did you do with the video out on 5900x?
Nothing
@@LAWRENCESYSTEMS So you can boot it without a GPU? um.. interesting
@@jefferytse certain mptherboards allow you to boot without video, the asrock rack x570d4u he's using is one of them
@@willynilly9954 Cool. That's what I need to know. I want to buy that to run Vmware but couldn't get my head around it.
Please take a swag at the cost of each system.
The 128GB RAM limit on this really sucks.
System Inventory never works for me with the X570D4U :/
You are not alone. That isn't working for no-one. Even worse, access to the BIOS settings from the web ui doesn't work either. But what can you do? It is basically the only AM4 server motherboard out there.
I did configure it once and never touched it again.
@@alexanderahman4884 thank you for your reply. I have built and sold many servers based on this MB and the previous x470d4u since ryzen 3. For now the cpus have a higher rma rate than the MBs
@@manuelcoliano1101 May I ask what type of problems you are having with your cpus (the RMA ones)? Is it by any chance ECC related? 🤔
@@alexanderahman4884 I havent tried using non ecc memory for a long time in this MBO. The CPUs that went into rma showed errors on the linux commandline. When googling these errors they seemed pretty common also with gaming MBOs. Though I had a case where proxmox showed very many corrected errors on the memory. A reboot "fixed" this one, since 2 months it didnt return - I'm using kingston server premier BTW
can this be used for plex / jellyfin?
Don't see why not
@@LAWRENCESYSTEMS thank you for reply the thing is that i have seen another video of someone mentioning that video decoding is disabled on this apps when the motherboard has IPMI and Bios does not have option to disable or enable this to be used or not.
Many apps don't require video card to decode but to make them faster at decoding/encoding.
But no ECC RAM...
Ahh, yes. That is the other thing I forgot to mention that might be upsetting to people. ¯\_(ツ)_/¯
bam! starts off with a d and balls on the server door. lol. nice touch.
I can do 2x performance for 4x less $$$ ha ha ha.
What’s your build
@@keithallen9054 Dont remember but it was clever AF why for what purpose you need ?
@@ChuckNorris-lf6vo something similar just looking for a solid build but saw 2x for 4x less I was intrigued