I built another server cluster... - Promxox HA Cluster w/ Ceph
Vložit
- čas přidán 31. 05. 2024
- MB/CPU - SuperMicro X10SDV-4C-TLN2F / Xeon 1521
Case - amzn.to/3CVaTvL
Boot SSD - amzn.to/3pohRqe
Ceph Storage - amzn.to/441AP55
-------------------------------------------------------------------------------------------
🛒 Amazon Shop - www.amazon.com/shop/raidowl
👕 Merch - www.raidowlstore.com
🔥 Check out today's best deals from Newegg: howl.me/cjNc3sze2O3
-------------------------------------------------------------------------------------------
Join the Discord: / discord
Become a Channel Member!
/ @raidowl
Support the channel on:
Patreon - / raidowl
Discord - bit.ly/3J53xYs
Paypal - bit.ly/3Fcrs5V
My Hardware:
Intel 13900k - amzn.to/3Z6CGSY
Samsung 980 2TB - amzn.to/3myEa85
Logitech G513 - amzn.to/3sPS6yv
Logitech G703 - shop-links.co/cgVV8GQizYq
WD Ultrastar 12TB - amzn.to/3EvOPXc
My Studio Equipment:
Sony FX3 - shop-links.co/cgVV8HHF3mX / amzn.to/3qq4Jxl
Sony 24mm 1.4 GM -
Tascam DR-40x Audio Recorder - shop-links.co/cgVV8G3Xt0e
Rode NTG4+ Mic - amzn.to/3JuElLs
Atmos NinjaV - amzn.to/3Hi0ue1
Godox SL150 Light - amzn.to/3Es0Qg3
links.hostowl.net/
0:00 Intro
1:16 Proxmox Cluster Hardware
6:46 Setting up the cluster
10:17 Overall thoughts on my new cluster - Zábava
I am glad you did not break your motherboard please read MBD-X10SDV-4C-TLN2F manual (can be downloaded from supermicro home page) that 4pin is not for atx 4 pin power its for dedicated DC supply. You need to either use 24pin atx or 4 pin. DO NOT use them both! Quote from manual: "Do not use the 4-pin DC power at PJ1 when the 24-pin ATX Power at JPW1 is connected to the power supply. Do not plug in both PJ1 and JPW1 at the same time."
Oh neat! 🙃
RTFM - The truest words to abide by.
@@longnamedude3947 its too long
@@longnamedude3947 * insert Michael Scott NOOOOO gif here *
@@RaidOwlAnother is PEBKAC
I'm gonna be honest as well. I DO NOT NEED a lot of my homelab stuff. However, i do love cosplay. Specifically cosplaying as a sysadmin/automation engineer.
The PCIe passthrough definitely works on those boards. If you’ve set the vfio stuff in /etc/modules/ and “intel_iommu=on” in PVE and still not booting, make user the BIOS has the VT-d extension and IOMMU enabled. Thanks for the tour of the new cluster!
Thanks for putting together this video and the previous one showing ProxMox VE HCI with less expensive hardware.
The two 10GbE switches shown are each a single point of failure. To upgrade the networking to HA, these could be replaced with two switches configured with MLAG. VLANs can be used to create the two logical networks shown: Host and Ceph.
For maintenance like internal drive or part replacement, having four nodes instead of the minimum three would allow one node to be safely removed at any time to perform orderly maintenance and upgrades.
When one of only three nodes is intentionally made unavailable to perform maintenance, the two remaining nodes are in a degraded state for some services (including Ceph), and if anything unlucky happens to one of the two remaining nodes during the maintenance window, there is no longer a cluster.
Damn dude, that’s a badass cluster setup. I just bought 3 2U PowerEdge R730’s, which is clearly less efficient, and you’re is really high performance. Me likey.
I really hope to see you do MORE videos about Proxmox!
Great tour! I love that you include the mistakes and it's not a "do this and the HA gods will bless you" tutorial. I also had a devil of a time getting iGPU passthrough to work on Proxmox, although I'm running it on a Dell 3930 (with USB-C-only iGPU display output). I had to use cpu=host, q35, Virtio-GPU, PCI passthrough with PCI-e & x-vga, USB port passthrough (for the dummy dongle to work). I would still get error 43, but a quick disable/enable cycle in Windows gets things back in order.
I'm still in the "just got my first HP Elitedesk mini PC as my new docker host instead of my NAS" phase of my homelab journey but I still enjoy watching there videos even though I don't ever expect to rackmount my servers (unless perhaps it's a very small one that can mount miniPC's).
8:59 THANK YOU!!!! I was looking all around to find out which order this went in!
This is essentially the same route I took, except I used 5 Dell 7050s 1L SFF PCs, NVMe for ProxMox, 1 TB SSDs on each node for Ceph and a dedicated backhaul network for Ceph. All in, it cost about $750 and my only regret was not waiting until Prime Day to get 2TB SSDs for about what I paid for the 1TB drives. 🤦♂️
I haven't made it the primary cluster yet though and am still trying to figure out what I really want to do with it. 😅
Great video though!
Would love a non-neckbeard approach/mindset to Ceph/CephFS/Rook setup on that cluster as a follow-up. Fighting thru that on my own setup.
Yeah I got some stuff to try
@@RaidOwl Awesome. More responses for the CZcams algorithm overloards.
Good thing you can directly use your ceph cluster as a csi backend! And if you create a cephs and mons you can even use RWX pvcs
I'm seriously contemplating ordering a few of the Topton Intel 8505 router boxes and running them in a cluster like this... More powerful processor with lower power draw than the SuperMicro you're using, 6x 2.5Gbps NICs for direct host-to-host connectivity without a switch (and enough extra ports for network connectivity), and they're completely passive on cooling. Only real downside is there's no PCIe expansion to speak of, and it's not Xeon / ECC, but for the price (around the same price as you spent for each of these, if not a touch cheaper) they would make for an awesome cluster!
I'm currently running one for my router and it's been rock solid, and I've got a couple of Chenbro 1U servers that are due for replacement.
Thanks for the video. Missing one thing though - simulated failure for one of the nodes
I ran a mirrored set of 870 evos in my proxmox cluster and the performance was okay, as long as i didn't update more than one VM at a time or download and install large packages/binaries. I/O delay would cause random VMs to become unresponsive and general instability
Promxox and ZFS really needs enterprise drives with larger caches and high endurance.
"cyberbullied by some neckbeard" - priceless. That kept me smiling right to the end and then some. Thanks owl.
Great video love this style and subject
Just a wonderful inspiring video thx 😊
I just love your content...hmmm feels like home ❤
I'm just here taking up space!
Regarding PCI Passthrough:
Some vendors literally block it from properly working, HP ProLiant Servers for example. Ive been cracking my head for literal weeks with them.
After trying it with some Lenovo Servers it worked instantly for me.
Just a heads up, that sometimes its literally impossible to get it to work
"I'm just here so I won't get fined."
- Someone else
Okay Marshawn
The cool thing about those inwin cases is that you can swap the position of the PSU and front I/O ports, swap the rack ears to the other side and then you have all your motherboard I/O and PCI slots at the front of the rack while leaving power at the back.
I'm just here taking up space - For PV(C) in your k3s Cluster, I recommend using "Rook with an external Ceph cluster", i.e. the Ceph storage provided by ProxMox.
Someone else mentioned Rook. Ima look into it for sure
you deserved this and you probably want to follow this thread down the line - you will find you do need this once you get all the kinks out - do a ha opnsense next - non virt total bare metal... you will want to max the ram out on these - more ram equals more better - great you have an upgrade path - you will want to go all nvme - that seems to be your weakest link - pls update with and make the cluster fabric 20g bonded and add a usb 2.5 for mgt inf - please explore other netfs options - nfs, zfs, ocfs2, sshfs, gluster
I built a proxmox cluster using Supermicro M11SDV-8C-LN4F AMD Epyc 3251 board. 8-core, 16-thread, 65W total usage with 4 sticks of ECC ram and a SATA SSD boot disk under load. Another Mini ITX board, and though it's Zen 1, its power usage is the reason I chose it. I need fast networking too, but storage is handled differently for high availability in my network. So the 1 PCI-e slot is used for a 10 Gbit nic, because though Epyc 3xx1 supports 10Gigabit networking on chip, this board doesn't have 10gig ports.
I would recommend you to replace those silicon power SSDs. I had a few of these in the datacenter running only as proxmox boot disk and all died after a few month.
I love these cases. They are very hard to get hands on.
I would love to have a proxmox cluster for my home lab. I could not get GPU passthrough to work with my setup either. Does proxmox do load balancing were it would move a vm to the other host if it is less busy then the other host? Thanks for sharing.
I am running 2 Dell r620's with 10c/20t and 64GB ram each. I need 1 more to make a matching trio. The 2 Dells pull about 130w. I am using Harvester right now but really thinking about switching to Proxmox because that where the cool kids hang (thats where the projects and tutorial videos are), So hard to find Harvester content.
I'm sorry to say...
I love you video's. Very well edited.
I’m sorry but….thank you
I'm looking at replacing an existing vSphere Enterprise with a shared storage enterprise "grade" virtualization platform. Maybe I missed it, but I seem to be having a problem finding anyone who can demonstrate High Availability (HA) of the hypervisor nodes in these three scenarios below. Everyone has videos on setting up the cluster, live migration but I'm not seeing anyone doing actual tests of a complete or partial failure of one of the cluster nodes.
1) Complete node fail -- just pull the power plug(s) out to simulate. How does Proxmox handle dozens of VMs powering on? Does it have a DRS type function where it will distribute the VMs across the remaining nodes? Is there an ability to have specified VMs prioritized over other VMs? Also, the ability to restart VMs in a specific order?
2) Partial fail where the hypervisor is in some sort of hung state and the VMs are down but the storage is still accessible and any file locks (if applicable) are still held?
3) Host isolation. What happens when the Proxmox host is unreachable from the management side but the VMs running are still accessible? Will it allow VMs to still run? Will it provide an option to restart VMs on other nodes?
Thanks.
I've just finished creating a pair of Proxmox servers for myself hosting my original 6x rPi's rebuilt as Debian VMs.
Each machine has the following specs :
- Inter-tech K-125L 1U rackmount case
- Akyga 200W PSU
- ASRock J5040-ITX M/board
- 32GB DDR4 2400 RAM (2x16GB)
- 1x 500GB Samsung 870 EVO SSD for the boot drive (overkill as I originally ordered 256GB but weren't in stock but the retailer supplied for the same price)
- 1x 4TB Samdung 870 EVO SSD for VM storage (again overkill, but i have plenty of available space and they're sheap)
- 3x Noctua 40mm NF-A4x20 FLX 5000 fans
Both machines run super cool and quiet and have plenty of power for my current needs, with each only using
if you had NVME drive and need for bandwidth, I'd probably pick 25Gbe or so NICs for the available PCIe slot,
another option would be to populate that PCIe slot with multi-nvme PCIe board, I think Sonnet makes some new one with 8 nvme bays on PCIe 4.0 x16 which is wild :D
7400 Micron drives are NVMe. Faster in Ceph than any consumer drive can dream of. And the quad core can't handle more than a single NVMe at full load anyway.
Hei, good setup, and very interesting video, thanks ^^
NICE!, used this same chassis in a firewall build. Good case but the back io shields were a PITA> !
Lol yeah I just avoided those
@@RaidOwl I saw :) Nice setup sir !
I don't know about proxmox, but with ESXi you can add variables to the VM to fix the code 43 error with GPUs. Although from my experience that error only came up with older NVIDIA cards. Basically you gotta tell the VM it isn't a VM.
thanks for another great video, awesome. have a great day
I’m just here taking up space!!
I watched the Livestream already, so I'm definitely just here taking up space
I'm Just Here Taking Up Space...but love the content.
lol I love how the official Ceph documentation questions the need for multiple networks, but the users went nuts all over the internet and demand that you use it
Love it
Please make a follow-up video on this setup when it completes 1 year with your new learnings along the way. 🙂
I’m here just taking up space 🙌
-some dude
I am here for the chat!! Allways easy to learn from
I wonder if the 00 is the code the bios is reporting to the IPMI, Then I'd hazard a guess its a cracked ball joint under the cpu possibly. A reflow might fix it. I have no real experience in that, but its just a random guess based on other things I've seen.
I concur, those Silicon Power (SPCC) SATA SSDs do suck. I discovered the company hardcoded the SMART data! They are all fixed to display 40 C. No matter what the actual temperature is. This seems to be a response to a review on Amazon that their SSDs were running as high as 60 C and failing prematurely. And that reviewer noted the replacements all read 40 C. I bought a 1 and 2 TB SATA SSD and both of mine never waver from 40 C. Even when cold booting at a much lower ambient temperature. Or testing them under CrystalDiskMark. F-cking Amazon pulled my review down with my findings! Buyer beware.
I would like to see some benchmarks. Ideally with database usage :)
Some filesystem Benchmarks would be nice
How does the secondary network for the CEPH storage work? Is it not at all connected to the main network? If so, do I have to manually assign IP addresses to the systems?
It’s connected to the main network but it has its own VLAN with proper DHCP addressing
@@RaidOwl Its not recommended to use DHCP for Ceph network (Ceph is tightly bounded to IP address). Yeah, it doesnt really matter in a 3 node ceph cluster, but its a really bad practice
@@Mcs1v reservations make everything possible and just fine.
I wanted to mimic this setup but for the switch that has the dedicated CEPH network...if that switch needs to reboot say for an update....would that wreck a lot of stuff since all 3 hosts lose communication to each other over CEPH? Have you tested that?
you can put the ceph cluster in maint mode or just let it pause on its own. source, lost two switches powering a cluster.
Im here just taking up space 😂
anything to be gained by glusterfs the 1TB X 3 SSD ?
I’m just here taking up space 😂
Just here taking up space
what is the read and write performance of your Ceph Cluster?
I think you're supposed to have TWO separate Ceph networks, one for their "private" and one for their "public" -- plus 10 gig for your proxmox vm network, making three, then a separate gig network for corosync and yet another gig network for proxmox system (separate from vm network). I'm just looking into this now though and I have read that putting the two ceph networks on one NIC is usually fine for most people. But I'm just figuring this out myself too.
"I am here just taking up space"
I'm just here taking up space, but I'm trying to understand the high availability part -- what sort of failures are we trying to protect against? the shared ceph pool seems to be the single point of failure that would take down the entire cluster? A single drive failure isn't an issue with raid, but what if that hardware failure that isn't a drive?
An entire server could blow up and everything would keep running
@@RaidOwl I think I've mixed something up then. Is there a 4th server that holds all the drives doing the ceph storage ( the larger 4U under the 3 nodes ), or is the ceph storage replicated across, and exists on, the drives in the node servers?
@@JonathanDavisJJ Ceph runs on each of the 3 nodes using each of the Micron SSDs in each of the nodes. So yeah, the ceph storage is replicated across all 3 nodes.
Love the realness, linux server nerd bro
Kinda weird request, and I know I most definitely should not be buying hardware based on aesthetics, but could you let me know what these chassis look like with rack mounted ubiquiti gear? Do the two silvers look good together, or do they clash?
What is your energy use under load at at rest? This might be just what I need.
Under load they pull just over 100W. At idle it’s like 85
My x10 gets quite warm. Have you done anything to the cooling?
Nah they have active cooling though. Get about mid 70s under load
Did you try and pass through the gpu without the rizer cable?
I did not 🤔
Does anyone have a link to the U.3 to NVME adapter cable? I'm having trouble finding how the U.3 drive connects to an M.2/NVME slot on a motherboard.
These are what I used: amzn.to/3uTXR1b
Wow! thanks for the quickly reply. Can't wait to try it it out. I have the 2 of the XeonD-1540 boards and it might be time to get a 3rd ! @@RaidOwl
Wish I woulda gone with the 1540s haha
re: gpu. It's either going to be you need this in your file: vfio_iommu_type1.allow_unsafe_interrupts=1 or you need hugepagesz=1G default_hugepagesz=2M in your grub and hugepages: 2 and balloon: 0 in your /etc/pve/qemu-server/VMID.conf or your hardware just can't handle outputting to the physical ports on the gpu, in which case if you turn off default gpu on its setting in hardware it will still work but only over vnc/for computation, which is not that useful in your case probably (it's way more useful if you are importing gpus to to hardware encode/decode for plex/jellyfin.)
I am here just taking up space :P
Lookup "Proxmox Kernel 5.15.60-1-pve Breaks PCI Passthrough" I spent days trying to get PCI passthrough working until I found out about kernel issue
Probably the better way to setup ceph would have been to get 2 switches in an MLAG and do an LACP with the 2 ports to get 20git for ceph and VMs.
Since you are in a non production environment where your servers get hit with 10g incomming traffic from the internet ceph has more ressources and failover capability
I'm just here taking up space.
I’m just here taking up space
im just here taking up space c:
I'm just space taking up here
Im just here taking up space
I'm just here taking up space :)
did you use eltro past
No no no I just got rid of one of my 2u servers to save on power and noise. I don't need a 3 1us for a cluster to run k8s like I've wanted to.....how loud are these....asking for a friend.
Actually verrrrry quiet. I forgot to put my sound test in here but I used some low adapters from Noctua and they work great.
I also don’t need these & want to know how loud they are… for … a… “friend”… haha
Would you still recommend the Zima board?
For sure, just manage your expectations
I am just here taking up space.
1amp for all three? Whats the voltage in the US? 220?
120
10:09 holy banana. no standoffs? :S
I'm just here taking up bandwidth
I have a similar setup, for those who want a 3 node cluster but don’t want to splash out on a 10g switch, you can use duel 10gig nics in a full mesh network so each node has a direct connection to each other, works well and removes a single point of failure! (The switch)
or go for the quad 25G nic in the pci-slot way and use them in mesh network with dac/fiber and leave the 10G copper for outbound networking.
Tried the spell too.
Now how do I revert, there's no snapshot for that.
Not sure if I missed it .. but power consumption (idle/load) per node would have been nice too - otherwise cool video!
About 30-35W per node
@@RaidOwl that would be quite much for me .. is that idle or load? in the case you showed 1 node can run the VMs and stuff while the other 2 nodes are basically fully idle (until there is a crash of node 1). are the 30-35W idle (for node 2 and 3) or load (node 1)? thanks for the reply
I must admit , when I first came across your channel, I found you and/or your method of presentation to be somewhat annoying. But the overlords at CZcams and their algo kept on pushing your content to my feed and after watching more of your videos, I have actually started taken a liking to your awkward sense of humor. I also enjoy that you share all of your mistakes and blunders with us, which any homelabber can relate to. So I guess I’ll hit that subscribe button!
Praise to the almighty CZcams overlords 🙏🏼
Has it enough CPU power to encode /transcode 4K videos for Plex?
I'm just here taking up space, I won't spend that kind of money.
How loud is it?
Gotta love those musical rodents
Sure do
I'm just here taking up space. 😅
If you have a time machine you could go back in time to stop yourself but if not then. I am just taking up space.
I'm just here taking up space... again.
l Bought the same motherboard but for the life of me it won’t connect to the internet tried many settings no IPv4 or 6 sends but wont receive any ideas would really help Thanks !!!!!!!
Funny story.. those chassis are used by one of our client's vendors as NVRs
I'm just here taking up space 🤷♂️
I am surprised you didn't go for the 8 core version
These were much more easily available
I’m just here, late, taking up space.
"I'm just here taking up space"
-Me
well said
Lol Here just taking space
My man!
i have a whole cabinet filled with stuff I "don't need" ... sigh