Setting Up My First Proxmox High Availability Cluster
Vložit
- čas přidán 12. 07. 2024
- This video is sponsored by Tuxedo Computers and the Aura 15 Gen 2.
Configure and buy one here: dbte.ch/aura15gen2
/=========================================/
In this video I'll show how I set up my new 3 node Proxmox High Availability Cluster.
Thanks to @ZimaBoardOfficialChannel for providing the devices that made this video and this setup possible!!
Here is the hardware I used:
ZimaBoard: dbte.ch/zima832
64GB USB Drive: amzn.to/3EhhP8e
1TB SSD: amzn.to/3A1EJha
/=========================================/
Timestamps:
0:00 Intro
0:52 Sponsor
3:30 Creating a Cluster
6:45 Configuring Storage
10:31 Deploying a Container for Replication & High Availability
13:30 Configuring Replication
15:45 Configuring High Availability
16:37 Verifying Operation
Proxmox Playlist:
• Why I Switched to Prox...
/=========================================/
Get early, ad-free access to new content by becoming a channel member, a Patron or signing up for the members' only website!
✅ / dbtech
✅ dbtech.fans/
✅ / @dbtechyt
/=========================================/
The hardware in my recording studio is:
✔ Custom PC w/ Ryzen 2600, 32GB RAM, RTX 2070, Assorted Storage
✔ Panasonic LUMIX G7 4K Digital Camera: amzn.to/3IGEOcb
✔ SAMSUNG 34-Inch SJ55W Ultrawide Monitor: amzn.to/395g9BZ
✔ LG 27UK650-W 27” UHD IPS Display with HDR 10: amzn.to/398pg4S
✔ WALI Premium Dual Monitor Stand: amzn.to/398AiqM
✔ Neewer Lights: amzn.to/3nZcoSX
✔ Light Power Supply:amzn.to/3Konpqf
✔ 55" Gaming Desk: amzn.to/3AkgHgw
✔ Sabrent USB-C Hub: amzn.to/3qFcwbV
✔ Das Keyboard 4 Professional: amzn.to/3G9rPxM
✔ Fuqido Big and Tall Gaming Chair: amzn.to/3IGegrq
/=========================================/
The hardware in my current home servers:
✔ Synology DS1621xs+ (provided by Synology): amzn.to/2ZwTMgl
✔ 6x8TB Seagate Exos Enterprise HDDs (provided by Synology): amzn.to/3auLdcb
✔ 16GB DDR4 ECC RAM (provided by Synology): amzn.to/3do7avd
✔ 2TB NVMe Caching Drive (provided by Sabrent): amzn.to/3dwPCxj
✔ TerraMaster F5-221 (provided by TerraMaster): amzn.to/3IfH2QD
✔ 5x6TB WD Red Plus NAS: amzn.to/3LnbPvC
✔ 8GB DDR3: amzn.to/3kfLTX3
✔ TerraMaster F4-423 (provided by TerraMaster): amzn.to/3kjUms5
✔ 2x8TB Seagate Barracuda Compute: amzn.to/3xBAO95
✔ 16GB TEAMGROUP Elite DDR4: amzn.to/3MzzFV9
✔ 512GB Silicon Power NVMe Caching Drive: amzn.to/3MzkBae
All amzn.to links are affiliate links.
/=========================================/
✨Find all my social accounts here:
✅ dbte.ch/
✨Ways to support DB Tech:
✅ / dbtech
✅ www.paypal.me/DBTechReviews
✅ ko-fi.com/dbtech
✅ Cashapp: cash.app/$dbtechyt
✅ Venmo: venmo.com/dbtechyt
✨Come chat in Discord:
✅ dbte.ch/discord
✨Join this channel to get access to perks:
✅ / @dbtechyt
✨Hardware (Affiliate Links):
✅ TinyPilot KVM: dbte.ch/tpkvm
✅ LattePanda Delta 432: dbte.ch/dfrobot
✅ Lotmaxx SC-10 Shark: dbte.ch/sc10shark
✅ EchoGear 10U Rack: dbte.ch/echogear10u - Věda a technologie
Thank you so much for this video :) Lots of help. Subscribed!
Thanks for the demo and info, have a great day
Thanks for watching!!
Great video again David, i love proxmox!
Me too! Thank you so much!!
👍This is the best video of the year on all of CZcams. Cannot thank you enough!
Wow, thanks! I really appreciate your comment!!! Glad the video was helpful :)
@@DBTechYT
I am keeping an eye out for the 3 Steam Decks ceph cluster that 45 Drives channel will be releasing later this week.
That sounds amazing!!
@@DBTechYT
It nicely bookends 2022, somewhat the year of all things Steam Deck. Their vid drops on Thursday I think. I gotta watch your latest video migrating to zfs tonight when I have time. Kindest regards.
Great explanation - been running 3 nodes on ceph but am planning to move over to zfs. Thanks
Looking good with the cap!
Thanks!! I got it from www.lamoodbighats.com/ Had to find a place that sells decent looking caps for big heads like mine lol
Thanks for this video!
Great Video Loved It. Thanks for the inspiration.
Thanks for watching!
Great video, proxmox is a really nice setup for a homelab.
I like the lightweight containers, but I also run kubernetes and traeffic
You can run LXC/LXD/Docker containers and K3B on Proxmox. No reason I can think of that traeffic wouldn't work also. Proxmox is much more capable than just for "homelabs"
Nice video! If you haven't already setup Proxmox Backup Server, do so! It's amazing. Does the obvious thing of backing up VMs, and containiners but also does dedup, replication to another backup server and tape support. But I also like that it supports backing up bare metal Linux PCs too.
I've definitely got a backup server set up and configured to back up once per day and verify the backups 🙂
Yes. Really great setup. I made a PBS VM on my windows TV PC. It's always running anyways so perfect for a backup server.
Excellent tutorials and walkthrough! thank you for sharing.... I watched several of your videos and do have a questions. I see you talked during the wrap up about CEPH and watch your other videos and also see you move away from CEPH due to the overhead (similar to your reason for leaving OMV and moving to ProxMox). Have your found a sweet-spot on storage implementation for proxmox? balanced resource usage and failover time in HA?
thank you again!!!
Always interested Dave.
This is really interesting and you made me wanna buy more hardware just to play this 😂
yeah, same here. I don't blame him though. I've had issues for a while when it comes to buying too much tech. haha.
Incredibly well done, as always, and I see the where we're headed 😀. I noticed you're using Omada, I just started with the line. Have the router and waiting to pick up the AP to replace the FiOS router I'm using as an AP for now
Thanks!! Loving the Omada setup quite a bit! Just got it and am still learning, but it's a nice learning curve :)
@@DBTechYT are you using the Omada router too? Just curious if you're running a separate firewall in front of it or letting it do the firewall/load balancing on its own. I'm in the same boat of just getting into the setup
I am using the router, too. I'm letting it handle everything and haven't had any issues yet :)
Honestly proxmox makes it so easy! Only problem I ran into was using just two nodes. Ended up giving my main node two votes and everything worked fine. Getting a pi 5 to use as the third node.
I didn't think Proxmox supported ARM architecture? Only x86?
@@connor7439 But any Linux machine can run a QDevice - a small service that communicates with each node via SSH and whose sole purpose is to break a tie to ensure a quorum.
Great video, thanks! It seems I cannot replicate unless I have ZFS on both ends... But still very usable, thanks!
Hi, great video...
just a suggestion: in a set like yours I will try definetively with a true network storage system like gluster.
Is not ceph but works very well with proxmox and with HA. The migration from a host to another is blazing fast and recover VM from a failed node faster than a replica solution like this.
It's a viable solution between ceph that require nore resources and little cluster like this.
In you case you can allocate the second ethernet to handle gluster.
Ceph doesn't use much resources you can set the ram amount but you need 10gbit ethernet
Awesome
Thank you so much!!!
Hi. Nice video.
Seems that after 8:57 where you create the storage for each node (with the same name as you ve mentioned) you ve forgot to show something,
else I can t explain the different behavior in my cluster.
What I mean by that is, if you keep checked the box on the first node only and not the others, then whichever node you re on gui you ll see only the Storage of the first node. All the others will be like they have no storage (even from each node's gui, it can t be seen).
The same goes if you go individually to node 2 or 3 and try to create a VM. At the tab where you need to choose storage space for the VM you can only choose local. No Storage. So something you missed to show I guess.
PS By the way you didn t check thin provision option for the Storage at Datacenter->storage->Storage->properties
New edit: The only way for the storage to be seen to other nodes too is go to Datacenter->storage->Storage->properties and inside that window list down at option Nodes all 3 nodes and check them. Then automatically that storage is visible to all other nodes too. Even though, what would be the purpose for all 3 nodes to see only the first one's storage and not theirs. In case of first node failure his Storage (which the other two see) would be absent so where is the HA and replication to this scenario?
The way I set things up works. I don't know what to tell you. I followed multiple other tutorials for my setup and it has been up since I made this video.
This is the MOST CRITICAL STEP that was completely missed in the video! If you don't do what dimitri says above, REPLICATION will NOT WORK! I love DBTech's vids. This comment saved me after hours of messing around. Thank you. This is the part to pay attention to: -> "The only way for the storage to be seen to other nodes too is go to Datacenter->storage->{name of your storage ie: Storage}->EDIT and inside that window list down at option Nodes all 3 nodes and check them."
I ran into the same issue (storage not showing on other nodes until updating datacenter-storage area to add other hosts). Many thanks for sharing! Great video btw
If all 3 nodes are pointed to the first zfs pool, are they not just using this storage? Where does it show defining in replication to use storage on node 2 or node 3? Or is the act of labelling it the same tricking it into using the same namespace so it just doesn't know any different? Also if the author of the video states something is 'very important' but doesn't explain why (turns out it's because he doesn't know), then he shouldn't be making the tutorial IMO
@@andrewjohnston359 Kinda harsh. At least dbtech's video's are opening up insights and the opportunity to investigate more. Though, it would be nice to get a definitive answer to your questions, as I also have the same.
Please keep making the videos dbtech, we're all learning here.
Excellent video. I really like your pace. I did not have to fast forward a all! If you have an application like postgres db running on your server how does that work when moving to a new server? So if a disk failed you would be at most 15 minutes old?
Thanks! Moving to a new server will result in losing any data that isn't backed up. So if you plan on moving to a new server, I would spin up the new server, then run a backup of the database on the old server, then restore it to the new server and shut down the old server. Shouldn't have much data loss at all that way
Nice clip, thank you! a question: must the cluster nodes have the same hardware configuration or can I have servers with different cpus, quantity of RAM?
You can mix and match
I tried doing this setup back on 6.x and failed, completely missed unchecking Add... ended up using Ceph.
Looking forward to seeing how you're going to migrate from a single node to the cluster.
Wow that was fast.
Do you know if this works via remote VPN or tunnel? I know it would be slower but would it work? I have 3 locations with internet. Was thinking of Tailscale tunnel on all three but have them in cluster and use HA?
Thx for the video!
Can I achieve what you describe here regarding Storage on zfs with only one disk? TIA
You can definitely set up proxmox zfs storage with just one drive
This is a great video. Could you show us how to configure a CEPH cluster? I've been looking into it and I haven't been able to find a good tutorial at all.
Thanks! I've actually talked to a few people who aren't keen on CEPH storage and have recommended using NFS storage to help with keeping overhead and complications to a minimum.
@@DBTechYT Interesting. I know you can't get to all the questions, so I thank you very much for your time in replying. I've been using NFS for a while now and I've heard so much good things about CEPH. So much so that it was worth doing a lab, but what you're saying makes sense. It's recommended that you have at least a 10G connection for the management due to the amount of chatter. That's a lot of overhead.
thank you so much for this video, it really helped me out setting up a three node cluster. I have one question though: let’s say that the first (master) node goes down and the lxc/vm’s move to the other nodes. you mentioned that after the first node comes back online the lxc/vm does not automatically move back, but continue to run on node 2 or 3. but how does replication work from that point on? asking as we set-up the replication from node 1 > node 2 and node 1 > node 3… if we continue to add data to the lxc will that only be available on the current node or should we set up replication for example from node 2 > node 1 / node 3? thanks so much for a great tutorial again and have a great weekend!
The replication should keep working and the nodes may even load-balance themselves as necessary and move containers around to make things more efficient.
@@DBTechYT probably the fastest response ever! thank you so much! you're the best!
hi i have a qustion please
do you know how to connect or join two proxomox locally seperated like one proxomox in a city and the other is in an other city
i don't know how t connect them to create a cluster ?
Hi David, any idea why it throws the error "no replicatable volumes found (500)" when trying to add a VM to replication? I'm using NFS storage in my cluster if it helps. Thanks
So does a proxmox cluster increase availability and stability or reduce the time for task completion by separating out the tasks, like, let's say, machine learning and data science calculations? Or is it both tasks are separated across the node, and high availability is there by design?
Proxmox is just a hypervisor. You can configure it to do all kinds of things, but it doesn't do any of the stuff you've talked about without multiple nodes working together in some capacity
Great
Thanks
Could you please do the same process with xcp-ng?
Maybe. I've never used it, but I may look into it :)
FYI: There are many videos on clustering for XCP-NG The YT channel czcams.com/users/TheTecknowledge Has videos for just about all aspects of XCP. I used to run that but found Proxmox a better overall solution.
Just have 1 proxmox server, so haven't try this out yet, but how would one go about doing this with ISCSI?
Can you do the same thing with the local storage if you set them up with ZFS at the initial install?
You should be able to buy you might have issues with high availability
@DBTechYT I have 3 nodes with a single internal SSD with Proxmox installed on. When I create a cluster, if i want to create a VM on node 2 there are no local drives showing. I'm at a loss at what to do here? I read it is not best practice to install proxmox to a usb thumb drive but you have and that would allow me to work around the physical limitation of the machines single SSD port and create the ZFS pool for HA, but is there any way to achieve what you done in this video using a single SATA SSD in each node with proxmox installed to it?
Thank you!! Very helpful video!
What did you send up doing? I am in a similar situation. Others hav used a USB thumb drive as well I see. Perhaps ifyou get a high quality USB thumb drive with decent read/write speeds maybe it'd fine? I learned from a Proxmox forum that ceph needs the entire internal nvme, and wont run off a partitition unforuantely.
same question here, what have you guys done?@@jenniferw8963
please do this demo again but with the ceph storage !!!
Definitely going to look into ceph
Running similar setup with three HP elitedesks. Why not CEPH right from the get go?😊
Honestly because I haven't learned how to do it yet lol
@@DBTechYT Dead simple. Will be good content for the next video 😜
woooow ^___^
I see your 3 Zimaboards on your server rack, Do you have a link to the shelves you have these Zimaboards on that are attached to your Server rack? I am looking for something to rack my Lenovo M900 tiny server and maybe more Lenovo servers in my future. This shelf looks like it is just the right size for the Zimaboards and I am looking for something to put my Lenovo M900 on...thanksjJim
So what I did was used some double-sided tape and stuck them to one of these: amzn.to/41bHlEm (affiliate link)
Then I just put that assembly on a standard shelf like what you might find here: amzn.to/43oloUo (affiliate link)
Some of the shelves I'm using in my rack came with the rack and the others came out of an EchoGear rack
@@DBTechYT thank you very much. Jim
Hello David,
I don't know if this is the right way to place a video request, but I'll just do it now.
Many people around me are currently switching from twitter to fediverse and I'm thinking about hosting my own instance of mastodon with Docker/NPM/Cloudflare, but unfortunately I haven't really found a good tutorial on the internet.
At 9:17 you mentioned the need to click add storage only on the first node and not the others but you didn t explain why is that
It's just the process for this setup
I am still little confused ..
Need to know if the below scenario will work.
I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work ..
Please guide
Once you install Proxmox, you can create a container and install Portainer in the Proxmox container and then install WordPress that way if you want. That's basically how I have all of my Proxmox containers set up.
@@DBTechYT thanks I understand 👍 now
The storage disks and size should be the same on every node?
I don't know that they HAVE to be, but I think it's best practice
One more q, it s doable to make a proxmox cluster with only 1ssd per node ?
@@meroxdev Absolutely. That's how I had mine set up until this morning :)
If something was missed in the video why dont DB Tech pull down the video and add the edit?
What are you talking about being missing? I can't think of anyone else who has had any issues with something being missed. Also, you can't just "add the edit" after a video has been uploaded. That's not how CZcams works.
The pain of pve cluster is when 1 cluster is down you cant login via web interface
Do you mean when one of the nodes is down that you can't login the web interface? If so, log into a different node's interface
@@DBTechYT well, i set up a cluster with two pve nodes. when A is down, i cant login from B via web interface. You have to ssh to B and reduce the number of expected node to login
@@tyrant-rex4126 Hi, you can, you didn't set the cluster correctly
For 2 nodes you don't have a voting system, which proxmox is based on, you must change the quorum.
# this is only made on one node, because it will be replicated to the other
edit /etc/pve/corosync.conf
# add this two lines in quorum "space"
expected_votes: 1
two_node: 1
after
provider: corosync...
Domain 1.1.1.1??????
Yeah. I misspoke. I realized it right as I said it, but didn't clarify. I should have said IP or DNS. It's for CloudFlare DNS
Why are you talking so fast? English is not the native language of everyone watching the video.
Unfortunately, I'm not talking fast. This is me slowed down as much as I can without feeling like I'm trying to talking like I'm drunk. I guess I would suggest playing the video at a slower speed? dbtechreviews.com/i-talk-fast
You can always set the video playback to 0.75x or even 0.5x speed if he talks faster than you can understand :)
And IMHO telling someone else how they should talk like in a Video you get to watch for free feels kinda rude :/
@@Felix-ve9hs This has been going on for years. Long enough and frequently enough that I put that "I talk fast" page on my website and refer people to it. I've always just talked fast. When I try to force myself to talk slower, it feels like I'm being patronizing to myself and my audience. Thanks for your comment and support!!
@@DBTechYT Yeah there are a few people on CZcams that I have to slow the video down and it works perfectly, but you're not one of them. I certainly don't want to hear you sounding drunk haha.
BTW, just in case there are more with 2 nodes cluster, and I don't know if this was the reason, but when I was creating the ZFS on the 2nd node, and with the "Add Storage:" flag unchecked, it just won't add the storage to the node. Replication wont work, error:
2022-11-11 18:29:00 500-0: (remote_prepare_local_job) storage 'VMS' is not available on node 'pvematrix'
2022-11-11 18:29:00 500-0: end replication job with error: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pvematrix' root@--192.168.1.199-- -- pvesr prepare-local-job 500-0 VMS:subvol-500-disk-0 --last_sync 0' failed: exit code 255
Once again I'm not sure if this happening because the cluster is only with 2 nodes.
I ran into the same issue/error on my 3 node cluster. Once you have created the ZFS pool on all nodes edit the storage properties on the datacenter level and select all nodes on which you have the pool.
have you explored eks anywhere? @dbtech