Use Docker Swarm! Auto Deploy Script with Highly Available Storage - GlusterFS
Vložit
- čas přidán 19. 06. 2024
- Docker Swarm is a great option to understand container orchestration, and might be the perfect choice for your homelab. It provides high availability and failover for services you are running. This is great to keep your favourite app running and is really useful when it comes to maintenance.
Docker Swarm Script:
github.com/JamesTurland/JimsG...
Docker Swarm Documentation:
docs.docker.com/engine/swarm/
Recommended Hardware: github.com/JamesTurland/JimsG...
Discord: / discord
Twitter: / jimsgarage_
Reddit: / jims-garage
GitHub: github.com/JamesTurland/JimsG...
00:00 - Overview of Docker Swarm & Benefits
04:12 - Script Walkthrough & VMs
15:26 - Swarm Deployment
18:10 - Checking It Works - Portainer
22:15 - Testing Failover & High Availability
24:25 - Outro - Věda a technologie
Thanks to You I now understand Traefik and almost hole my lab is on docker with traefik. Next step docker swarm and kubernetes :) good job like always!
That's awesome, thanks.
@@Jims-Garage Which ubuntu 23.04 image did you specifically use? Just wondering? Is it this one (lunar-server-cloudimg-amd64.img) instead of (lunar-server-cloudimg-amd64-disk-kvm.img) Thanks
@@chrisumali9841 cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img
Thanks for the demo and info, Happy Holidays, and have a great day.
Happy holidays!
Thank you, one of best in homelab, As always, excellent deployment process with detailed explanation.
Very kind, thanks!
Excellent walk through! 🎉
Thanks
I am glad you and a few others (I found videos posted earlier this year) are talking about Docker Swarm. This technology should not be left behind because while Kubernetes has its place, it is overkill and unnecessarily complex for some applications. Docker Swarm and Gluster running on a clustered hypervisor is an awesome highly available solution. I hope your video brings a renewed interest in the continued development and support for Docker Swarm.
Thanks, appreciate the kind words. I completely agree, Swarm is probably the sweet spot for homelab use.
I completely agree. I've been using Docker standalone for a few years and recently upgraded to Docker Swarm with Gluster. I absolutely love it.
I tried Kubernetes as well (K3S). I was able to setup Rancher with Longhorn for storage, but I wasn't loving it. It was overkill and extremely complex for what I wanted to do. While I won't be giving up on Kubernetes, I'm glad I went back and tried Docker Swarm after because I appreciated even more with how simple it is.
I'm hoping that within a few years Kubernetes will become easier as innovations are introduced. Surely there will be improvements.
Awesome video. Thank you for being thorough.
Glad it was helpful!
Glad it was helpful!
Excellent, thank you!
You are welcome!
Great video! I want to do this in my VMware home lab.
You should!
Really like all of your videos, your home lab like how I want mine to be, with the exception of the Sophos firewall!! (turned my XG into a PFSense box) Except I'm using XCP-ng at the moment as it seems to be more in use in the commercial environment than Proxmox.
Thanks. Both of those are solid alternatives, good work
Hey Jim. I had used docker swarm long time ago and was really impressed. Too bad it faded out of fashion. As you said, it would be a very good fit for homelabs.
Totally agree. It's super lightweight in comarison.
Hey Jim, thanks for posting this. I haven't used docker swarm in a long time because I thought it wasn't support any longer but I will just back in and give it go again. Thanks for your hard work sir. I made try to redeploy this into a LXD service as well.
Thanks. From everything I've heard LXC doesn't play nicely with the overlay network.
Great video, been waiting for it for a while. A few questions/comments:
- Why not use tail scale so all nodes would see each other and avoid opening ports across different geographic locations?
- Why not use a proxmox container instead of the VMs? Any performance issues?
- Why not create a container template and easily add a clone to the swarm?
- What performance issues to expect with GlusterFS?
Thanks :)
1) This cluster is local, VPNs are not required.
2) LXCs are notorious for causing problems with the overlay network. Plus, VMs are more secure than LXCs due to using a different kernel.
3) I used cloud-init for creating the nodes, I mentioned this and referenced the video I have on it.
4) GlusterFS isn't the most performant, but as the data is replicated on each node it should be sufficient. If you want performance you're better off with Ceph or Kuberenetes.
Great stuff man thanks for sharing! Quick question though what if we have 3 proxmox nodes with ceph enabled and setup 3 docker managers and 3 workers on each of the proxmox nodes; from storage perspective do I need GlisterFS if I mount ceph using ceph-fuse package as a second storage for the VMs?
Ceph is likely a better option, and especially now that glusterfs is being retired. I'm planning to move onto it later.
Hey Jim.
I get it, Docker Swarm benefits with a number of worker nodes and all that jazz, sure.
But probably, I'm missing some distinctive detail, the most important thing - what are the benefits of a virtual cluster, whose nodes resides on the same physical server?
How this HA setup will benefit, for example, when file system starts to fail or some RAM issues occurs on the physical server itself? Or are these kind of setup designed to solve other problems, not related with some hardware issues?
Hey, you raise a valid point. There's less point on the same machine, but if you've watched my Kubernetes videos you will know that I split between physical Proxmox nodes. This gives all of the benefits you are asking about. On the same machine, the best use is that you can reboot the VM for updates without breaking your services.
@@Jims-Garage Turns out, I learned something, huh. Thank you for your videos and this clarification.
You can have a 3 node cluster, with all 3 being managers/worker combo’s.. and you always need an uneven number of managers for quorum consistency.
Thanks, Neil. That's good to know for people who want a 3-node setup.
Yes that's exactly what I am looking for! Further I need a script similar to what was in video for 5 nodes, where can I find it?
Thanks for the excellent explanation.
I have a question, after all the role of Docker Swarm is to continue work on another server. According to your video everything is done from one server. How to connect another server and install Docker Swarm
As long as networks are routable and the right ports are open, nodes can be on different machines and networks.
Hi Jim, Thanks for the video. I have a question about portainer; since the free version supports 3 nodes only, did you buy a license to support 5 nodes? Or is it possible to support 5 nodes without purchasing a license?
I use the community edition, there's no limit for nodes.
@Jims-Garage I for the life of me can't get this to work. It's a great script, but I can't get the SSH auth to work properly. I get a permission denied error whether I use the public key from the Proxmox host or the "admin" VM. I do have a SSH public key for my laptop as part of the cloud-init process. That could be the problem but I'm pretty green with Linux. I don't want to manually run the install process from the script but that's where I'm at now. Help please...
As always.....Good job man! I have a question: When I have configured everything as in this video, can I then install a Docker application directly using Portainer and I will have all the benefits such as HA etc?
Second thing: have you ever seen SWARMPIT? Maybe you've already seen it but it looks like a very nice GUI to "handle" all the Docker Swarm!
What do you think?
Yes, installing via Portainer will give you the benefits of swarm. I haven't used swarmpit as I find Portainer to be perfect.
I tried following your kubernetes videos, ended with only nginx running, too complicated to go immediately after docker (sooner or latter I'll understand it better, but now right now).
I will be trying this docker swarm very soon.
Great, it's a good introduction into the world of container orchestration. Hop on Discord if you're stuck.
Cool, does GlusterFS work also with databases? Do you need to turn off the cluster to do system updates?
It should work with databases with replicas set to 1. As with all databases it's recommended to backup using database tooling, and if you can't you should then stop the service first.
Hey Jim.
I was wondering if it's possible to run something like gluetun and qbittorrent in swarm. I can't seem to get it working. I tried using the shadowsocks proxy that comes with gluetun but cannot seem to get it working. Either way thanks fora great video :) Subscribed
Thanks, it should certainly be possible but I haven't done it I'm afraid. I use Kubernetes
thank you Jim
i have a question concerning the certificates
i have a 5 server cluster and want to spread the load over the several servers ergo than i need all 5 different certificates to be specified or can i use only one from my proxmox cluster master?
The best way is perhaps to generate your own key and use that. Otherwise, create on a single host and then migrate the VMs. Should honour the original key.
@@Jims-Garage thank you again Jim and will do the second option you suggested but first need to create 5 cloud-init servers...thank you for that tutorial too
Gluster is basically end-of-life (in a year) so I wouldn’t base a new setup on it, I think there are better options. It also has terrible performance in my experience, but that might be related to the amount of data.
I agree, there are more performant options, but it's a good option to get started. I'll look to replace glusterfs in the future.
@@Jims-Garage it isn’t really a matter of performance. Just like you wouldn’t install Windows 7 on a new computer, you shouldn’t build a new setup using Gluster. It’s fine to keep using it if you already are, but don’t invest energy into building on it.
Oh really, I was about to setup my own Now! Is there a suggestion for me? Which one should I go for in docker swarm if not glusterfs?
@@Jims-Garage First, thanks for a great video! In the process of setting up something new, do you have suggestion(s) for an alternative to Glusterfs for a hybrid CPU (arm, amd, etc) set up? Maybe BeeGFS?
@@raymerp9 thanks, sadly not anything I have experience with as I'm x86
I've set this up in a test lab with a 3-node manager cluster and use NFS as the central storage for volumes. Works great in my testing, but now I'm wondering how easy it would be to migrate an existing standalone environment into the cluster? Any comments on this aspect?
Should be a case of copying the data over and amending the deployment yaml. Should be easy to do.
@@Jims-Garage OK, but is that just for stacks or does that apply to containers as well? I'm testing "SnapDrop" right now which was created as a container; it does not seem to replicate like stacks do. Would I need to deplay everything using a stack?
Actually, I just discovered that you need to add these as a "Service" so that you can then indicate whether they are Global or Replicated. My SnapDrop test is working great now.
Thanks for your awesome video - docker swarm is still useful for the homelab, that's for sure!!!
@@scubeedu2 you need to deploy using the swarm yaml template. It looks a little different to a standalone deployment.
I’m adding this to the watchlist to check out later. Is this viable for homelab deployment? Everyone seems to suggest k3s lately.
If you have the time and resources then k3s (or preferably RKE2 - I have a video) is better. However, docker swarm is a viable option for a homelab that is far simpler and does most of what you'd want.
I have checked out your RKE2 video yesterday. Very nicely explained how to set it up, thank you. But it seems too resource intensive for a small homelab. I like docker and the simplicity of it, if it can be set up for HA and LB I think I’d give it a go.
@@NikolaNovoselec swarm will give you that
Could you explain how to fit traefik into this docker swarm setup?
I'll likely come back to swarm in the near future.
Finally tried to take this on today (2 Node Proxmox/CEPH). I tried with cloud image VM and they just hang on the -serial0 message when loading up the console. Tried with LXC containers, they will spin up and I can manually install docker and such, but the script from github appears to work, hangs a bit prompts me for sudo but the password appears in plain text and you have to ctrl+c to terminate.
So I am going to go to bed and try again in the morning.
Let me know how you get on
@@Jims-Garage after I said I was going to bed, I kept at is. No luck with VMs but in LXC containers they cannot be “unprivileged” or you cannot create the volume.
I seem to have everything working except the mount itself. I was able to write to fstab, but the actual “mount.glusterfs … /mnt” returns a mount failed error.
So that’s the next rabbit hole to dive into. And then of course back tracking to figuring out the VM side of things I wonder if it’s the issue you faced in the stream with the kvm image (I haven’t watched that yet) but my next order of trouble shooting it to you the regular cloud image.
@@Jims-Garage came back to it. I have it working on Proxmox VM's now. 3 managers 3 workers. might add 3 more workers (1 manager 2 workers per Proxmox nodes? maybe? )
With the news of RH dropping support for Gluster, do you still think this is a viable path to start on if one is about to deploy a network storage solution from scratch?
No. I'd look into ceph for now (albeit Kubernetes is perhaps where you might want to be)
@@Jims-Garage Ugh, those both sound way above my skill level. Thanks for the gut check though!
@@RyanMeray if you simply want to play with swarm and learn fundamentals this will be fine. It's just not something I recommend building around.
@@Jims-Garage My goal is to set up a HA 5-node cluster for file storage and backup. Looking like Proxmox VE with CEPH may check the boxes, I'll just have a lot of learning ahead.
how to setup 5 nodes with proxmox ?
Check my Kubernetes video on cloud-init. That will show you how to create multiple VMs quickly.
first xD
My hero 😻
GlusterFS performance is pathetic even after allocating a separate physical network for its brick. Secondly, its quite heavy on processor. I would certainly look into any other fs giving similar features.
Local performance seems acceptable though?
I am curious, do you have any benchmark available? Would NFS be a better solution?