Build a Kubernetes Home Lab from Scratch step-by-step!
Vložit
- čas přidán 9. 07. 2024
- Building a Kubernetes Home Lab environment from scratch might seem like a daunting task. However, it is actually pretty straightforward with the right tools in place. In this video, we are going to use Kubeadm to initialize a Kubernetes cluster from scratch in our home lab environment. Using Kubeadm takes some of the heavy lifting out of the underlying complexities of Kubernetes. We look at how to build the Kubernetes home lab cluster from scratch step-by-step. Join me in the adventure!
Subscribe to the channel: / @virtualizationhowto
My blog: www.virtualizationhowto.com
_____________________________________________________
Social Media: / vspinmaster
LinkedIn: / brandon-lee-vht
Github: github.com/brandonleegit
Introduction - 0:00
Talking about the lab environment and virtual machines needed - 0:53
How to leverage virtual machine snapshots in your home lab when learning Kubernetes - 2:13
Disabling virtual machine swap - 3:22
The actual commands to disable swap - 3:50
Installing Docker - 4:13
Installing CURL - 4:25
Adding the repository key for Kubernetes - 4:30
Adding the actual Kubernetes repository - 4:45
Running apt update - 4:55
Installing kubeadm, kubelet, kubectl, and Kubernetes - 5:02
Explaining why we are only running commands on the Kubernetes master node - 5:16
Running the kubeadm init command - 6:00
kubeadm init command finishes successfully and have the kubeadm join command - 6:20
The worker nodes are now caught up with the master node - 6:58
We still have the special join command to copy and paste into our worker nodes - 7:20
Notice the output of workers. They have successfully joined the Kubernetes cluster - 7:45
Kubectl commands are failing. This is because we haven't copied our kubeconfig to our HOME directory - 7:54
Nodes are in the not ready state. This is because we haven't deployed container networking - 8:30
We are going to deploy Calico for container networking - 8:45
After container networking we check the STATE of our Kubernetes nodes - 9:16
Wrapping up and thoughts on building a Kubernetes Home Lab from Scratch - 9:45
If you would like to read a full write up of how to build your Kubernetes Home Lab from scratch using virtual machines, read my blog post found here:
- www.virtualizationhowto.com/2... - Jak na to + styl
This tutorial was quick and to the point. Was exactly what I needed to get over an issue I had when I was installing my k8s. Thank you!
Thank you. Completely demystified building my first K8s cluster. #subscribed
Appreciate the steps. I also enjoyed the multicast storm in the background.
Just completed "Kubernetes Cluster in Minutes in VMware vSphere using Rancher" and now I see new tutorial on the same topic, this is great! You're one of the best instructors on YT please keep it coming 🙏
Maor,
Thank you so much for your kind words! It is comments like this that make the time and effort of creating content for the channel worthwhile. Thank you again!
Brandon
Very straight. Thank you.
Hey. man
Thanks a lot for this video
I was stuck a bit with deploying k8s cluster in my home lab, but all videos I've watched before was something like "K8s and minikube in 1 hour"
So now I've managed with deploying cluster on my CentOS 7 VMs and Proxmox as hypervisor
Very helpful, thank you so much.
this is great stuff. thanks for sharing
clean elegant and intuitive. thanks.
thank you Shang!
Installing K8s as a container instead of a VM won’t help save some time or Do you recommend better to install it on a VM? Good video
Great totorial ..
I'm looking for one thing
I have a pc, which I don't want to install it as a node .. could one install only kubectl on it and just give commands to cluster
something like docker context .. where I just need docker-client and I'm switching contexts .. is that possible ?
Hi!
thanks for the really good video, what versions did you use?
So can i Install Kubernetes on different machines in KVM and just run a cluster on different architectures or do I have to differentiate between x86 on one side and Arm on the other ?
calico doesnt install (calico pods are stuck at pending and nodes are stuck at NotReady). Do you know any other methods to install calico, especially newer versions (compatible with new Kubernetes)? I would really appreciate the help, been stuck for a week almost.
Thanks!
I am using virtual box I have 4gb ram in my computer
What adapters are you using in your vms ?
how did you create those 3 virtual machines?
outstanding content
noob here, will this be enough to work an environment for the K8s admin exam. Thinking on using three VMs on azure tought.
Is there any way that this could be achieved with no virtualisation, fully installed only on one OS?
My machine does not support virtualisation, and cannot find any BIOS updates
in a other video i saw a guy connecting the nodes with nat und bridges together. Did you do such thing too?
Awesome video tutorials you've come up with. Also, very professional and to the point. Keep it up ;-)
i'm fond of supermicro for lab environment, learning good stuff here.
Thank you Angry Bird! I really appreciate that and comments like this makes the time and effort involved worth it.
Sorry, missed how master and worker nodes are created?
Amazing and complete tutorial for a beginner. Short, concise, and it makes sense. But it would be good to explain like what calico does.
@lruini626, calico is the container networking overlay technology used in the Kubernetes cluster.
I have followed the same steps exactly (1 node only) but my kube-system pods crash resulting to rendering the one node cluster off
Something with calico pod starts the issue but I cannot identify the error.
I have looked on the configuration of containerd, dockerd, and logs but I have made zero progress so far.
Anyone has faced similar issues ?
connection refused to the server :6443, - did you specify the right host or port?
Hello, Thanks for the video
I am Ryad and I am new to Kubernetes. I have an issue with my cluster. I am trying to build a 3CP and 4W cluster, but when I tried to join the second control plane to the first one, it crashed, and I couldn't do anything.
So here is my question:
Do I need to install a simple cluster first like you did in the video and add the control planes after, or can I directly install all 3 control planes?
Which type of Linux do you recommend (Debian, CentOS 7, Ubuntu) for which version of Kubernetes?
Thanks a lot.
Have a nice day.
please for the nodes i'm gonna use a bridge network or NAT , thank you in advance
I did copy the configuration files but it still says connection refused when I run kubectl get nodes.
Also the second node does not connect to control plane with key. Timeout error.
i had problem with running kubectl command on worker nodes
error message: 'The connection to the server localhost:8080 was refused - did you specify the right host or port?'
Awesome video man! thanks. I was taking a course on K8 but It only went over how to setup using premade K8 tools on teh cloud. This is much beter and now I know how to create a local cluster and commit to master, as well ss implement from scratch on cloud (or at least I have an idea on where to start from). Stellar video man!
awe713, thanks for your comment! So glad it was helpful!
Thanks for the insight.
Please what did you do to be able to copy and paste across the nodes?
@tech-jah.tech-talents create a topic over on the VHT forums here: www.virtualizationhowto.com/community. Thank you for the comment!
It will run for a moment, the pods restart a few times "proxy and manager", then I lose all connection to it with kubectl? Then randomly it works and I can see the pods yet again being up. Then it starts over.
Anyone had this happen?
Error:
dial tcp *.*.*.*:6443: connect: connection refused
Very nicely done. Can you show to expose a working NodePort or LoadBalancer on the local network? Assume you have a bridge, br0 working, and all VMs get their own DHCP address from pfsense or some router.
Hey,
Great Video. Was there any version mismatch with Docker and Kubernetes since docker is going to be deprecated with Kubernetes? Thank you.
It will not get deprecated
Arjun, thanks for the comment! Containerd will be supported moving forward
Just awesome 👏
I’m K8s certified (CKA & CKAD) but I can still tell you that you tough me something. Straight to the point and clear. Keep doing this nice work. I’ve joined your CZcams community.
Can you do a follow up tutorial how to add another Master node to make it fully HA?
What WebUI would you recommend to manage a Home Kubernetes cluster similar like we use a cloud service like Azure?
@ralmslb, create a topic over on the VHT forums here: www.virtualizationhowto.com/community and let's discuss. Thank you for the comment!
Where do you get kube.config file?
How did you get 3 hosts?I used virtual box. Can I get 3 instances and get them working as master and working nodes?
Sure thing Manish. It doesn't matter on the underlying hypervisor. You should be able to spin up 3 nodes in VirtualBox and have a master and 2 workers.
What version of Ubuntu you are using because I did a lot of troubleshooting and I still have the same issue with running "kubeadm init" ?
Andres, I believe when i made the video this was using Ubuntu 20.04 and before the change was made to remove Docker shim. Can you post the error you are getting?
Why did you run your commands using sudo when you were already root?
That threw me off. Installing as non root will not work. Took me a while to catch the '#' prompt sign as I depended too much on his use of sudo to imply non-root. Lol.
thanks for the material , really helpful. i got this error though , any workaround ? ------ The connection to the server master:6443 was refused - did you specify the right host or port?
Ray, make sure you have exported your kubeconfig and copied it to the right location:
mkdir -p $HOME/.kube && \
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && \
sudo chown $(id -u):$(id -g) $HOME/.kube/config
@@VirtualizationHowto thx it works for a while then same connection refuse happens !!
like it works then after a while same problem happens
why sudo all the time if you are root anyways?
Just to confirm all the commands you ran on the master besides kubeadm init needs to be run on the worker nodes?
@wallmachine8780 yes that is correct. Also, please checkout the updated steps here: www.virtualizationhowto.com/2023/12/how-to-install-kubernetes-in-ubuntu-22-04-with-kubeadm/. Please join the forums if you would like more personalized help as we can discuss further there: www.virtualizationhowto.com/community
why don't you use an open source hypervisor at home?
Thanks. This video sums it up very well. Thank you. One question though. Do you know if it's possible to add a node to cluster created with Docker Desktop?
What is the purpose of setting up a k8s cluster? Is it the same as testing using minikube?
Lara, a k8s cluster allows having the true redundancy needed for business-critical apps. You don't want to have a single node cluster for production as it would mean if your single Kubernetes host went down, so would all the containers. Having multiple hosts in a true cluster protects you from these types of scenarios.
i got this error when doing kubeadm init ----- error execution phase addon/kube-proxy: unable to create ConfigMap: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
Ray, check out here: github.com/kubernetes/kubeadm/issues/2701
How do I change this so I can use this behind a DHCP isp? I have a DDNS, but I can't seem to figure out how to change the config to point to a FQDN in kubernetes?
@josef_kul Thank you for the comment! Sign up on the forums and I can give more personalized help here: www.virtualizationhowto.com/community
But why docker? Why not containerio/something else?
Fast tutorial!!!
And yeah installing k8s from scratch is pretty easy and straightforward.
And even setting up a load balancer: Metallb is easy.
A distributed storage could be a bit more komplex. At least I haven’t done it so far.
Awesome tutorial
Thank you so much for your efforts
Subscribed
Thank you Absesse!
Would the process be the same if I have my VMs as EC2s in AWS?
this video shows how to in aws, ec2:
czcams.com/video/-lHKvZ2qYMM/video.html
thanks for the tutorial, just a quick question: why are you running commands as sudo when you are already root?
Using `sudo` as root... outstanding move.
Smokey thanks for the comment! Makes it easy for any to follow along regardless if you are using root or a sudo user :)
Can this be applied to standard machines. I have 3 mini lenovo desktops that I want to cluster together?
i guess if you install linux OS on those mini lenovos you should be able to do it
It should work the same. I am going to try on 3 dell optiplexs running Ubuntu
What to do if the kube-apiserver just dies after a short while?
connection refuse, nothing listening on port 6443
Thank you for the comment! Sign up on the forums and I can give more personalized help here: www.virtualizationhowto.com/community
I got error after error, the links in the description showed different steps from what was shown in the video. I needed to adapt a bunch of stuff to get be able to get it somewhat running, looking for information in other places at times to fill the gaps.
Either show every command and step properly in the video, or have a a companion article with the proper steps, otherwise this video will cause more confusion that anything else.
Can you please put me on track how you have managed to copy and paste the JOIN command in the worker machines? If I am running in my local 3 different VMs in Virtual Box to simulate the ControlPlane and worker nodes, how you can communicate the copy-paste between them? I totaly got lost here. Thank you.
on all the VMs, Setup OpenSSH-server...... use 3 Terminal Windows on host to SSH into VMs....... then Copy Paste
@@thekarananand I will try thank you so much
How did you split your screen into three. I tried this several times to no avail. You said you would guide every step of the way. That's my problem with self learning, one small problem and you are stuck.
They are just three terminals
Hola, qué herramientas de backup recomiendan para Kubernetes? gracias, Saludos
Great video sir -hope you share your all of your commands on that subject ☺️
did you use bridged networking or NAT for your VMs? I am thinking for ingress bridged would be easier? Thanks
Maybe I'm just unlucky, but I encounter a persistent problem after adding the 2 worker nodes. The control plane will display the master node, but after adding the 2 workers, executing "kubectl get nodes" will result in a connection (:6443) error. I attempt to "export KUBECONFIG=/etc/kubernetes/admin.conf" again along with copying the admin.conf and assigning permissions. The first "kubectl get nodes" is successful, but after that it errors again. I think the master node is failing. Any suggestions?
I just tested it again with only the master control plane node and it failed after running "kubectl get nodes" twice - 3rd attempt failed.
Ended up ditching Calico for Flannel and everything is working now!
Great video! Can you please make a tutorial on how to create multiple K8s clusters on a single server?
Hi, I am new to K8S myself but I don't believe there is a reason to create cluster within a node. Instead, you can deploy multiple pods to a node. The node can only have one IP address and therefor you cannot create multiple nodes (cluster) within a single node. You can deploy multiple pods to any given single node (which may be what you are actually looking for); To do this, you can use "Deployment" object within your {deployment-config} file.
@@ase713 Or, in this video, the three VMs shown form one cluster and all three VMs are in one server. Simply repeat the process with another set of VMs for another cluster. Controlling the two clusters with kubectl will then require context switching. But heck, why not simply have one cluster with six VMs and partition the resources with namespaces?
"sudo kubeadm init" just doesnt work coz of kubelet. Wasted 2 days trying and debugging with diff diff Ubuntu versions but its still the same. I dont know how for him it goes so smooth.
It seems like the kubelet isn't running or healthy
shantidut, thank you for the comment! Try the following. I have tested in my lab and this works when receiving the same error you are receiving with Ubuntu 22.04. Most likely you are running into a cgroup driver issue...it must match what is used by kubeadm or the kubelet won't come online.
Run the following to check if it is running systemd:
------------------------------------------------------
if grep "SystemdCgroup = true" /etc/containerd/config.toml &> /dev/nul; then
driver=systemd
else
driver=cgroupfs
fi
echo "driver is ${driver}"
----------------------------
If it returns systemd....run the following two commands:
mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
systemctl restart containerd
You should be able to run the following command and see that it grabs the images without error:
kubeadm config images pull
If the images pull correctly, you should be able to successfully bring up the cluster:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
Take a look at the official Kubernetes documentation on cgroup driver here: kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/
@@VirtualizationHowto bro tysm i spent so much time lookin for a solution to this!
At 8:07 I am intermittently having this error appear even after following instructions. Ubuntu 22 LTS. Also having the same error but regarding to port 6443. I can have success with running kubectl after re-running the export command (while as root), but the very next time kubectl is run I get the same error again. I've tried this on a VM and on a bare-metal machine. It's kinda driving me insane. :-) Advice please?
Having the exact same issue and also driving me insane. Have you managed to figure out what is causing it?
@@abell12 Hi. No, I never did figure this out, and gave up.
@@sam2b77 Hey, I actually did figure this out. Apparently it's some issue with Ubuntu 22.04 with containerd and it not creating the config file correctly.
You can fix it by running these commands
mkdir -p /etc/containerd/
containerd config default | sudo tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
@@abell12 Great thanks! I was looking for a solution for a long time, it worked!
Nice! :-D Now my cluster works fine on my PC. I had some trouble with the calico-part. I guess they have changed their name to tigera, or maybe the whole project changed?
1. 3:48 why u need to run sudo if u already root?
2. could u install as non root user
nbctcp3, thank you for your comment! I generally show as sudo even if root as this allows the steps to work even if non-root. You should be able to configure this as non-root using sudo commands.
How is creating a cluster with machines of different CPU architectures?
I wouldn't advise doing that. While it can be done, it can lead to weirdness and other issues.
@@VirtualizationHowto I have a homelab for learning purposes. Gonna try to have a kubernetes cluster with x86_64, arm 64 and arm 32
Wish me luck haha
🔴 Kubernetes 1 : Comment installer correctement kubernetes master et son dashboard 🚀
czcams.com/video/RUm_t2P1Uws/video.html
hello and thank you
But we had a lot of problems.
Thank you for sharing this with everyone. One question I have - On the Kmaster node - I lost the join information that I needed to join the kworker1 and kworker2 to the cluster - how do I get the join command with the key to display again?
Mike this command will reprint: kubeadm token create --print-join-command
is this video outdated? i followed the instructions and got [ERROR CRI]: container runtime is not running. and then i found K8s dropped docker support...
Yang, most likely you are running into a cgroup driver issue between systemd and cgroupfs. Do the following:
Run the following to check if it is running systemd:
------------------------------------------------------
if grep "SystemdCgroup = true" /etc/containerd/config.toml &> /dev/nul; then
driver=systemd
else
driver=cgroupfs
fi
echo "driver is ${driver}"
----------------------------
If it returns systemd....run the following two commands:
mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
systemctl restart containerd
You should be able to run the following command and see that it grabs the images without error:
kubeadm config images pull
If the images pull correctly, you should be able to successfully bring up the cluster:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
Take a look at the official Kubernetes documentation on cgroup driver here: kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/
@@VirtualizationHowto Not the original poster but I have the same issue... I tried your solution but I get "unexpected token near `echo`". So I removed the echo statement and got no response.
what do I have to do?
Question from an ignorant person in the hope someone with the knowledge will answer and either explain or guide me on my learning. I have 3 laptops, one of them my use-daily one. The others with smaller processing power, I am running Llama.cpp on my main laptop. Will the Kubernets help me to achive a greater processing power if I turn the other two into a slaves, or are Kubernets only for web base applications?
Would you rather virtualize some nodes to run kubernetes on (as you show in your video), or make a bare-metal kubernetes installation of a single controller node, considering you'd only have a single spare PC to run your kubernetes home cluster?
I'm using proxmox with 4 Ubuntu Server VMs as K8S nodes, installing with kubeadm, to get that different experience.
Thanks for showing us how to do it from scratch.
I suggest not cutting the terminal output part to make the video short!
swiss knife👌
Kubernetes' built-in dockershim component was removed in release v1.24 , so Docker will not be supported anymore i guess
'docker' doesn't mean anything in this context. You can still build containers with docker and run them with containerd, cri-o or whatever.
@@endymionspr9916 exactly, it's the runtime, and only an "issue" if you are upgrading preexisting environments that utilize the Docker runtime; wasn't too much of a pain. Just gotta cordon off and drain your nodes, or simply scale down replicas, followed by removing docker and updating kubelet, depending on which runtime you desire.
You don't need to use sudo when you're already root.
Thanks for sharing the info. Is this going to be a series or was this just a "bit sized" one off video? Cheers! 😉
Frank, thanks for the comment. That is a great idea and definitely one I will consider. A series would be a great way to cover additional topics. Thanks again!
This is a great tutorial, thank you for that. I'm running into an issue. I have been searching google for answers. I got this error and never got the join command output. If you have any pointers, i'd greatly appreciate it.
[WARNING SystemVerification]: missing optional cgroups: blkio
error execution phase preflight: [preflight] Some fatal errors occurred:
Marco what OS are you using?
Hi Marco, had the same issue with Ubuntu 22.04. Also had issues with the kubelet constantly crashing and restarting. Ubuntu 20.04 worked a treat. @VirtualizationHowto - thank you, great tut!
If you would have put the commands on the note paid, it would have been awesome
Syed, here you go :)
##Disable swap
sudo swapoff -a && \
sudo nano /etc/fstab
##Install Docker in Ubuntu
docs.docker.com/engine/install/ubuntu/
#Install Curl
sudo apt install apt-transport-https curl -y
##Download Google GPG key and add repo
curl -s packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add && \
sudo apt-add-repository "deb apt.kubernetes.io/ kubernetes-xenial main"
##Update and install Kubernetes tools
sudo apt update && \
sudo apt install kubeadm kubelet kubectl kubernetes-cni -y
##Build the Kubernetes cluster
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
##Copy kube config
mkdir -p $HOME/.kube && \
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && \
sudo chown $(id -u):$(id -g) $HOME/.kube/config
##Kubernetes overlay networking
kubectl apply -f docs.projectcalico.org/manifests/calico.yaml
##Kubernetes join command if needed
kubeadm token create --print-join-command
##Problem with kubelet failing?
Run the following to check if it is running systemd:
------------------------------------------------------
if grep "SystemdCgroup = true" /etc/containerd/config.toml &> /dev/nul; then
driver=systemd
else
driver=cgroupfs
fi
echo "driver is ${driver}"
----------------------------
If it returns systemd....run the following two commands:
mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
systemctl restart containerd
You should be able to run the following command and see that it grabs the images without error:
kubeadm config images pull
If the images pull correctly, you should be able to successfully bring up the cluster and rerun the kubeadmin init command
@@VirtualizationHowto thanks this is of great help.
I’ve tried the clusters I can’t get it I don’t know how to wire them. I don’t know how fuck thing you guys are talking about. I’ve got a lot of computers I want to put together like we talk about 12 or 13 computers and summer new summer old I’d like to have them all running in the cluster. You guys just confuse the hell out of me but I like to came anyways didn’t want you to waste your time. Thanks.
I thought the purpose of using kubernetes in a home lab was to have each node on a different physical machine?
@user-tg3nd6me9l thank you for the comment! It would be in a true production scenario. Most will run VMs on different physical hosts and make sure these are always on different hosts for that reason. In a lab environment for learning, many may run these on the same host to create their K8s cluster.
Sadly this doesn't works in local environment (hypervisor KVM) I tried following your video here are the issues I faced.
1. Docker isn't supported so had to go in seperate guide.
2. Faced issues with iptable which i had to search manually to fix
3. faced issue with kubeadam init where it's giving error that unable to create service account. ( Was not able to fix this)
An updated video would be really helpful where the VMs doesn't have a public IP and it's totally done in internal network
Aviso para quem veio em 2024, este tutorial não funciona mais para instalação, é só um aviso.
sheeet dont work. init command fail's with multiple errors.
Time for google.
Problem with all these videos is they stop short of remote control through RBAC or Rancher. No one should be administering their cluster from the control node.
Ari, definitely a good shout and RBAC is a must for production workloads.
Installation is failling, please let me know how to install in right manner
is it me or is his voice and face not synced right
Yeah it’s A bit off but not much.
it was okay until you mentioned calico and didnt talk much on that.
@64242359, hop over to the forums and I can give you more personalized help www.virtualizationhowto.com/community
Are you Bruce Lee long lost son?
guilty as charged, j/k :)
Not exactly "from scratch".