Low Power Cluster - Small, Efficient, BUT Powerful!
Vložit
- čas přidán 22. 06. 2024
- I built a lower power, efficient, and near silent server cluster! Although this cluster is small and efficient, it's still powerful enough to run a high availability Kubernetes cluster with many services running in High Availability mode! There are so many options with running a small cluster like this, the possibilities are endless!
Video Notes: technotim.live/posts/low-powe...
A HUGE thanks to Datree for sponsoring this video!
Combat misconfigurations. Empower engineers.
www.datree.io
Disclosures:
- I purchased everything with my own money
Intel NUC 11 - amzn.to/43TK8nS
Intel NUC 12 - amzn.to/3MZjC6C
SAMSUNG 980 PRO SSD 2TB PCIe NVMe - amzn.to/41vUOrk
SAMSUNG 870 EVO SATA - amzn.to/3KQMriU
G.Skill RipJaws DDR4 SO-DIMM Series 64GB - amzn.to/3AlstI8
Mk1 Manufacturing - www.mk1manufacturing.com/cart...
See the kit here:
kit.co/TechnoTim/efficient-lo...
Want to set these up with Proxmox? No problem!
• Build a Low Power, Eff...
Looking to se up a cluster yourself? Check out the video and the docs!
• The FASTEST Way to run...
Thanks (Red Shirt) Jeff Geerling for the clip!
/ @jeffgeerling
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a CZcams member: / @technotim
Merch Shop: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
2nd channel: / @technotimtalks
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - How to Build a Low Power Cluster
00:18 - Hardware Specs
01:20 - Rack Mounting Intel NUCs
02:17 - Remote Control & Power Control
02:58 - Provisioning the Cluster
03:28 - Why Kubernetes?
03:39 - Automated Install of Kubernetes
04:20 - Kubernetes Policy Management (Today's Sponsor, Datree)
05:46 - Installing a High Availability Kubernetes Cluster
06:51 - Testing High Availability with NGINX
08:57 - Introducing Chaos
10:04 - What Can We Do with a Cluster?
10:35 - How Much Power Does This Cluster Used?
11:20 - What Do I think of a Low Power but Powerful Cluster?
Thank you for watching! - Věda a technologie
I like the clean rack mount most of all. I just finished a project with two NUCs and agree on their handy utility! (And I'm glad Red Shirt Jeff has been pestering you lately... finally I can get some work done!)
Nice cameo
Thanks not Red Shirt Jeff!
Hi Jeff,I got you here!!
Yup the OG Jeff is here ❤
wow see Jeff here 🎉
I'm so glad to see you getting into low(er) power boxes. Excited to watch this!
Thanks for the video. I run a DN2820FYK Celeron now for more than 9 years. It was my entrypoint to Linux on a x64 platform. Evaluated numerous stuff on it. Learned about Docker and so on. NUCs are super reliable from my point of view and worth every penny over time.
The Mk1 hardware looks pretty nice. Glad to see you having lots of fun in the lab!
Crazy you did this! Exactly what i did about 8 months ago. I had a dell R720, readynas, 2 switchs (with POE) AND BOY WAS MY POWER METER SPINNING. Got to the point where I didnt see the need to have such a power hungry server in the house and moved it to a data center where the space and bandwith was far cheaper than my costs to have it at home. But having this 3 node cluster was perfect for my needs and i could still do some low end development on the home lab if need be. Great video man looking forward to the next one!
Literally how Chick-Fil-A stores run their edge clusters in-store. K8s on NUCs.
Found this channel researching NUCs...absolutely awesome content. Liked and Subbed! 🎉
My cluster is 3 NUCs for workers and 3 hp mini PCs for the control plane. Been using this setup for a while. Very stable
👍👍- phenomenal! Thanks Tim!
I'm super glad I found your channel. This is the pure kind of nerdery I need in my life. Love it.
Nice work! I love those little 4"x4" boxes and have a few in my lab
This is really good! I was just thinking of deploying a full functioning project locally because I don't have money to maintain cloud services fees while still in development! I think you just saved me my friend.
Great video 👍 This makes me wanting to setup a kubernetes cluster myself 😊
Great video, being learning a lot from you, thanks for your work. That storinator is using more power than my entire homelab :D
such a professional video review,envy of your hardware!
That’s pretty awesome. As you can see from my profile picture. I use the little Lenovo Thinkcentre and I use them as proxmox cluster. The SFF desktop below them is my OPNSense box. As you mentioned, they’re not as power sipping as the pi but they’re still very low powered compared to full size desktop or rack mounted servers. I use docker for my containers just because I find it simpler but I’ve been meaning to start using Kubernetes. Again really cool video and it’s cool to see these micro sized computers serve a purpose and in my case given a second life. Keep up the good work and can’t wait for your next video. Cheers!
I love your videos even when it is not my game sometimes:) ❤ Keep on being awesome !
Love all the low power pc content!
Tim! Great video. I’m going to have to explore this. I’ve been thinking of selling my 8 x Raspi 4B (8gb) cluster and doing something like this. By the way, for your last Cloud Init video, I put together a ansible playbook based on the video to automate the process. It was awesome, used it to make templates with a single click across five proxmox nodes. Thank you for all you’re doing! 👏
With 8GB Pis going for 220€, you can build more than a 3 node NUC cluster, lol
Love this video, it's exactly into my hobby sphere.
A NUC is great because of a small footprint. If you have a rack (and it's somewhere where a bit more noise doesn't matter), you'll get a lot more back for your buck with an actual server or 3. Typically, the dell/lenovo/hp machines are quite affordable, especially if you buy used/refurbished. I'm running 2 lenovo's with a 6-core I5, 64GB of RAM and 2 500GB SSD's each.
For a homelab, anything "tiny mini micro" is great, and suits a K8s cluster quite well, but for me the real strength of a homelab comes from building experience in setup + teardown.
From that perspective, getting 3 machines to form a Proxmox cluster would make a lot more sense. On this setup with virtualization, you could easily fit 4 or 5 clusters with a controlplane and 3 nodes per k8s cluster, and have a bit more room to play with.
In any case - for anyone wanting to build more hosting / devops experience, I highly recommend finding a small machine, cram as much RAM into it as you can, and start setting stuff up.
Right now I'm working on an "end to end terraform solution" where I setup 4 VM's on proxmox with cloud init on an ubuntu template, initiate k0s as a distro with 1 controlplane and 3 nodes, and then also automate all of the final details, like storage using nfs to my nas, metallb and ingress.
Challenges always remain, but it's fun to perfect (and then toss out and start over with something else).
Correct in all aspects, I just think he wanted to do something different and jump straight to the k8s side of things. Spot on though!
Cool to see I’m not the only one clustering mini PC’s. 👍 those small form factor devices are quite powerful lately.
I love the NUCs, they are awesome
Relating to the remote KVM part: there are NUCs with full vPro functionality with remote KVM over VNC. They also got a web interface to power cycle etc. - and there are also command line tools to start them remotely
The problem is to find them - cannot find them anywhere on the internet (just simply nuc ...)
Nice info, thanks for sharing :)
Love the NUC line. Definitely a real tb port.
The unexpected Geerling cameo had me laughing out loud!
I'm not sure if I should be proud or afraid that I start to understand what you're talking about and envy your setup after joining devops 2 years ago...
I am thinking to run 3 R86S but as a proxmox cluster
the 2 10gig interfaces are really interesting, you can use couple of ways
1x for client, 1x for ceph network
2x for ceph ring network
and on top of proxmox cluster you can of course run k8s nodes as well so it seems best of both worlds
Nice work !
At home I use 6 Lenovo Tiny as Proxmox Cluster to run a Docker Swarm, a K3S Cluster, ...
Great video Tim, thank you. I use K3sup to provision, but I'm very interested in your Ansible version.
Great video! I just had to comment about the cost though... holy crap that rack mount drawer is the same price as one of the NUC's! That's an insane cost for what is essentially a folded steel drawer.
Drag race? Your NUC k3s cluster vs mine 💪🏼
Let's go! 🚀
I would love to see a livestream contest, judged by Tom from lawrence system.
I either ran into you or your doppelganger just now in downtown Houston. Couldn't put my finger on who I was reminded of before the convo ended but wanted to say thanks after my memory was sufficiently jogged. Have appreciated your work for some time now. ✌️
Hey, and thank you! Not me I am way up north but would appreciate that you came here to tell me!
@@TechnoTim haha well your twinner is here and in shorts.
Tim's K3s Ansible playbook really simplifies setting up an HA Kubernetes cluster. On Ubuntu, it might be faster to set up microk8s but after trying both, I'm like k3s slightly more because of lower idle CPU usage.
And even with stateful apps (which is... uhm... pretty much everything?), you can still achieve almost-HA with a single replica in a HA cluster + replicated storage (e.g. OpenEBS Jiva or GlusterFS). Just need to set tolerations to something like 10 seconds. So if the node (that carries the pod) is down, 10 seconds later, a new pod will automatically spins up on another online node. That's HA enough for me.
Really looking forward to your next live on Thursday because i work Tuesday until 8 unless your going live later on in the day
Planning to do this in the future too, but I think I'm going to wait for mainstream ARM to come a little further to get the power usage down even further. Though maybe I should do a cost/benefits analysis, maybe switching to NUCs now and then again later saves the hardware costs in power costs.
Assuming 20w each for the nucs, and 5w each for some future arm thing, and $0.20/kWh. That works out to a difference of 45w. 0.045*24*365 = 394.2kwh per year, at a cost increase of about $79 vs the theoretical arm setup. That's about 1 raspberrypi in a case with a PSU per year. Some cstate tuning, and maybe scheduling a shutdown would likely close the gap. I'm not sure which cpufreq governor Ubuntu server ships with by default, but switching from performance to ondemand would also save power.
If Tim's text overlay of 9watts average is correct, it's closer to 140kwh savings, or about $28. These mobile CPUs in the usff form factors really are very energy efficient.
Great video, I have a question with the NUC's. What would you do if you want to add more storage as in external hard drives?
Great video. If I can suggest wherever your reading from to be lower. Looks like your reading from the ceiling or something
Those Thunderbolt ports will directly do 10gig+ Ethernet host to host (iPerf around 14). 3 hosts, 2 ports each so you can daisy chain. Takes some configuration, but it's a cheap way to get fast networking.
I liked the Jeff Geerling "Red Shirt Jeff" cameo.
I'm good with docker and have been using it for a few years but man, I gotta get off my butt and get into kubernetes.
This is a perfect setup I think. A bit expensive for me. I've bought NUC i5 8 Gen. It supports virtualization but not those fancy features. I think home server in NUC is an optimal solution.
Since those nucs have 2 Thunderbolt ports, would you please try and use them as network connections? If you connect them to each other you'll only need to modprobe the thunderbolt-net driver I think. (Be sure to use real Thunderbolt cables!) Maybe the speed will be really good! TB does 40Gbit/s after all, so if you get like 10G that would be amazing!
Did you look at charmed kubernetes options with JuJu? It even integrates quite nice with MAAS 😃
any plan for a followup on this? Interested in seeing how you go on with all the services you mentioned. How do you setup the storage and running traefik as ingress. Would be cool to see some homeservices for instance zigbee2mqtt with an dongle. You can add a tag to the one who has it connected, and let k3s move the container around.
Hello, I have a question about clusters, maybe will be useful in order to work in video edition with software like Davinci Resolve?Can you do an example of that in case it is possible? cause video edition in Davinci Fusion requieres a lot of machine and maybe if one person can use 2 or 3 cheap pcs instead of a monster pc maybe would be a cheap solution
Hi Tim, not sure if you've done a video on it, but do you manage your Unifi network with either Terraform or Ansible as part of your home lab setup?
Heya, thx for the monitor! Cool vids as well!
Thanks man! Enjoy! That monitor makes an appearance in many of my videos!
"Red shirt" Jeff is a bit too much chaos.
What addons do you have in VS Code to help with Kubernetes?
I'm not sure if the NUC's are compatible, but there's lots of Intel processors with AMT that allow out of band management on the same interface. I do this on my OptiPlex desktops using the MeshCommander desktop application. This might be something to look into if you'd like to save the PiKVM for something else!
Thank you! When I bought the first NUC, I thought they came with AMT, which they didn't and the reason why I went PiKVM!
@@TechnoTim
1st: Another excellent video, thanks Tim.
2nd: I'm not 100% sure what I'd do with it, but I want it
3rd: Some Intel NUC models have vPro (at a steep price increase)
What can you do with longhorn? Can you setup something like ceph over the 2.5G nics, or install 10G usb dongle nics for more speed?
My thing with nucs is the ram!
Tim. You need to stop making me have tech envy. I am trying to stop myself from buying a system76 meerkat or an Intel NUC for using at my desk.
9:00 😂🤣😂🤣😂🤣😂🤣😂🤣THE BEST and HILARIOUS part.....!!!
TIM be aware of RED_SHIRT_Jeff .....🤫🤫🤫
Tim, dont mess with Red Shirt Jeff... :D
3 x Lenovo Tiny m920q high availability, zfs replicated, proxmox cluster. They each have pice slot and I have Mellanox ConnectX-3 dual SFP+ installed in each. They each run 5.5 idle without the dual sfp+. With the dual SFP+ they idle around 11 watts. Mine each have 32gb ram, and currently 512GB NVME.. Will be upgrading the nvme soon enough. No ECC but I am okay with that.
Gostaria que tivesse mais conectores SATA para fazer um NAS storage.
Hey Tim so where is you master node, not in one of your NUC, right?
I have a question, would it be possible to create a VM that would otherwise use cores from several cluster devices?
Since NUCs have vPro technology, have you tried to use MeshCommander to access AMT/KVMoIP instead of using the pikvm?
Not all NUCs have VPRO / AMT. I found out the hardware, but it was a happy little accident because now I can use PiKVM to control everything!
This looks nice for a lab play with. For power saving, I find proxmox on PC better as I can run the same services on fewer machines.
I can also put 10Gbe interfaces, hba cards, gpu’s for ML , and more in them.
Hello, I've just made a corosync and pacemaker cluster between two debian 12 machines and I want to install CasaOS on the cluster so the load is balanced between the machines. Is it possible to install CasaOS on the cluster and if it is, how?
What cable is that supplying power & data to the SSD? It’s very interesting, I haven’t seen that before
Could you run proxmox on each node and make each node a separate k3s cluster using VMs?
How does kubernetes handle P and E cores in modern CPUs?
I've been keeping my homelab on 10th gen intel due to never getting a solid answer to this.
09:00 chaos 😂❤🎉
Hope @jeffgeerling saw this
I’m confused, what storage provider are you using?
Is there a reason not to use Intel ME (vPro) and buy expensive KVM stuff? The Intel NUCs and many more mini PCs have already Intel ME build in, including the capability to remotely mount FD and CD Images.
Just wondering if I can use different spec of computer for each node.
Vs RPi? You forgot to mention the architecture. It seems there are many containers that do not run on ARM processors.
Just a question everybody seems to be let out (or I don't have the right understanding of HA) what about if a pod/service runs with a database and a node goes down? How does Kubernet know who is holding the latest database entry? Thanks 👍
where did you get that large kvm?
I'm amused by your notion of wanting low power while I'm thinking about how sweet a R940 would be to have as a Truenas box. 🤪
Nice idea.
BTW, it's much better to use new intel 12th gen low-power cpu like N100.
A small pc just look like intel NUC with N100 inside, just cost about 100 US$, without mem and ssd.
That's exactly the path I'm going to take. I have 3 NUCs, a new one from Beelink and two really old ones from eBay which I'm looking to replace. The ones he linked are great but I can get an N100 from CyberGeek for half the price, more RAM and a TB SSD.
red shirt jeff cameo!!!! 😂
Not having ecc support is a deal breaker for me, the worst thing about it is that you may not know if there is a problem when memory corruption happened.
@TechnoTim how is this managing E / P cores with K3s/ K8s
RED SHIRT JEFF CAMEO
Got rid of my 7 boys Proxmox cluster. Now using only 3 AMD 4800U servers. Yes it's 15W each, yes it's 16CPUs each, yes it has 2.5Gb LAN. Only issue is that it has only 1 slot for NVME and no SSD disk. But it's a bomb and my proxmoxies say thank you daddy every time I see these 30 VMs dancing 🎉🎉🎉❤
😂
No ECC? How come?
should look into consul / nomad stack for something lighter weight than kubernetes
That is what I run at home on my NUCs, love it
Don't let red shirt Jeff near the server!
Does Intel NUCs support ECC Sodimms?
It does not
Moving off VMs like this: we shall refer to this as LITERALISATION.
I see The Chaos isn't properly credited! 🤭
Let's be real the tidy cables never last long for us lol
3:21 LastPass? With all your servers and clusters I’m really surprised you don’t self host Bitwarden/Vaultwarden or similar self hosted password manager.
Do they send worldwide?
Amazon should!
@@TechnoTim I mean the rack mount provider
Bahaha redshirt Jeff being a chaos monkey
Who let Red Shirt Jeff out?
This seemed to have rapidly age like milk as Intel announced no more nucs. *But* there are other great options still!
No it hasn’t. NUCS will be kicking around for years to come as the warranty will still be honored, much like what has happened to Intel Optane. Also, mini PCs are here to stay, so just swap in any mini pc and reasons remain the same.
Cool but kind of expensive
You can set and configure any devices. The main problem is the money to buy hardware 😢 and at the end there need to be some purpose to build those setups. Ideally it would serve business that would generate money.
The experience could also be the difference between being on helpdesk and going up to devops salary. My homelab experience was way more valuable in getting me from $70k to $125k than work experience alone.
@@stevemulcahy5014 oh yeah I learn a ton when I'm playing with my homelab too!
Are these devices POE powered?
Nope
No but I can see how you think they were, the power output in the UniFi screens was via their PDU and not a switch.
why not harvester?
I don't know if 3K USD worth of these machines are being "efficient" , surely not for the money. 3,000 USD you can get same power for years on Hetzner and not get stuck with soon to be dated hardware.
Why does every CZcamsr who shows off their cluster on CZcams never talk about what they actually practically use them for (other than generic answers that everyone knows/says about what you "could" use them for)?
Check out the rest of my channel, I have about 100 use cases
You mean you went with K3s because my linkedin comments were a compelling enough argument…
when you like ubuntu you will love the original (debian) ;)
Awesome video, but I don't mess with Kubernetes unless someone is paying me for all the bullshit you go through setting it up.