Saving Power in my Homelab with Auto-Shutdown for Proxmox Backup Server
Vložit
- čas přidán 5. 04. 2024
- Today I'm trying to reduce the power consumption of my [Proxmox Backup Server](/posts/2023/pbs_intro/). The HP Microserver is great for what I need, but it's kinda loud and I'm working on optimizing my power bill. The homelab is the largest single consumer of electricity aside from the air conditioning in the summer, so it's something I'm looking at heavily.
I tried doing S3 speep (normal suspend to RAM) and found that the Microserver doesn't support this. I then didn't have enough space on my 16G boot disk (microSD card) for a swap partition to hibernate to, and don't want to mess with the partition tables on my zfs drives to add one there, so hibernate is out too. This leaves me with actually fully shutting the system down and booting back up in the future. Downside to this is the RTC won't wake it back up, I need to have another system send a Wake on LAN magic packet. No big deal, I just need to make sure I send it early enough that the backups aren't scheduled to start until after the server is fully booted.
Blog post with all of the scripts:
www.apalrd.net/posts/2024/pbs...
Support me on Ko-Fi if you enjoy my content and find it useful:
ko-fi.com/apalrd
Feel free to chat about my upcoming projects on Discord!
/ discord - Věda a technologie
To enable WOL for low cost offbaord-NICs (RTL8125, etc) it might be needed to run something like "ethtool -s enp2s0 wol g" once per boot for the card to keep the port up after shutdown.
Thanks for the video!
uhh, fetching the running task list is really cool!
what I do for some of my systems is have them plugged into a smart outlet which completely cuts power once there's less than 5W of power draw (system is off)
then, to boot it back up, I've got power loss recovery set to "always on" in the bios and just turn on the outlet
I don't have ipmi on anything so this is a great way to squeeze out the last couple of watts when the system is off
I learn so much from your videos. Thank you!
Great video and thanks for sharing it! It just arrived on time since I did something similar finished today. In my case I end up configuring a nodered flow to perform backups sequentially and controlled from my 3 servers (wanted to avoid doing backup of a server while asynchronously start garbage collection or a remote sync which impacts a lot the performance just by guessing this will take 30 min and other task 40 etc...). To avoid that, I run one backup at a time and check the task created for that backup. Once finished jump to the next... once all finished sync job (which is very unpredictable in terms of when will be finish) and finally prune + garbage. This approach helps to keep everything under control and one thing at a time for performance reason. I was worry to play with expected timings that sometimes screw up everything :S
Another thing that isn't considered is using a smart plug with the "power on with power connected" feature that is present in most BIOSes. Tuya-based plugs have a feature where if power consumption is low or none for 40 minutes, it turns off the plug automatically. Of course it can also be controlled via home assistant as well.
Also it is also possible to schedule a shutdown every hour that only runs when the 15 minute load average drops below a certain value (aka the system is idle). That works well too.
that's perfect! I really needed this but never got to it. I will definitely try something similar, based off your documentation. Thanks for the video and the write up!
This is a useful topic and good ideas! Aligning it to your off peak power is great.
It's still possible it will run all the way into the next on-peak time, but usually just the infrequent media backup does that
Yes, this is the thing I was looking for! Thanks for the vid!
Thank you for all the education you're sharing 🌼
Can I ask you how to automate some simple tasks to be done across the different systems? like updating a configuration file on all of them. They say Ansible is one solution, but I'm not sure; it feels there should be something friendlier in a way...
Thanks a lot for your videos :-)
I run Proxmox on refurbished Dell Optiplex (Intel i3, i5) and Acer/Asus Chromebox (Intel i7) SFF desktops. With a little bit of kernel command line magic you can dynamically drop the CPU to 400MHz when idle, this saves quite a bit of power.
If booting on ZFS root:
edit the /etc/kernel/cmdline file and append
cpufreq.default_governor=powersave
to the end of the kernel boot command line. Then run
proxmox-boot-tool refresh
to updated the bootloader and reboot Proxmox to take effect.
After rebooting, you can run the below to verify the realtime CPU frequency:
watch -n .5 grep MHz /proc/cpuinfo
I have my NAS running on a Haswell crippled to use only one core and "powertop --auto-tune".
That is plenty of power to saturate my 1Gig LAN.
With just the boot drive(2.5 inch external), I got it down to use 10W of power on idle.
Now that server holds my PBS lxc as well.
Cheers for the video, the task list idea might make its way into my solution.
Currently i use home assistant to power on a wifi plug, once the power goes on the server automatically boots up (i forget the bios option for this but basically if theres power, its on, all be it after a 2 mins sleep). Shutting down its a bit janky it works out if its shut down and then turns then plug off at the mains.
WoL packets should be considered unreliable. A slight modification for more robustness is a program that continually sends WoL packets every few seconds until the target system responds to pings (or otherwise is determined to be online). If you combine it with systemd's dependency sequencing you could guarantee that your backups don't try to start until it's ready.
Its quite similar to what I got. only shutdown is just a hard shutdown no checking if any tasks are still running. time to change that! Thanks man
Came here after recommendation from "Guy with Swiss Accent " and subscribed . Your video contents are great and hopefully you will surpass Andreas contents ..
My secondary machine on which I run second PBS takes only 8W while idle (MinisForum Venus UM773) that as per UK prices gives me in total cost of 20 GBP/year - probably no need to turn in off and on and bring more exposure to failure due to power on/off. But the idea is great and makes sense if your PBS consumes too much power.
Neat!
Thanks for this, love your videos. Could you do an instructional video on how to spin down disks on proxmox (without using a VM)? I have found a few threads here and there about using hdparm, and also about pvestatd needs configuring to prevent waking the disks etc. It's all a little complicated and could do with a clear instructional video, as my nas disks basically don't get used for 10-12 hours per day (overnight) so would be good to spin them down and saving energy.
Also something about making a single zfs pool to be accessible by multiple lxc would be useful. Although there are other videos out there on this, I think you could do a better job at making this more understandable. Thanks!
zfs will end up writing transaction groups regularly, so spinning down zfs disks doesn't end up working. You can unmount the filesystem first, and then it won't auto-spin-up until the time comes where a script remounts it, but that means the data is inaccessible during that time.
As to zfs data sharing, I touched on it a little bit in my Jellyfin video. I could go back to it.
I am using WOL from HomeAssistant to wake up the PBS but for shutdown your script is nice and I will use also . Thanks
Glad you like it!
This is solid. I'm curious to hear your thoughts about running a server off a thumb drive / sdcard like you mentioned.
For the SD card, I have the SD card formatted ext4, noatime, no swap, and I put all of the PBS logs on the zfs pool (not the sd card), so it's not getting written to that much. Still not perfect, but all of the important data is on the zfs pool anyway.
@@apalrdsadventures awesome thx.
I used to turn off my NAS and Proxmox at night, like 11:30 pm - I'm usually in bed by 11 anyway. And my Ras Pi had a backup Pihole and a Wake on Lan script and it worked great.
But the problem is you lose your ARC cache on Truenas. SO i just host proxmox always on stuff on a Micro PC that uses like 10 watts at idle most of the time.
Losing ARC shouldn't be a big deal, it can read it back in as needed in the future
@@apalrdsadventures true since it’s an SSD nas anyway it’s not like it’s that slow.
It sort of sounds to me like the better option here would be to use one of those fancy mains power backup units (jackery?) or whatever flavour of the week, then put a relay on it to only charge the battery from mains when its on off-peak power hours :)
In my case I have truenas scale on an older desktop in the basement and I just looked into the cpu speedstep stuff to make sure it runs the cpu as slow as possible when there is no load.
Took a bunch of googling to find the commands and I already forgot how i did it now :(
For backup low capacity charger it's probably great for saving money? not really it would takes years to recoup seeing his computation of 96usd per year if the server was off 24/7 instead of on 24/7.
Shutting it down removes the load entirely instead of shifting it to a cheaper price with a battery.
Not sure it would be worth the effort. My UPS stays in a pretty much fully charged state so the large share of the power draw is by whatever is plugged into it and running. And as such the battery charger is probably doing minimal work. The only time that should ever change is after a power outage where the batteries were drained down whatever amount. And in which case the battery charger would start charging the instant mains power was restored. In that instance I would want the batteries topped off as quickly as possible, no matter what time of day it was, in case the power were to fault again.
I found a slightly different solution. My proxmox backup server is always on - it's a VM on the main proxmox server (consumes 100W) itself, which mounts an nfs directory on truenas scale, running on the proxmox server itself. Clearly, this is not a good enough solution.
Then I have a secondary proxmox server (consumes 40W) running truenas in a VM. I have a script that does a zfs incremental backup between the two truenas scale deployments, with wake/suspend around it.
The proxmox backup to the backup server takes about 4-5 hours to run. The incremental zfs backup varies, but it typically a few minutes. So that is how long each day my secondary server is awake.
worked flawless for me! Question: why backup independently the various servers you mention? Mine are in a cluster and with one job at the cluster level all go through. Are you saying the cluster will do them one by one anyway?
I have two separate servers (not clustered), one is 'production' and one is 'dev'. The webserver is also off-site hosted so it's a third separate system.
A cluster with a backup job will kick off all of the backups at once, but each individual VM/CT backup and also each VM/CT 'prune' is a separate 'job' to the backup server, so you will see a lot of jobs in the job history when backing up a cluster.
Thanks for the quick answer. I think I understand now ;) I would much appreciate a video on backing up random zfs datasets that are not necessarily for any given VM/CT if that is possible towards the PBS. Thanks again!
I actually have a blog post on the topic, but I'm waiting for a few months to make a video since there are some changes to pxar / pbs client that will greatly improve speed and minimize reading every single file when doing backups without qemu snapshots.
www.apalrd.net/posts/2023/ultimate_migrate/#large-dataset-backups-to-proxmox-backup-server
Thanks again. I'll check it out right away.
Great video.
But if the backup server is turned off, the logs on the VEs will go crazy not finding the shared backup location.
I've tried it and it gives a ton of these:
pvestatd[1370]: BackupServer: error fetching datastores - 500 Can't connect to x.x.x.x:8007 (No route to host)
Any solution for that?
Are you worried about the increased wear on your backup drives booting up and shutting down once per day?
That increased ware is offset by not spinning all day.
There's probably more wear from startup/shutdown than on-time, since HDDs wear from both cases. I'm also not super concerned with these Seagate Exos SAS drives. I'd probably be more concerned with desktop drives.
Don't most motherboard support wake from RTC in the bios ?
This one has very little in the BIOS, it doesn't support a lot of things (like suspend, but also the iGPU is completely absent).
It's funny that Dell poweredge servers for example don't have wake from RTC. I have mine set up to wake up over idrac, over a VPN, for off-site backups
iLO is another option for me, and yeah the server BIOS are the worst.
@@apalrdsadventures Hum didn't know, thanks! Weird that only consumer MB have the option, would a thought it's more of an enterprise feature.
Usually the option isn't available in the BIOS UI but the OS can configure it. That's how systemd timers could wake up the system.
Is there an opportunity to use anacron, so you don't have to worry excessively of the server has made it back online in time for a particular time trigger?
Systemd timers will run later if an event was missed.
In this case, the timers are on other systems (backups are 'push'), so their timers will trigger and the job will fail with errors if the backup system isn't up.
@@apalrdsadventures ah, I guess that's one advantage of completely shutting the dust down, vs hibernating.
i say leave it on all the time - goto a mobile tiger lake platform? good point about enterprise gear people get these big systems and then they get the power surprise but if you use a system basically all the time just leave it on - if you serve via r proxy you sort of have to; also this vid points to trend of consolidation - instead of 5-6 boxes try to run a couple boxes both with ups and running at optimal efficiency - save enough in a few years to make a difference #shelly relays
лайка, откуда такая майка?
лайк за лайку :)
Russian propaganda t-shirt during war? Nice! Maybe next day you can wear one with Hitler, Stalin or a swastika?
по свастике это к украинцам
@@user-rw6qd7fz4m ты что себе, холоп, позволяешь?! Кто тебе разрешил на западные сайты заходить?! В наказание тебе 12 часов молитв фуражке Гитлера в Храме Вооружённых сил на коленях (как тебя учили родители) и после этого поедешь на 6 месяцев на урановые рудники отрабатывать неповиновение Царскому указу.