Migrate Docker Volumes from one Host to another // backup and restore
Vložit
- čas přidán 15. 07. 2024
- Migrate Docker Volumes, and how to access data inside a Docker Volume? I'll show you how you can easily back up and restore Docker Volumes and move the container data to another host. #Docker #Volumes #HomeLab
Backup in Linux Servers - Docker Volumes, and Databases: • Backup in Linux Server...
Docker Documentation: docs.docker.com/storage/volumes/
My Docker CLI Cheat-Sheet: github.com/christianlempa/che...
Follow me:
TWITTER: / christianlempa
INSTAGRAM: / christianlempa
DISCORD: / discord
GITHUB: github.com/christianlempa
PATREON: / christianlempa
MY EQUIPMENT: kit.co/christianlempa
Timestamps:
00:00 - Introduction
00:43 - How Docker stores data in containers
01:36 - Docker Volumes or Binds
02:05 - Some example Volumes and Binds
03:52 - Access files inside named Volumes
04:21 - Backup, Restore and Migrate data
________________
All links with "*" are affiliate links.
You haven’t compressed the tar, you need the z options.
Oh damn, you're right! I just updated the cheat-sheet on github ;)
I’m so used to seeing “z” there that it stood out to me, too. For beginners: Don’t forget to then also suffix “.gz” to the file name (to indicate that it’s also a compressed gzip file that then contains a tarball).
How about a video about using Portainer’s backup feature?
@@WokeSoros Does portioner have a backup feature?
Outstanding mate, from a Computer Scientist and a Professor, you have a great talent and a great team.
All my best.
Thank you so much!
offen/docker-volume-backup is an absolute blessing to easily automate docker volume backups.
Yeah that's a great suggestion! Maybe at some point it's worth another video ;)
Perfect Christian, exaclty what I was looking for! Will help me restore a clients data to their VMware VM containers. :)
One of the best videos ever! Very useful!
Thank you so much 😃
You can't beat the amount of useful content that is packed into each episode.
Thanks mate! :)
Thank you for doing this tutorial. The more I learn about docker the more excited I get about what it can do.
I was able to get your backup command to work perfectly and to scp that file over to the new host. However, I can not get the restore to work at all. In this video you stopped short of demonstrating the restore, I assume because it was obvious to most. However I'm falling short of knowing what's going wrong. I keep getting PATH errors related to "bash" executable, and I'm not even sure how to track that down since I don't understand what it's trying to tell me.
I would be most grateful if you covered restoring using your one-line command in a future video.
Thank you for the content that you put out. It's most appreciated.
Yeah Same for me !
It feels like clickbait
Thank you very much for all your videos, they are awesome
Thank you so much! :)
Bester Mann 😀
Danke für all deine hilfreichen und interessanten Videos!
Hehe gerne! ;)
Content that you created is much clear and useful ) . Keep making more content. We will be wait content about `Kubernetes` for upcoming sessions.
Thank you so much! And yeah more Kubernetes content is coming in the future 😁
Exactly what I needed.
Thanks!
Clear and understandable. U r genius
Thanks 😊
oh yes.. Another awesome one.
Thanks again!
thank you so much
how i could make backup and restore when in 1 container i have many volumes. for example container with zabbix which consist from mysql\ssl\snmptraps\keys volumes???
thanx , youe ease my job
@6:20 I just want to add that the PWD variable is an acronym that stands for Present Working Directory. I hope that helps someone remember the variable function, it helped me.
Almost 😊 It actually stands for print working directory
Thx bro for the great explanation, But i have a point in restore section which I've got while doing it on Mysql docker volume,
unfortunately the restore command here won't work when use it on mysql container so I had to create a new volume and extract the backup file into it before running the new container.
Hi Christian, thanks so much for this video, super helpful. I am however not seeing the devops folder in order to test the backup and restore commands you use. Could you please advise?
(assuming everything is binded) Alternatively, can I just copy the directory where I made folders for my containers in a tarball and export my container config as stacks, then in my fresh install - copy over your existing directories to where they once were and re-deploy the containers with stacks (.yaml files) that I exported?
How can we run that migrate command for the container which is having more than one container path?
or you could just compress the binded docker volume directory with tar and scp it to the binded location then run the docker stack with the binded volumes, i did that, after getting stuck. It worked beautifully, next create a cron script that backup the data to a backup server. Quick question if there are different addition of data into the container from different servers how do i combine the new changes into one server?
This is really good, thank you :).
Is there a way to migrate or clone an app running with docker-compose, which has both named and bind volumes?
Thank you! :) You can use the same technique to backup binds as well. It doesn't make a huge difference if the container was deployed via dockercli, portainer, or docker-compose.
I'm currently hosting a nextcloud container on a Raspberrypi and want to move it to a desktop computer that will function as my new server (both running Ubuntu). Is this the prefer method to move my data from one machine to another? Would you suggest just backing up the data and create a new container 'From the scratch'? I don't have any 'special' apps on Next cloud. I am using it exclusively as cloud storage
Good day Mr. Lempa, I hope you are doing well! I have a question for you. Suppose I had a Docker Swarm that, at the moment, consists of 4 raspberry pi 4s each with a 500gb m.2 SSD drive plugged into them. Suppose further that I am also running Portainer to manage my containers. Now suppose that what I want to achieve is data persistence across these 4 Nodes so that if, say, the node containing the data for my Gitea instance goes down and maybe even stays down, another node should start up the service and also still have the necessary data. Any idea on how I could achieve this?
You see what I mean?
Can't see cheat sheet of the video at the description link.
HI. I am getting the error "docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bash": executable file not found in $PATH: unknown." when I try to restore. Please point me in the right direction
Haven't had this issue before unfortunately :/ Maybe you can ask on our discord if someone has seen this before?
Has the same problem, found the solution;
Instead of
docker run --rm --volumes-from CONTAINER -v $(pwd):/backup busybox bash -c "cd CONTAINERPATH && tar xvf /backup/backup.tar --strip 1"
use
docker run --rm --volumes-from CONTAINER -v $(pwd):/backup busybox /bin/sh -c "cd CONTAINERPATH && tar xvf /backup/backup.tar --strip 1"
@@aokijikuzan6453 thanks I will try that
@@aokijikuzan6453 nice it worked for me, THANKS !!!
what if container has more CONTAINERPATH?
You are my favorite bald online friend
Haha, thanks mate :D
My Raspberry Pi Lite OS runs on a 64GB Micro SD card boot drive and I have a secondary 2TB SSD to store data in. I plan to install Jellyfin and PhotoPrism and NextCloud which requires installing Maria MySQL Data Base. What has me confused is how I specify all docker data storage for PhotoPrism, JellyFin and Nextcloud to take place on the 2TB SSD drive and not the 64GB Micro SD card boot drive? Do you have a video that shows correct docker Volume syntax that enables data to get stored onto a 2TB SSD secondary data drive?
Perfect Video. Thank You. I hope is corect wrided. My english is Not god. But i understand you.
Thank you! That's great to hear :)
One thing I don’t understand about Volumes is they seem built for hello-world sized projects on a single drive systems. Volumes seem neat, but a restricted alias of Binds which doesn’t allow you to specify which drive you want data stored. Most my docker use cases require data storage that exceeds my OS drive size, but that’s where volumes stores data because it’s built into docker, not on my Data drive. Is there something I’m missing about Volumes that would make them useful for larger storage needs?
Step:1 Stop docker services
sudo systemctl stop docker
sudo systemctl stop docker.socket
sudo systemctl stop containerd
Step:2 make new root directory structure
sudo mkdir -p /new_dir_structure
Step:3
sudo mv /var/lib/docker /new_dir_structure
Step: 4 Create daemon.json abd add the following
sudo vim /etc/docker/daemon.json
{
"data-root": "/new_dir_structure/docker"
}
Step:5 Restart docker (Give 777 permission / change root ownership of the folder if needs)
sudo systemctl start docker
Step:6 check/verify the changed location by
docker info -f '{{ .DockerRootDir}}'
I'm new to docker, but could you not mount the volume inside a path instead of a named path and just use normal methods to back it up?
Edit: Found your content the other day and want to give you props for your information and delivery.
Thanks mate, yeah you can simply work with just binds, but I thought it might be useful for people to know how to migrate in case they use named volumes.
Is there any way I can do this on Heroku? It accepts commands by Shell. I'm using a bot, but the docker image was archived by the admin. Can you help me? I don't understand any of this. I just wanted to backup this docker and upload on dockerhub to help others who don't have a bot working before adm archived everything. Could you help me, please?
How reliable is the backup if the container is running and accessing it while you make the tar? Hmmm....
Hi, when I try to generate the tar file, I get this error
can't open '/backup/backup.tar': No such file or directory
could you please help.
Same problem. Didi you find a solution? Edit: Never mind, I was using the wrong docker container in the command.
can you make tutorial on how to run multiple services in same docker container ?
Not sure what you mean by that.
Hi, great video thnx
I have a nginx container on a pi that I want to move to docker on proxmox, but this container has multiple paths. do I need to run the backup command multiple times?
Host/volume Path in container
/home/pi/nginx/data /data
/home/pi/nginx/letsencrypt /etc/letsencrypt
/home/pi/nginx/config.json /app/config/production.json
If you can back it up on the host with one command thats fine, guess it isn't a DB.
@@christianlempa in the CMD you explain, only one path, I have three different paths. And the db is a separate container
Hi @Chris, just wanna let you know, u had exposed you admin token @3:40
Thanks for the superB video!
Thanks! That's on my old server, but thanks for the hint!
Looks like you need to update your cheat sheet repo's readme cause a lot of the links are broken.
Yeah you're right :/ Still on my todo list
Hi there - I'm struggling to get my portainer instance to run via the web GUI- I have no idea how its become the way it has, i've not changed anything's actively and can only assume an update has been done passively. I've had a look around and I can't see any ports applied to the container - this maybe a red-herring but wanted to address this first just in case.
"docker ps -a | grep portainer
33f01ee62da7 portainer/portainer-ee:latest "/portainer" 3 months ago Restarting (2) 36 seconds ago portainer"
This is what im seeing... Please help if you can.
Hi, wenn ich das auf meinem raspberry pi ähnlich mache wie du beschreibst (aber ohne busybox) und am Ende auf dem neuen Raspberry Pi "sudo docker load -i image.tar " eingebe, erscheint immer: no such file or directory
an was kann das liegen?
Ich starte meinen alten pi von SD karte. an USB hängt die neue M.2 SSD, auf welcher der neue pi schon funktionsfähig installiert wurde und welche einfach als usb-stick gemountet ist. dort kopiere ich die tar-datei hin.
Dann starte ich von dieser SSD und will die tar mit dem genannten befehl wieder verfügbar machen..... :(
Just 30 mins ago i was thinking about backupping my docker databases
Haha, perfect timing :)
Hi there, I tried restoring compressed backup with your cheat sheet, but it doesn't work with busybox image. Then I tried using ubuntu image and it works.
Can you open an issue on the github repo? That would be nice, I'll take a look then.
no encuentro el Cheat-Sheets del que hablas
What about if you have a couple containers being ran by docker-compose?
Should work the same way, with docker-compose.
@@christianlempa What if there is no mount point when I inspect the container?
I can't find your devops cheat-sheets. Its just me?
Dust against the mailer in in time setting
It’s very simple,police finding me now
Captain busy box doesn't elaborate at all what that should be if we don't use it super special busy box personal preference configuration he uses.
Why not just use something like Duplicati for this?
The main focus is on migration, so it's important to know how to do it manually. But Duplicati is also a topic that I've covered in the past, therefore I've put it in the description and as a "watch next" suggestion ;)
@@christianlempa I absolutely found value in the video. It's crucial to understand how these things are done manually for any kind of real understanding of your system. Thank you for the reply.
Github Link doesnt work, the page you showcase is nowhere to be found !!
Same
You run an image not a container. You start a container. And --rm remove the container file system after using it. It doesnt remove other containers
True, thanks for sharing
Please do not use scp anymore. It is deprecated. Rsync is a more robust, faster and more reliable solution for copying files between servers.
Thanks for the heads-up! I'm going to have a look :)
Sir, half of you video got deleted in editing.
you really should have taken the 3 minutes to also explain how to restore an image with the backed-up file. Becuase the command in your cheat sheat is really confusing, and its unclear if i have to create the image first, or what container names I should use in the command....
Good point, I even looked at my cheat-sheets yesterday and had a hard time understanding the command again :D
I'll give a quick update with more explanation on the docs, soon. Thanks for the headsup!
@@christianlempa I created a pull request for this today (wth everything but the command to be added because I, too, don't know)
For Windows users without WSL2 replace $(pwd) with %cd% in the backup a container command:
docker run --rm --volumes-from CONTAINER -v %cd%:/backup busybox tar cvfz /backup/backup.tar CONTAINERPATH