Welcome! Log In Create A New Profile

Advanced

How to back up a live rootfs using rsync

Posted by bodhi 
How to back up a live rootfs using rsync
November 18, 2020 04:34PM
I. INSTALLATION


A. Setup rsync to back up a live rootfs to a network backup server


A.1. Network Client side rsync

This is where the rootfs is, while the system is running.

1. Install rsync

apt-get install rsync

2. Download and copy the attached script rsync.backup.daily to root home folder

cp -a rsync.backup.daily /root/rsync.backup.daily
chmod +x /root/rsync.backup.daily

3. Create the rsync password file. And secure it.

echo "mybackuppasswd" > /root/rsync.backup.daily.pswd
chmod 600 /root/rsync.backup.daily.pswd

4. Set up a cron job to run at 11:25 AM dailly

Install cron if not already installed
apt-get install cron
Edit the crontab
crontab -e
And add the following line at the end (or replace/modify if this time slot is already in the crontab)
25 11 * * * /root/rsync.backup.daily
List the cron jobs to doublecheck
crontab -l

5. Setup logrotate

Install logrotate if not already installed
apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf

/var/log/rsync.backup.daily.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 4
    compress
}


A.2. Network Server side rsync

This is where the back up server is.

1. Install rsync

apt-get install rsync


2. Enable rsync daemon at boot

Edit /etc/default/rsync to set enable flag to true

RSYNC_ENABLE=true

3. Download and copy the attached /etc/rsyncd.conf

cp -a rsyncd.conf /etc/rsyncd.conf

4. Create the rsync credential file. And secure it

echo "mybackup:mybackuppasswd" >  /etc/rsyncd.scrt
chmod 600 /etc/rsyncd.scrt

5. Setup logrotate

Install logrotate if not already installed
apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf
/tmp/rsyncd.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 8
    compress
}

6. Start rsync daemon

/etc/init.d/rsync start


B. Setup rsync to back up a live rootfs to a local USB drive

This section B is independent from section A above. Meaning there is no need to the whole set up as described in section A.1 and A.2 in order to do backup to USB.

1. Install rsync if not already installed

apt-get install rsync

2. Download and copy the attached script rsync.backup.ondemand to root home folder

cp -a rsync.backup.ondemand /root/rsync.backup.ondemand
chmod +x /root/rsync.backup.ondemand

3. Running on-demand backup

- Format a USB drive with Ext3 (Ext4 is OK if uboot on the rootfs box can recognize it)
- Label it with something meaningful, e.g. BackupRootfs

3.1 Mount the USB backup drive to /media. This is required so that we can avoid mistake in being at the wrong place.

3.2 Start the back up from the USB drive

cd /media/BackupRootfs
/root/rsync.backup.ondemand &

From now on the rsync will do incremental backup to the USB when we run the back up commands above (steps 3.1 and 3.2).

Note:

TBD - this process will be customized further to make the script run automatically whenever the backup USB drive is inserted.


II. Glossary

/root/rsync.backup.daily
The main script. Scheduled by cron

/etc/rsyncd.conf
The rsync conf file for the backup server (used by rsync daemon)

/root/rsync.backup.daily.pswd
The password file that rsync uses to authenticate the client backup job with the backup server.

/etc/rsyncd.scrt
The credential file that rsync on the with the backup server side uses to authenticate the client backup request.

/root/rsync.backup.ondemand
The on-demand script. Back up the rootfs to a USB drive.


/var/log/rsync.backup.daily.log
The client side log file

/tmp/rsyncd.log
The backup server side log file

=====================

Updated 01 Jan 2023

I've updated the script rsync.backup.ondemand to make it a bit more robust. I'm making sure the USB can be used as the rootfs and a boot drive.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 19 time(s). Last edit at 01/01/2023 05:15PM by bodhi.
Attachments:
open | download - rsync.backup.daily (1.5 KB)
open | download - rsyncd.conf (373 bytes)
open | download - rsync.backup.ondemand (1.5 KB)
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 03:37AM
:thumbsup!
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 07:53AM
Backed up and compressed I assume.
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 04:53PM
LeggoMyEggo Wrote:
-------------------------------------------------------
> Backed up and compressed I assume.

rsync compress the files to save bandwith while sending. However, the backup area is an uncompressed folder ( a mirror of the source folder / ).

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 09:41PM
Looking forward to this. It's something I've wanted to look into for a while now to handle the remote units I take care of.
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 10:51PM
Feel free to ask questions while I am modifying the 1st post (you might have done research before about this subject).

The set up is a little bit tedious, especially the rsync syntax is tricky.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
November 27, 2020 06:22PM
This tutorial is almost done. I just need to explain a few things, i.e. what names to change, etc.

It can be used now by experience users who know how to customize the set up for their own configurtion.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
November 29, 2020 02:33PM
bodhi, good work, here was how my rsync.backup.on demand looked on my pogoplug pro v3 test device

/media/USB# /root/rsync.backup.ondemand
<13>Nov 29 15:09:33 root[12560]: Started rsync on-demand backup for ppdebian
<13>Nov 29 15:28:46 root[12683]: Finished rsync on-demand backup for ppdebian


is there a way to show progress while running or did I miss something?

Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6
Re: How to back up a live rootfs using rsync
November 29, 2020 03:31PM
echowarrior108,

> is there a way to show progress while running or
> did I miss something?

That's part of what I need to describe in Section II.

The log you can monitor is in the rsync command.
/var/log/rsync.backup.daily.log
/tmp/rsync.backup.daily.log

And in the backup server,

/tmp/rsyncd.log

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)



Edited 1 time(s). Last edit at 11/29/2020 03:33PM by bodhi.
Re: How to back up a live rootfs using rsync
November 29, 2020 05:00PM
see I missed it lol

Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6
Re: How to back up a live rootfs using rsync
November 29, 2020 06:53PM
echowarrior108 Wrote:
-------------------------------------------------------
> see I missed it lol


:) the first time you do back up it will be massive (logrotate will eventually compress it). From now own the log will be smaller.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
November 29, 2020 11:37PM
I did find a little shortcut to watching by using htop.I use a second terminal then simply run htop, what I do then is hit f4 for search and type in rsync and anything running rsync will show as it runs. What is nice is that it shows how much memory is being used at the same time.

It was massive on the first try for sure! it goes fast now.

currently I am running an ssd on the sata drive and rsynking to usb as described so I am sure it is a bit faster than most but I will be trying out on my mpcie drive device soon which will take a bit longer.

Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6
Re: How to back up a live rootfs using rsync
November 30, 2020 12:19AM
> I did find a little shortcut to watching by using
> htop.I use a second terminal then simply run htop,
> what I do then is hit f4 for search and type in
> rsync and anything running rsync will show as it
> runs. What is nice is that it shows how much
> memory is being used at the same time.

Yes, it is a handy way to see what's going on!

This rsync setup has saved my ass a few times :)) the key is don't do --detete for a daily backup. Once in a while, you would turn on --delete to get rid of old files in the backup server. When it happens that you accidentally delete some file because of fat fingers or just sleepy, or tipsy, you can just restore that file from the backup server immediately :)

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
November 30, 2020 03:16AM
I can't tell you how many times I used to start all over with programming because of this or that.... I used to dd copy every so often then remove the copied usb which is effective to a certain extent however this is a much better way for sure!

Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6
Re: How to back up a live rootfs using rsync
November 30, 2020 06:17PM
> I can't tell you how many times I used to start
> all over with programming because of this or
> that.... I used to dd copy every so often then
> remove the copied usb which is effective to a
> certain extent however this is a much better way
> for sure!

If you writing code or scripts, you should really be using version control such as git or svn. And rsync backup daily using this process.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
November 30, 2020 10:39PM
Quote
If you writing code or scripts, you should really be using version control such as git or svn. And rsync backup daily using this process.

only toying with a few scripts thus far, kinda new to that. actually been trying a little bit of anything to see what this device can do then if I like it i incorporate it to my main plug which right now is mostly a home server. I may try git or svn later when I can dedicate more time to it:)

Echowarrior108

device: pogoplug-pro v3

Currently running:
Linux version 5.15.158-oxnas-tld-2 (root@tldDebian) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT Sun May 12 18:45:05 PDT 2024
Debian 12.6
Re: How to back up a live rootfs using rsync
February 21, 2021 04:50PM
FYI,

I've done with the tutorial in the 1st post.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Re: How to back up a live rootfs using rsync
January 01, 2023 05:16PM
Updated 01 Jan 2023

I've updated the script rsync.backup.ondemand to make it a bit more robust. I'm making sure the USB can be used as the rootfs and a boot drive.

-bodhi
===========================
Forum Wiki
bodhi's corner (buy bodhi a beer)
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: