Welcome! Log In Create A New Profile

Advanced

How to back up a live rootfs using rsync

Posted by bodhi 
How to back up a live rootfs using rsync
November 18, 2020 04:34PM
I am almost done on this tutorial.

===================================================================================

I. INSTALLATION


A. Setup rsync to back up a live rootfs to a network backup server


A.1. Network Client side rsync

This is where the rootfs is, while the system is running.

1. Install rsync

apt-get install rsync

2. Download and copy the attached script rsync.backup.daily to root home folder

cp -a rsync.backup.daily /root/rsync.backup.daily
chmod +x /root/rsync.backup.daily

3. Create the rsync password file. And secure it.

echo "mybackuppasswd" > /root/rsync.backup.daily.pswd
chmod 600 /root/rsync.backup.daily.pswd

4. Set up a cron job to run at 11:25 AM dailly

Install cron if not already installed
apt-get install cron
Edit the crontab
crontab -e
And add the following line at the end (or replace/modify if this time slot is already in the crontab)
25 11 * * * /root/rsync.backup.daily
List the cron jobs to doublecheck
crontab -l

5. Setup logrotate

Install logrotate if not already installed
apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf

/var/log/rsync.backup.daily.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 4
    compress
}


A.2. Network Server side rsync

This is where the back up server is.

1. Install rsync

apt-get install rsync


2. Enable rsync daemon at boot

Edit /etc/default/rsync to set enable flag to true

RSYNC_ENABLE=true

3. Download and copy the attached /etc/rsyncd.conf

cp -a rsyncd.conf /etc/rsyncd.conf

4. Create the rsync credential file. And secure it

echo "mybackup:mybackuppasswd" >  /etc/rsyncd.scrt
chmod 600 /etc/rsyncd.scrt

5. Setup logrotate

Install logrotate if not already installed
apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf
/tmp/rsyncd.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 8
    compress
}

6. Start rsync daemon

/etc/init.d/rsync start


B. Setup rsync to back up a live rootfs to a local USB drive

This section B is independent from section A above. Meaning there is no need to the whole set up as described in section A.1 and A.2 in order to do backup to USB.

1. Install rsync if not already installed

apt-get install rsync

2. Download and copy the attached script rsync.backup.ondemand to root home folder

cp -a rsync.backup.ondemand /root/rsync.backup.ondemand
chmod +x /root/rsync.backup.ondemand


II. Customization and Operation

TBD.

/root/rsync.backup.daily
/etc/rsyncd.conf
/root/rsync.backup.daily.pswd
/etc/rsyncd.scrt
/root/rsync.backup.ondemand
/var/log/rsync.backup.daily.log
/tmp/rsync.backup.daily.log
/tmp/rsyncd.log

-bodhi
===========================
Forum Wiki
bodhi's corner



Edited 14 time(s). Last edit at 11/29/2020 03:32PM by bodhi.
Attachments:
open | download - rsync.backup.ondemand (1.5 KB)
open | download - rsync.backup.daily (1.5 KB)
open | download - rsyncd.conf (373 bytes)
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 03:37AM
:thumbsup!
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 07:53AM
Backed up and compressed I assume.
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 04:53PM
LeggoMyEggo Wrote:
-------------------------------------------------------
> Backed up and compressed I assume.

rsync compress the files to save bandwith while sending. However, the backup area is an uncompressed folder ( a mirror of the source folder / ).

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 09:41PM
Looking forward to this. It's something I've wanted to look into for a while now to handle the remote units I take care of.
Re: rsync - How to setup rsync job to back up a live rootfs
November 19, 2020 10:51PM
Feel free to ask questions while I am modifying the 1st post (you might have done research before about this subject).

The set up is a little bit tedious, especially the rsync syntax is tricky.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: How to back up a live rootfs using rsync
November 27, 2020 06:22PM
This tutorial is almost done. I just need to explain a few things, i.e. what names to change, etc.

It can be used now by experience users who know how to customize the set up for their own configurtion.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: How to back up a live rootfs using rsync
November 29, 2020 02:33PM
bodhi, good work, here was how my rsync.backup.on demand looked on my pogoplug pro v3 test device

/media/USB# /root/rsync.backup.ondemand
<13>Nov 29 15:09:33 root[12560]: Started rsync on-demand backup for ppdebian
<13>Nov 29 15:28:46 root[12683]: Finished rsync on-demand backup for ppdebian


is there a way to show progress while running or did I miss something?

learning is living! Echowarrior108



device pogoplug-pro v3

Debian GNU/Linux bullseye/sid \n \l
Linux version 4.14.180-oxnas-tld-1 (root@tldDebian) (gcc version 8.3.0 (Debian 8.3.0-6) ) #1 SMP PREEMPT Fri Nov 15 04:12:45 PST 2019
Re: How to back up a live rootfs using rsync
November 29, 2020 03:31PM
echowarrior108,

> is there a way to show progress while running or
> did I miss something?

That's part of what I need to describe in Section II.

The log you can monitor is in the rsync command.
/var/log/rsync.backup.daily.log
/tmp/rsync.backup.daily.log

And in the backup server,

/tmp/rsyncd.log

-bodhi
===========================
Forum Wiki
bodhi's corner



Edited 1 time(s). Last edit at 11/29/2020 03:33PM by bodhi.
Re: How to back up a live rootfs using rsync
November 29, 2020 05:00PM
see I missed it lol

learning is living! Echowarrior108



device pogoplug-pro v3

Debian GNU/Linux bullseye/sid \n \l
Linux version 4.14.180-oxnas-tld-1 (root@tldDebian) (gcc version 8.3.0 (Debian 8.3.0-6) ) #1 SMP PREEMPT Fri Nov 15 04:12:45 PST 2019
Re: How to back up a live rootfs using rsync
November 29, 2020 06:53PM
echowarrior108 Wrote:
-------------------------------------------------------
> see I missed it lol


:) the first time you do back up it will be massive (logrotate will eventually compress it). From now own the log will be smaller.

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: How to back up a live rootfs using rsync
November 29, 2020 11:37PM
I did find a little shortcut to watching by using htop.I use a second terminal then simply run htop, what I do then is hit f4 for search and type in rsync and anything running rsync will show as it runs. What is nice is that it shows how much memory is being used at the same time.

It was massive on the first try for sure! it goes fast now.

currently I am running an ssd on the sata drive and rsynking to usb as described so I am sure it is a bit faster than most but I will be trying out on my mpcie drive device soon which will take a bit longer.

learning is living! Echowarrior108



device pogoplug-pro v3

Debian GNU/Linux bullseye/sid \n \l
Linux version 4.14.180-oxnas-tld-1 (root@tldDebian) (gcc version 8.3.0 (Debian 8.3.0-6) ) #1 SMP PREEMPT Fri Nov 15 04:12:45 PST 2019
Re: How to back up a live rootfs using rsync
November 30, 2020 12:19AM
> I did find a little shortcut to watching by using
> htop.I use a second terminal then simply run htop,
> what I do then is hit f4 for search and type in
> rsync and anything running rsync will show as it
> runs. What is nice is that it shows how much
> memory is being used at the same time.

Yes, it is a handy way to see what's going on!

This rsync setup has saved my ass a few times :)) the key is don't do --detete for a daily backup. Once in a while, you would turn on --delete to get rid of old files in the backup server. When it happens that you accidentally delete some file because of fat fingers or just sleepy, or tipsy, you can just restore that file from the backup server immediately :)

-bodhi
===========================
Forum Wiki
bodhi's corner
Re: How to back up a live rootfs using rsync
November 30, 2020 03:16AM
I can't tell you how many times I used to start all over with programming because of this or that.... I used to dd copy every so often then remove the copied usb which is effective to a certain extent however this is a much better way for sure!

learning is living! Echowarrior108



device pogoplug-pro v3

Debian GNU/Linux bullseye/sid \n \l
Linux version 4.14.180-oxnas-tld-1 (root@tldDebian) (gcc version 8.3.0 (Debian 8.3.0-6) ) #1 SMP PREEMPT Fri Nov 15 04:12:45 PST 2019
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: