How to back up a live rootfs using rsync

Re: How to back up a live rootfs using rsync

bodhi — Sun, 01 Jan 2023 17:16:51 -0600

Updated 01 Jan 2023

I've updated the script rsync.backup.ondemand to make it a bit more robust. I'm making sure the USB can be used as the rootfs and a boot drive.

Re: How to back up a live rootfs using rsync

bodhi — Sun, 21 Feb 2021 16:50:44 -0600

FYI,

I've done with the tutorial in the 1st post.

Re: How to back up a live rootfs using rsync

echowarrior108 — Mon, 30 Nov 2020 22:39:16 -0600

Quote
If you writing code or scripts, you should really be using version control such as git or svn. And rsync backup daily using this process.

only toying with a few scripts thus far, kinda new to that. actually been trying a little bit of anything to see what this device can do then if I like it i incorporate it to my main plug which right now is mostly a home server. I may try git or svn later when I can dedicate more time to it:)

Re: How to back up a live rootfs using rsync

bodhi — Mon, 30 Nov 2020 18:17:46 -0600

> I can't tell you how many times I used to start
> all over with programming because of this or
> that.... I used to dd copy every so often then
> remove the copied usb which is effective to a
> certain extent however this is a much better way
> for sure!

If you writing code or scripts, you should really be using version control such as git or svn. And rsync backup daily using this process.

Re: How to back up a live rootfs using rsync

echowarrior108 — Mon, 30 Nov 2020 03:16:53 -0600

I can't tell you how many times I used to start all over with programming because of this or that.... I used to dd copy every so often then remove the copied usb which is effective to a certain extent however this is a much better way for sure!

Re: How to back up a live rootfs using rsync

bodhi — Mon, 30 Nov 2020 00:19:15 -0600

> I did find a little shortcut to watching by using
> htop.I use a second terminal then simply run htop,
> what I do then is hit f4 for search and type in
> rsync and anything running rsync will show as it
> runs. What is nice is that it shows how much
> memory is being used at the same time.

Yes, it is a handy way to see what's going on!

This rsync setup has saved my ass a few times :)) the key is don't do --detete for a daily backup. Once in a while, you would turn on --delete to get rid of old files in the backup server. When it happens that you accidentally delete some file because of fat fingers or just sleepy, or tipsy, you can just restore that file from the backup server immediately :)

Re: How to back up a live rootfs using rsync

echowarrior108 — Sun, 29 Nov 2020 23:37:44 -0600

I did find a little shortcut to watching by using htop.I use a second terminal then simply run htop, what I do then is hit f4 for search and type in rsync and anything running rsync will show as it runs. What is nice is that it shows how much memory is being used at the same time.

It was massive on the first try for sure! it goes fast now.

currently I am running an ssd on the sata drive and rsynking to usb as described so I am sure it is a bit faster than most but I will be trying out on my mpcie drive device soon which will take a bit longer.

Re: How to back up a live rootfs using rsync

bodhi — Sun, 29 Nov 2020 18:53:06 -0600

echowarrior108 Wrote:
-------------------------------------------------------
> see I missed it lol

:) the first time you do back up it will be massive (logrotate will eventually compress it). From now own the log will be smaller.

Re: How to back up a live rootfs using rsync

echowarrior108 — Sun, 29 Nov 2020 17:00:36 -0600

see I missed it lol

Re: How to back up a live rootfs using rsync

bodhi — Sun, 29 Nov 2020 15:31:22 -0600

echowarrior108,

> is there a way to show progress while running or
> did I miss something?

That's part of what I need to describe in Section II.

The log you can monitor is in the rsync command.

/var/log/rsync.backup.daily.log
/tmp/rsync.backup.daily.log

And in the backup server,

/tmp/rsyncd.log

Re: How to back up a live rootfs using rsync

echowarrior108 — Sun, 29 Nov 2020 14:33:19 -0600

bodhi, good work, here was how my rsync.backup.on demand looked on my pogoplug pro v3 test device

/media/USB# /root/rsync.backup.ondemand
<13>Nov 29 15:09:33 root[12560]: Started rsync on-demand backup for ppdebian
<13>Nov 29 15:28:46 root[12683]: Finished rsync on-demand backup for ppdebian

is there a way to show progress while running or did I miss something?

Re: How to back up a live rootfs using rsync

bodhi — Fri, 27 Nov 2020 18:22:16 -0600

This tutorial is almost done. I just need to explain a few things, i.e. what names to change, etc.

It can be used now by experience users who know how to customize the set up for their own configurtion.

Re: rsync - How to setup rsync job to back up a live rootfs

bodhi — Thu, 19 Nov 2020 22:51:08 -0600

Feel free to ask questions while I am modifying the 1st post (you might have done research before about this subject).

The set up is a little bit tedious, especially the rsync syntax is tricky.

Re: rsync - How to setup rsync job to back up a live rootfs

renojim — Thu, 19 Nov 2020 21:41:44 -0600

Looking forward to this. It's something I've wanted to look into for a while now to handle the remote units I take care of.

Re: rsync - How to setup rsync job to back up a live rootfs

bodhi — Thu, 19 Nov 2020 16:53:40 -0600

LeggoMyEggo Wrote:
-------------------------------------------------------
> Backed up and compressed I assume.

rsync compress the files to save bandwith while sending. However, the backup area is an uncompressed folder ( a mirror of the source folder / ).

Re: rsync - How to setup rsync job to back up a live rootfs

LeggoMyEggo — Thu, 19 Nov 2020 07:53:30 -0600

Backed up and compressed I assume.

Re: rsync - How to setup rsync job to back up a live rootfs

Gravelrash — Thu, 19 Nov 2020 03:37:11 -0600

:thumbsup!

How to back up a live rootfs using rsync

bodhi — Wed, 18 Nov 2020 16:34:38 -0600

I. INSTALLATION

A. Setup rsync to back up a live rootfs to a network backup server

A.1. Network Client side rsync

This is where the rootfs is, while the system is running.

1. Install rsync

apt-get install rsync

2. Download and copy the attached script rsync.backup.daily to root home folder

cp -a rsync.backup.daily /root/rsync.backup.daily
chmod +x /root/rsync.backup.daily

3. Create the rsync password file. And secure it.

echo "mybackuppasswd" > /root/rsync.backup.daily.pswd
chmod 600 /root/rsync.backup.daily.pswd

4. Set up a cron job to run at 11:25 AM dailly

Install cron if not already installed

apt-get install cron

Edit the crontab

crontab -e

And add the following line at the end (or replace/modify if this time slot is already in the crontab)

25 11 * * * /root/rsync.backup.daily

List the cron jobs to doublecheck

crontab -l

5. Setup logrotate

Install logrotate if not already installed

apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf

/var/log/rsync.backup.daily.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 4
    compress
}

A.2. Network Server side rsync

This is where the back up server is.

1. Install rsync

apt-get install rsync

2. Enable rsync daemon at boot

Edit /etc/default/rsync to set enable flag to true

RSYNC_ENABLE=true

3. Download and copy the attached /etc/rsyncd.conf

cp -a rsyncd.conf /etc/rsyncd.conf

4. Create the rsync credential file. And secure it

echo "mybackup:mybackuppasswd" >  /etc/rsyncd.scrt
chmod 600 /etc/rsyncd.scrt

5. Setup logrotate

Install logrotate if not already installed

apt-get install logrotate

Add the following section for rsync at the end of /etc/logrotate.conf

/tmp/rsyncd.log {
    missingok
    size 256k
    su root root
    create 0660 root root
    rotate 8
    compress
}

6. Start rsync daemon

/etc/init.d/rsync start

B. Setup rsync to back up a live rootfs to a local USB drive

This section B is independent from section A above. Meaning there is no need to the whole set up as described in section A.1 and A.2 in order to do backup to USB.

1. Install rsync if not already installed

apt-get install rsync

2. Download and copy the attached script rsync.backup.ondemand to root home folder

cp -a rsync.backup.ondemand /root/rsync.backup.ondemand
chmod +x /root/rsync.backup.ondemand

3. Running on-demand backup

- Format a USB drive with Ext3 (Ext4 is OK if uboot on the rootfs box can recognize it)
- Label it with something meaningful, e.g. BackupRootfs

3.1 Mount the USB backup drive to /media. This is required so that we can avoid mistake in being at the wrong place.

3.2 Start the back up from the USB drive

cd /media/BackupRootfs
/root/rsync.backup.ondemand &

From now on the rsync will do incremental backup to the USB when we run the back up commands above (steps 3.1 and 3.2).

Note:

TBD - this process will be customized further to make the script run automatically whenever the backup USB drive is inserted.

II. Glossary

/root/rsync.backup.daily
The main script. Scheduled by cron

/etc/rsyncd.conf
The rsync conf file for the backup server (used by rsync daemon)

/root/rsync.backup.daily.pswd
The password file that rsync uses to authenticate the client backup job with the backup server.

/etc/rsyncd.scrt
The credential file that rsync on the with the backup server side uses to authenticate the client backup request.

/root/rsync.backup.ondemand
The on-demand script. Back up the rootfs to a USB drive.

/var/log/rsync.backup.daily.log
The client side log file

/tmp/rsyncd.log
The backup server side log file

=====================

Updated 01 Jan 2023

I've updated the script rsync.backup.ondemand to make it a bit more robust. I'm making sure the USB can be used as the rootfs and a boot drive.