Need help speeding up a backup

[Matt Graham]

On 2022-07-15 08:40, Mark Phillips via PLUG-discuss wrote:
> I am trying to back up the contents of the drives (ie /) to an
> external usb drive using rsync. It is taking a really long time.
> After 26 hours of continuous operation I have only transferred
> 138 GB out of 2+ TB

1G takes about 11.3 minutes?  That's super slow.  1G takes about 30 
seconds on a USB2 external disk here.  How is this mounted?
 
> rsync  --no-compress --info=progress2 -avAXEWSlHh --exclude=
> {'/run','/mnt','/swapfile','/boot','/dev','/proc','/sys',
> '/run','/mnt','/media','/lost+found','/swapfile.extended','/tmp'}
> / '/media/mark/Seagate Portable Drive/tsunami-backups-Jul_13_17-39/'
> Any suggestions on how I can speed this up and not lose any data?

You don't need -AX unless you're using ACLs and xattrs, and you 
probably aren't.  You probably don't want -W.  -H is IIRC heavyweight 
and you probably don't have many (or any) hard links on your 
filesystems.  The path to the backup disk looks like an automounter is 
involved, that can cause dumb things like being mounted with sync.

First suggestion, just use -av .  Second suggestion, instead of trying 
to back up everything, back up only the irreplaceable stuff.  This is 
usually /home and /etc , maybe /usr/local .  When rsyncing a lot of 
stuff (800G not 2T though), a significant amount of time gets spent 
stat()ing every single file and dir on both source and destination.  How 
many files are you dealing with?  /home here has 272G and 711889 inodes 
according to df -i.  An initial sync would take roughly 136 minutes, 
subsequent syncs would take roughly 4 minutes.  That's what I recall 
from the last time I used rsync to backup.

While it's written in Golang and therefore suspect, restic ( 
https://restic.github.io/ ) has an interesting feature that allows you 
to retain a varying number of hourly, daily, weekly, monthly, and yearly 
snapshots of whatever it is you want to back up.  The documentation 
about how to get to a specific restore point is not great, and the 
"mount restore point" functionality horks up directory permissions.  
However, it's quite fast at backing up stuff, and it can use S3 buckets 
and some other not-your-computer networky things instead of local disks 
as backup destinations.

-- 
Crow202 Blog: http://crow202.org/wordpress
There is no Darkness in Eternity
But only Light too dim for us to see.