<div dir="ltr"><div>I don't know why your copy is slow, but I know how to make it take less time.<br></div><div>I think you are defeating the purpose of rsync, you are creating a new directory every time, which means it needs to copy everything every time.</div><div><br></div><div>What I do is do an rsync -rau --delete from the directory I want to copy to the same place every time</div><div>i.e. rsync -rau --delete excludedirs fromdir todir<br></div><div>rsync is smart enough to only do copies of files that changed, your command is copying some files that never change or change very few times.</div><div>Doing it to the same place makes the sync go much faster.</div><div><br></div><div>After the sync, I then do a tar czf output-daily-todaysdate-time.tgz ./directorytosave <br></div><div><br></div><div>then I do a find output-daily* and delete the older ones after so many days</div><div><br></div><div>I also use fsarchiver to make a copy of my system partition (it does copy everything)<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 15, 2022 at 4:32 PM Matt Graham via PLUG-discuss <<a href="mailto:plug-discuss@lists.phxlinux.org">plug-discuss@lists.phxlinux.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2022-07-15 08:40, Mark Phillips via PLUG-discuss wrote:<br>
> I am trying to back up the contents of the drives (ie /) to an<br>
> external usb drive using rsync. It is taking a really long time.<br>
> After 26 hours of continuous operation I have only transferred<br>
> 138 GB out of 2+ TB<br>
<br>
1G takes about 11.3 minutes? That's super slow. 1G takes about 30 <br>
seconds on a USB2 external disk here. How is this mounted?<br>
<br>
> rsync --no-compress --info=progress2 -avAXEWSlHh --exclude=<br>
> {'/run','/mnt','/swapfile','/boot','/dev','/proc','/sys',<br>
> '/run','/mnt','/media','/lost+found','/swapfile.extended','/tmp'}<br>
> / '/media/mark/Seagate Portable Drive/tsunami-backups-Jul_13_17-39/'<br>
> Any suggestions on how I can speed this up and not lose any data?<br>
<br>
You don't need -AX unless you're using ACLs and xattrs, and you <br>
probably aren't. You probably don't want -W. -H is IIRC heavyweight <br>
and you probably don't have many (or any) hard links on your <br>
filesystems. The path to the backup disk looks like an automounter is <br>
involved, that can cause dumb things like being mounted with sync.<br>
<br>
First suggestion, just use -av . Second suggestion, instead of trying <br>
to back up everything, back up only the irreplaceable stuff. This is <br>
usually /home and /etc , maybe /usr/local . When rsyncing a lot of <br>
stuff (800G not 2T though), a significant amount of time gets spent <br>
stat()ing every single file and dir on both source and destination. How <br>
many files are you dealing with? /home here has 272G and 711889 inodes <br>
according to df -i. An initial sync would take roughly 136 minutes, <br>
subsequent syncs would take roughly 4 minutes. That's what I recall <br>
from the last time I used rsync to backup.<br>
<br>
While it's written in Golang and therefore suspect, restic ( <br>
<a href="https://restic.github.io/" rel="noreferrer" target="_blank">https://restic.github.io/</a> ) has an interesting feature that allows you <br>
to retain a varying number of hourly, daily, weekly, monthly, and yearly <br>
snapshots of whatever it is you want to back up. The documentation <br>
about how to get to a specific restore point is not great, and the <br>
"mount restore point" functionality horks up directory permissions. <br>
However, it's quite fast at backing up stuff, and it can use S3 buckets <br>
and some other not-your-computer networky things instead of local disks <br>
as backup destinations.<br>
<br>
-- <br>
Crow202 Blog: <a href="http://crow202.org/wordpress" rel="noreferrer" target="_blank">http://crow202.org/wordpress</a><br>
There is no Darkness in Eternity<br>
But only Light too dim for us to see.<br>
---------------------------------------------------<br>
PLUG-discuss mailing list - <a href="mailto:PLUG-discuss@lists.phxlinux.org" target="_blank">PLUG-discuss@lists.phxlinux.org</a><br>
To subscribe, unsubscribe, or to change your mail settings:<br>
<a href="https://lists.phxlinux.org/mailman/listinfo/plug-discuss" rel="noreferrer" target="_blank">https://lists.phxlinux.org/mailman/listinfo/plug-discuss</a></blockquote></div>