Rsync does have a service. I would suggest looking at it.

On Sat, Feb 8, 2020, 12:00 PM Matt Graham <mhgraham@crow202.org> wrote:

> On 2020-02-07 11:20, Nathan (PLUGAZ) wrote:
> > On 2020-02-06 20:14, Matt Graham wrote:
> >> The "filefrag" utility will
> >> tell you how fragmented an individual file is.  I don't see anything
> >> about defragging tools for ext234 in portage/sys-fs/ but
> > Defragging an ext4 filesystem is done with e4defrag
>
> Putting the important stuff at the end is called "burying the lede" and
> you're not supposed to do that.  :-)  What does e4defrag -c
> /dev/whatever tell you?  My ext4 filesystems are all either 0 or 1 frag
> status, and the 1 frag was the one that's 87% full with a mix of file
> sizes.  Defragging is supposed to be necessary only if the frag status
> goes over 30.
>
> >>> Now this is running over an NFS link
> >> What was the rsync displaying after 75 minutes?
> >> 46,000 files seems like it should take a minute or so unless it had
> >> to
> >> transfer all the files in full.
> > The volume on the server is mounted with noatime,nodiratime,noexec
> > The export on the server is (rw,sync,no_root_squash,no_subtree_check)
>
> OK, you may want to try async.  I don't think it'll make much
> difference.
>
> > mount` shows the actual options as:
> > rw,noexec,noatime,nodiratime,vers=4.2,rsize=1048576,
> > wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,
> > sec=sys,fsc,local_lock=none
> > Both machines have their time synced.
>
> This is probably fine.  I couldn't use 4.2 for some reason, had to
> leave it at 4.1, but I don't think this'll cause the problem you're
> seeing.
>
> > Reading files from the server to my desktop is as fast as I would
> > expect a gigabit network to be. Writing large files to the network
> > storage works as I expect. It's just all of these small files
> > that are killing me.
>
> I made an 8G ext4 LV on machine1, exported it via NFS, on machine2, I
> did "mount -t nfs 192.168.1.20:/mnt/rsynctemp /mnt/other -o (the options
> you had)" , then started rsyncing a 7.5G dir tree to /mnt/other from
> machine2.  It *crawled* and the bottleneck was file creation, not
> fragmentation, since the filesystem had just been mkfs'ed.  The
> combination of NFS and ext4's write barriers makes creating a file at
> least an order of magnitude slower than it would usually be.  When the
> LV is mounted with ext4 defaults, rsyncing a 10,000 file directory of
> 157M in size to an empty dir took 54 minutes.  When the LV is mounted
> with nobarrier, doing that took 2 minutes.  (Curiously, the same
> operation with the LV as ext3 is taking much longer even though ext3
> doesn't do barriers.  Of course, portage is compiling qtwebengine right
> now, which is hogging the CPU.)
>
> So: mount the thing with nobarrier if you want it to create many small
> files faster.  This means that if the power goes out, your data will get
> corrupted.  Your call.  Or try running the rsync daemon on the NFS
> server, since NFS is good for reading, good for writing, and horrible
> for creating files?  That'll show totally different network behavior
> since it removes NFS's weirdnesses.
>
> --
> Crow202 Blog: http://crow202.org/wordpress
> There is no Darkness in Eternity
> But only Light too dim for us to see.
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> https://lists.phxlinux.org/mailman/listinfo/plug-discuss