Filesystem Optimization

[Matt Graham]

On 2020-02-07 11:20, Nathan (PLUGAZ) wrote:
> On 2020-02-06 20:14, Matt Graham wrote:
>> The "filefrag" utility will
>> tell you how fragmented an individual file is.  I don't see anything
>> about defragging tools for ext234 in portage/sys-fs/ but
> Defragging an ext4 filesystem is done with e4defrag

Putting the important stuff at the end is called "burying the lede" and 
you're not supposed to do that.  :-)  What does e4defrag -c 
/dev/whatever tell you?  My ext4 filesystems are all either 0 or 1 frag 
status, and the 1 frag was the one that's 87% full with a mix of file 
sizes.  Defragging is supposed to be necessary only if the frag status 
goes over 30.

>>> Now this is running over an NFS link
>> What was the rsync displaying after 75 minutes?
>> 46,000 files seems like it should take a minute or so unless it had 
>> to
>> transfer all the files in full.
> The volume on the server is mounted with noatime,nodiratime,noexec
> The export on the server is (rw,sync,no_root_squash,no_subtree_check)

OK, you may want to try async.  I don't think it'll make much 
difference.

> mount` shows the actual options as:
> rw,noexec,noatime,nodiratime,vers=4.2,rsize=1048576,
> wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,
> sec=sys,fsc,local_lock=none
> Both machines have their time synced.

This is probably fine.  I couldn't use 4.2 for some reason, had to 
leave it at 4.1, but I don't think this'll cause the problem you're 
seeing.

> Reading files from the server to my desktop is as fast as I would
> expect a gigabit network to be. Writing large files to the network
> storage works as I expect. It's just all of these small files
> that are killing me.

I made an 8G ext4 LV on machine1, exported it via NFS, on machine2, I 
did "mount -t nfs 192.168.1.20:/mnt/rsynctemp /mnt/other -o (the options 
you had)" , then started rsyncing a 7.5G dir tree to /mnt/other from 
machine2.  It *crawled* and the bottleneck was file creation, not 
fragmentation, since the filesystem had just been mkfs'ed.  The 
combination of NFS and ext4's write barriers makes creating a file at 
least an order of magnitude slower than it would usually be.  When the 
LV is mounted with ext4 defaults, rsyncing a 10,000 file directory of 
157M in size to an empty dir took 54 minutes.  When the LV is mounted 
with nobarrier, doing that took 2 minutes.  (Curiously, the same 
operation with the LV as ext3 is taking much longer even though ext3 
doesn't do barriers.  Of course, portage is compiling qtwebengine right 
now, which is hogging the CPU.)

So: mount the thing with nobarrier if you want it to create many small 
files faster.  This means that if the power goes out, your data will get 
corrupted.  Your call.  Or try running the rsync daemon on the NFS 
server, since NFS is good for reading, good for writing, and horrible 
for creating files?  That'll show totally different network behavior 
since it removes NFS's weirdnesses.

-- 
Crow202 Blog: http://crow202.org/wordpress
There is no Darkness in Eternity
But only Light too dim for us to see.