Filesystem Optimization

[Nathan (PLUGAZ)]

When I ran the initial test that I stopped after 75 minutes, I should
have noted, I pushed the files to a new directory, so there was no
comparison performed. Nevertheless, the comparison part is rather quick.
The files are transferring at kilobits per second for some reason. I
watched my netdata output while running the transfer, along with other
data transfers, and it shows an iowait of well over 25% min during any
network transfer. 

I backed everything up and wiped the drives. I created a software raid 1
and then luksEncrypted the raid device, formatted and mounted. I skipped
the LVM this time around. I want to test with a little less overhead and
see how it goes. If this performs better I might leave it. Otherwise, if
it still gets slow, I'm going to replace this beloved old A6 with a
shiny new Ryzen 3. 

On 2020-02-06 19:52, Bob Elzer wrote:

> Well if you create a new filesystem and do an rsync then there is nothing to compare so the copy should go fast. 
> 
> If you have 46k files and they need to be compared before overwriting then that may take a little longer. 
> 
> Try copying a 2gb file across your nfs and see how long that takes. I once had a config error that caused my network copies to run slower than they should. 
> 
> Also run you rsync a second time to a full tmpfs and check the timing I suspect it will take longer. Not sure how many of your files change, but you might have to let some change to get a better reading. 
> 
> On Thu, Feb 6, 2020, 12:35 PM Nathan (PLUGAZ) <plugaz at codezilla.xyz> wrote: 
> 
>> I realize ext4 does not easily fragment, but when you have a large 
>> volume with lots of files of differing size, how can you optimize it?
>> 
>> I have a 2TB mirrored array that has hundreds of thousands of less than 
>> 12KB files and hundreds of files that are more than 1MB and of course 
>> lots of movies and such which can be 1 to 4GB. Over the years it has 
>> gotten really slow.
>> 
>> I have a shell script that basically runs rsync against my home 
>> directory and pushes it to a specific folder on my file server (part of 
>> this 2TB array).
>> 
>> Typically the script runs in the wee hours when I'm asleep. But the 
>> other day I decided to run it just to watch it and see what happens. It 
>> was horrendously slow!
>> I tried timing it. I ran time { rsync -av /home/myuser/.cache/ 
>> remote:/backup/dir/.cache/; } and after 75 minutes I cancelled it. There 
>> are 46k files in that folder and it is roughly 2GB... 75 minutes it 
>> wasn't finished. Now this is running over an NFS link just FYI.
>> 
>> So I created a 4GB tmpfs and mounted it where I needed and ran my time 
>> backup again and it took 2 minutes and 6 seconds. Obviously my network 
>> is not the issue.
>> 
>> So today I'm trying to find places to store 2TB of data so I can 
>> rearrange things, but I'm wondering...
>> 
>> Is there a program that watches and optimizes placement of files on a 
>> hard drive? I know these exist for windows, but linux?
>> ---------------------------------------------------
>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phxlinux.org/pipermail/plug-discuss/attachments/20200206/ebea0a19/attachment.html>