When I ran the initial test that I stopped after 75 minutes, I should have noted, I pushed the files to a new directory, so there was no comparison performed. Nevertheless, the comparison part is rather quick. The files are transferring at kilobits per second for some reason. I watched my netdata output while running the transfer, along with other data transfers, and it shows an iowait of well over 25% min during any network transfer.

I backed everything up and wiped the drives. I created a software raid 1 and then luksEncrypted the raid device, formatted and mounted. I skipped the LVM this time around. I want to test with a little less overhead and see how it goes. If this performs better I might leave it. Otherwise, if it still gets slow, I'm going to replace this beloved old A6 with a shiny new Ryzen 3.

On 2020-02-06 19:52, Bob Elzer wrote:

Well if you create a new filesystem and do an rsync then there is nothing to compare so the copy should go fast.

 

If you have 46k files and they need to be compared before overwriting then that may take a little longer.

 

Try copying a 2gb file across your nfs and see how long that takes. I once had a config error that caused my network copies to run slower than they should.

 

Also run you rsync a second time to a full tmpfs and check the timing I suspect it will take longer. Not sure how many of your files change, but you might have to let some change to get a better reading.

 

 

 

 

On Thu, Feb 6, 2020, 12:35 PM Nathan (PLUGAZ) <plugaz@codezilla.xyz> wrote:

I realize ext4 does not easily fragment, but when you have a large
volume with lots of files of differing size, how can you optimize it?

I have a 2TB mirrored array that has hundreds of thousands of less than
12KB files and hundreds of files that are more than 1MB and of course
lots of movies and such which can be 1 to 4GB. Over the years it has
gotten really slow.

I have a shell script that basically runs rsync against my home
directory and pushes it to a specific folder on my file server (part of
this 2TB array).

Typically the script runs in the wee hours when I'm asleep. But the
other day I decided to run it just to watch it and see what happens. It
was horrendously slow!
I tried timing it. I ran time { rsync -av /home/myuser/.cache/
remote:/backup/dir/.cache/; } and after 75 minutes I cancelled it. There
are 46k files in that folder and it is roughly 2GB... 75 minutes it
wasn't finished. Now this is running over an NFS link just FYI.

So I created a 4GB tmpfs and mounted it where I needed and ran my time
backup again and it took 2 minutes and 6 seconds. Obviously my network
is not the issue.

So today I'm trying to find places to store 2TB of data so I can
rearrange things, but I'm wondering...

Is there a program that watches and optimizes placement of files on a
hard drive? I know these exist for windows, but linux?
---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss