When I ran the initial test that I stopped after 75 minutes, I should
have noted, I pushed the files to a new directory, so there was no
comparison performed. Nevertheless, the comparison part is rather quick.
The files are transferring at kilobits per second for some reason. I
watched my netdata output while running the transfer, along with other
data transfers, and it shows an iowait of well over 25% min during any
network transfer.
I backed everything up and wiped the drives. I created a software raid 1
and then luksEncrypted the raid device, formatted and mounted. I skipped
the LVM this time around. I want to test with a little less overhead and
see how it goes. If this performs better I might leave it. Otherwise, if
it still gets slow, I'm going to replace this beloved old A6 with a
shiny new Ryzen 3.
On 2020-02-06 19:52, Bob Elzer wrote:
> Well if you create a new filesystem and do an rsync then there is nothing to compare so the copy should go fast.
>
> If you have 46k files and they need to be compared before overwriting then that may take a little longer.
>
> Try copying a 2gb file across your nfs and see how long that takes. I once had a config error that caused my network copies to run slower than they should.
>
> Also run you rsync a second time to a full tmpfs and check the timing I suspect it will take longer. Not sure how many of your files change, but you might have to let some change to get a better reading.
>
> On Thu, Feb 6, 2020, 12:35 PM Nathan (PLUGAZ) <plugaz@codezilla.xyz> wrote:
>
>> I realize ext4 does not easily fragment, but when you have a large
>> volume with lots of files of differing size, how can you optimize it?
>>
>> I have a 2TB mirrored array that has hundreds of thousands of less than
>> 12KB files and hundreds of files that are more than 1MB and of course
>> lots of movies and such which can be 1 to 4GB. Over the years it has
>> gotten really slow.
>>
>> I have a shell script that basically runs rsync against my home
>> directory and pushes it to a specific folder on my file server (part of
>> this 2TB array).
>>
>> Typically the script runs in the wee hours when I'm asleep. But the
>> other day I decided to run it just to watch it and see what happens. It
>> was horrendously slow!
>> I tried timing it. I ran time { rsync -av /home/myuser/.cache/
>> remote:/backup/dir/.cache/; } and after 75 minutes I cancelled it. There
>> are 46k files in that folder and it is roughly 2GB... 75 minutes it
>> wasn't finished. Now this is running over an NFS link just FYI.
>>
>> So I created a 4GB tmpfs and mounted it where I needed and ran my time
>> backup again and it took 2 minutes and 6 seconds. Obviously my network
>> is not the issue.
>>
>> So today I'm trying to find places to store 2TB of data so I can
>> rearrange things, but I'm wondering...
>>
>> Is there a program that watches and optimizes placement of files on a
>> hard drive? I know these exist for windows, but linux?
>> ---------------------------------------------------
>> PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail settings:
>> https://lists.phxlinux.org/mailman/listinfo/plug-discuss---------------------------------------------------
PLUG-discuss mailing list -
PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss
(text/plain)