<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /></head><body style='font-size: 10pt; font-family: Verdana,Geneva,sans-serif'>
<p><br /></p>
<p>When I ran the initial test that I stopped after 75 minutes, I should have noted, I pushed the files to a new directory, so there was no comparison performed. Nevertheless, the comparison part is rather quick. The files are transferring at kilobits per second for some reason. I watched my netdata output while running the transfer, along with other data transfers, and it shows an iowait of well over 25% min during any network transfer.</p>
<p><br /></p>
<p>I backed everything up and wiped the drives. I created a software raid 1 and then luksEncrypted the raid device, formatted and mounted. I skipped the LVM this time around. I want to test with a little less overhead and see how it goes. If this performs better I might leave it. Otherwise, if it still gets slow, I'm going to replace this beloved old A6 with a shiny new Ryzen 3.</p>
<p><br /></p>
<p><br /></p>
<p id="reply-intro">On 2020-02-06 19:52, Bob Elzer wrote:</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0">
<div id="replybody1">
<div dir="auto">Well if you create a new filesystem and do an rsync then there is nothing to compare so the copy should go fast.
<div dir="auto"> </div>
<div dir="auto">If you have 46k files and they need to be compared before overwriting then that may take a little longer.</div>
<div dir="auto"> </div>
<div dir="auto">Try copying a 2gb file across your nfs and see how long that takes. I once had a config error that caused my network copies to run slower than they should.</div>
<div dir="auto"> </div>
<div dir="auto">Also run you rsync a second time to a full tmpfs and check the timing I suspect it will take longer. Not sure how many of your files change, but you might have to let some change to get a better reading.</div>
<div dir="auto"> </div>
<div dir="auto"> </div>
<div dir="auto"> </div>
<div dir="auto"> </div>
</div>
<br />
<div class="v1gmail_quote">
<div class="v1gmail_attr" dir="ltr">On Thu, Feb 6, 2020, 12:35 PM Nathan (PLUGAZ) <<a href="mailto:plugaz@codezilla.xyz" rel="noreferrer">plugaz@codezilla.xyz</a>> wrote:</div>
<blockquote class="v1gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><br /><br />I realize ext4 does not easily fragment, but when you have a large <br />volume with lots of files of differing size, how can you optimize it?<br /><br />I have a 2TB mirrored array that has hundreds of thousands of less than <br />12KB files and hundreds of files that are more than 1MB and of course <br />lots of movies and such which can be 1 to 4GB. Over the years it has <br />gotten really slow.<br /><br />I have a shell script that basically runs rsync against my home <br />directory and pushes it to a specific folder on my file server (part of <br />this 2TB array).<br /><br />Typically the script runs in the wee hours when I'm asleep. But the <br />other day I decided to run it just to watch it and see what happens. It <br />was horrendously slow!<br />I tried timing it. I ran time { rsync -av /home/myuser/.cache/ <br />remote:/backup/dir/.cache/; } and after 75 minutes I cancelled it. There <br />are 46k files in that folder and it is roughly 2GB... 75 minutes it <br />wasn't finished. Now this is running over an NFS link just FYI.<br /><br />So I created a 4GB tmpfs and mounted it where I needed and ran my time <br />backup again and it took 2 minutes and 6 seconds. Obviously my network <br />is not the issue.<br /><br />So today I'm trying to find places to store 2TB of data so I can <br />rearrange things, but I'm wondering...<br /><br />Is there a program that watches and optimizes placement of files on a <br />hard drive? I know these exist for windows, but linux?<br />---------------------------------------------------<br />PLUG-discuss mailing list - <a href="mailto:PLUG-discuss@lists.phxlinux.org" rel="noreferrer">PLUG-discuss@lists.phxlinux.org</a><br />To subscribe, unsubscribe, or to change your mail settings:<br /><a href="https://lists.phxlinux.org/mailman/listinfo/plug-discuss" target="_blank" rel="noopener noreferrer">https://lists.phxlinux.org/mailman/listinfo/plug-discuss</a></blockquote>
</div>
</div>
</blockquote>
<p><br /></p>
</body></html>