From plug-discuss-bounces@lists.phxlinux.org Tue Feb 11 10:36:23 2020 Return-Path: X-Original-To: lurker@lists.phxlinux.org Delivered-To: lurker@lists.phxlinux.org Received: from phxlinux.org (localhost [127.0.0.1]) by phxlinux.org (Postfix) with ESMTP id 6001A32A0008; Tue, 11 Feb 2020 10:36:22 -0700 (MST) X-Original-To: plug-discuss@lists.phxlinux.org Delivered-To: plug-discuss@lists.phxlinux.org Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) by phxlinux.org (Postfix) with ESMTPS id B2AFE32A0005 for ; Tue, 11 Feb 2020 10:36:18 -0700 (MST) Received: by mail-il1-f182.google.com with SMTP id b15so4169049iln.3 for ; Tue, 11 Feb 2020 09:36:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=butash-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=gDE+5Ra07NnNp5QpBOzQJk659J+x74RVu/UtCJ7HSFU=; b=nvxlDGG/6DSYYM3NcsFTB/bOu8SPB/2Ckp4NJehUZ4FU2PjQgFc22gR9k9fNqAxF+x 57okAIt5vSPWk9pS6uKuxLmtyqFSEyQS8AjuEl3afvd/3aQWvDcHwOb+/udeFI3oUV1y pcJ387YwbXihIpZ0NrhOZoERgN6yo/wYugg5nP/yt2F5xsRiuL2bhvd7Npybv89Y4Gx4 GzUNnJG1TGC4yJSge14Y3ZXl1wx1fX85+CaRMBwmX8GASVguSe8dnKUlSiL47vC+AQPD x015jdBZ0RMOEheytwnlSZroCbthGzgOnATQO+t9+UYbdLcFO+i4cztl6KI9If7SdKxX 1H2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=gDE+5Ra07NnNp5QpBOzQJk659J+x74RVu/UtCJ7HSFU=; b=TUGAgPsT5Vu/yeS7eAGEr8TejALrEF0Yw4cnn6MgctGAQ4n7aoYs76cJfZfugdvOXk GzEW/rb0yOrQMMjeKrtxrSip+369ubGysceQa1bIsg+/+gMT/qXo7I4DD4EyJH3WDBol GrAdkFGGmdyYjGI7zXJ+E3AJlxPcZVwwe9pmSSV9OEKhN4obLebALEM39DxbKEWuKHMb bty4VrOBZP7ifarGrZ85P0vCxf9Okw3BV4EbW6RyCWOewufTsm9Hvw14fLBGpKKms4af I7EgpeJyAbx/RWdIO5/fk62yyUN05GDIIoK/9j1vOkkQ6FmLNOcvbWzgngq0vYeRL3UG ssGw== X-Gm-Message-State: APjAAAVq0WxCE9CutDpKRfib+dezidfemkZa0kG9nNm5gSPmAhR3AQdV LL0yjvalLzMzY5SwgA0+XuCJxsM0NUPIwYZT/Q2rLaZh X-Google-Smtp-Source: APXvYqxUS4pQv+toGTLlRPesINfMIV3iVQsxb0D6m6PvqTDxyZWi37ccKDUH7TjwGZF53KO4mCdkSSwfTQjL6vYR4AE= X-Received: by 2002:a92:4d3:: with SMTP id 202mr7586620ile.291.1581442577554; Tue, 11 Feb 2020 09:36:17 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Michael Butash Date: Tue, 11 Feb 2020 10:36:06 -0700 Message-ID: Subject: Re: Filesystem Optimization To: Main PLUG discussion list X-BeenThere: plug-discuss@lists.phxlinux.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Main PLUG discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Main PLUG discussion list Content-Type: multipart/mixed; boundary="===============1648247267712210422==" Errors-To: plug-discuss-bounces@lists.phxlinux.org Sender: "PLUG-discuss" --===============1648247267712210422== Content-Type: multipart/alternative; boundary="0000000000001172b1059e504b3c" --0000000000001172b1059e504b3c Content-Type: text/plain; charset="UTF-8" I've been doing something similar for years, where I use unison instead of rsync, but I delta changes between a laptop, desktop, and two filers (they sync between themselves), and never seen huge performance issues like that. Mine is some 100gb of files from 30mb visio/ppt files to (10's/100's of) thousands of .txt files of network configs from myself and customers I work for, and never had any issues on a local lan with this. I use all SSD's on my workstation/laptops, and my filer is ext4 on synology with 4x hgst 6tb spinners. A rescan of the local and remote share with unison tends to take around 10min on my laptop via cifs, much faster, maybe 3-4min on my desktop via nfs. I did this not long ago, not precise timing, but what I remember of doing it. This is also with ext4, lvm, and luks encryption, my laptop with a m.2 toshiba disk and my desktop a pair of 960pro samsung m.2's in mdraid 1. I've not run into slow-down or fragmentation issues of any sort in linux really, and both my laptop and desktop are now some 3+ year old now. I know between my synologies a full resync between the two of some 15tb of data takes roughly a week, and both have 4x 1gbe links in port channels worth of bandwidth through my arista switches... Furthermore, because of encryption I don't run trim or any real ssd wear leveling features, rather I found Samsung built-in wear leveling has been the best and unneeded to interact with. My toshiba m.2 in my xps15 has been solid so far too - normally I kill ssd's in 3-12mo without fail. I've expected to see performance issues over the years with them, but never since going ssd, other than they work, or they die quickly. Have you run iotop or anything else to look at disk usage during that if really an i/o issue? You should see an abundance of waits on the disk if so, as well as what is accessing your disks at that moment. Maybe try Unison instead of rsync as a test over nfs or cifs, not sure if something weird with rsync too. Worst case, buy a cheap couple TB spinner usb drive to offload to, zero the drive, reformat (cool kids seem to like xfs or zfs these days), and try again on clean disks. -mb On Thu, Feb 6, 2020 at 7:43 PM Nathan (PLUGAZ) wrote: > > When I ran the initial test that I stopped after 75 minutes, I should have > noted, I pushed the files to a new directory, so there was no comparison > performed. Nevertheless, the comparison part is rather quick. The files are > transferring at kilobits per second for some reason. I watched my netdata > output while running the transfer, along with other data transfers, and it > shows an iowait of well over 25% min during any network transfer. > > > I backed everything up and wiped the drives. I created a software raid 1 > and then luksEncrypted the raid device, formatted and mounted. I skipped > the LVM this time around. I want to test with a little less overhead and > see how it goes. If this performs better I might leave it. Otherwise, if it > still gets slow, I'm going to replace this beloved old A6 with a shiny new > Ryzen 3. > > > > On 2020-02-06 19:52, Bob Elzer wrote: > > Well if you create a new filesystem and do an rsync then there is nothing > to compare so the copy should go fast. > > If you have 46k files and they need to be compared before overwriting then > that may take a little longer. > > Try copying a 2gb file across your nfs and see how long that takes. I once > had a config error that caused my network copies to run slower than they > should. > > Also run you rsync a second time to a full tmpfs and check the timing I > suspect it will take longer. Not sure how many of your files change, but > you might have to let some change to get a better reading. > > > > > > On Thu, Feb 6, 2020, 12:35 PM Nathan (PLUGAZ) > wrote: > > > > I realize ext4 does not easily fragment, but when you have a large > volume with lots of files of differing size, how can you optimize it? > > I have a 2TB mirrored array that has hundreds of thousands of less than > 12KB files and hundreds of files that are more than 1MB and of course > lots of movies and such which can be 1 to 4GB. Over the years it has > gotten really slow. > > I have a shell script that basically runs rsync against my home > directory and pushes it to a specific folder on my file server (part of > this 2TB array). > > Typically the script runs in the wee hours when I'm asleep. But the > other day I decided to run it just to watch it and see what happens. It > was horrendously slow! > I tried timing it. I ran time { rsync -av /home/myuser/.cache/ > remote:/backup/dir/.cache/; } and after 75 minutes I cancelled it. There > are 46k files in that folder and it is roughly 2GB... 75 minutes it > wasn't finished. Now this is running over an NFS link just FYI. > > So I created a 4GB tmpfs and mounted it where I needed and ran my time > backup again and it took 2 minutes and 6 seconds. Obviously my network > is not the issue. > > So today I'm trying to find places to store 2TB of data so I can > rearrange things, but I'm wondering... > > Is there a program that watches and optimizes placement of files on a > hard drive? I know these exist for windows, but linux? > --------------------------------------------------- > PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org > To subscribe, unsubscribe, or to change your mail settings: > https://lists.phxlinux.org/mailman/listinfo/plug-discuss > > > --------------------------------------------------- > PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org > To subscribe, unsubscribe, or to change your mail settings: > https://lists.phxlinux.org/mailman/listinfo/plug-discuss --0000000000001172b1059e504b3c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I've been doing something similar for years, where I u= se unison instead of rsync, but I delta changes between a laptop, desktop, = and two filers (they sync between themselves), and never seen huge performa= nce issues like that.=C2=A0 Mine is some 100gb of files from 30mb visio/ppt= files to (10's/100's of) thousands of .txt files of network config= s from myself and customers I work for, and never had any issues on a local= lan with this.

I use all SSD's on my workstation/la= ptops, and my filer is ext4 on synology with 4x hgst 6tb spinners.=C2=A0 A = rescan of the local and remote share with unison tends to take around 10min= on my laptop via cifs, much faster, maybe 3-4min on my desktop via nfs.=C2= =A0 I did this not long ago, not precise timing, but what I remember of doi= ng it.=C2=A0 This is also with ext4, lvm, and luks encryption, my laptop wi= th=C2=A0a m.2 toshiba disk and my desktop a pair of 960pro samsung m.2'= s in mdraid 1.=C2=A0 I've not run into slow-down or fragmentation issue= s of any sort in linux really, and both my laptop and desktop are now some = 3+ year old now.

I know between my synologies a fu= ll resync between the two of some 15tb of data takes roughly a week, and bo= th have 4x 1gbe links in port channels worth of bandwidth through my arista= switches...=C2=A0

Furthermore, because=C2=A0of en= cryption I don't run trim or any real ssd wear leveling features, rathe= r I found Samsung built-in wear leveling has been the best and unneeded to = interact with.=C2=A0 My toshiba m.2 in my xps15 has been solid so far too -= normally I kill ssd's in 3-12mo without fail.=C2=A0 I've expected = to see performance issues over the years with them, but never since going s= sd, other than they work, or they die quickly.

Hav= e you run iotop or anything else to look at disk usage during that if reall= y an i/o issue?=C2=A0 You should see an abundance of waits on the disk if s= o, as well as what is accessing your disks at that moment.=C2=A0 Maybe try = Unison instead of rsync as a test over nfs or cifs, not sure if something w= eird with rsync too.=C2=A0 Worst case, buy a cheap couple TB spinner usb dr= ive to offload to, zero the drive, reformat (cool kids seem to like xfs or = zfs these days), and try again on clean disks.

-mb=



On Thu, Feb 6, 2020 at 7:43 PM Nathan (= PLUGAZ) <plugaz@codezilla.xyz> wrote:


When I ran the initial test that I stopped after 75 minutes, I should ha= ve noted, I pushed the files to a new directory, so there was no comparison= performed. Nevertheless, the comparison part is rather quick. The files ar= e transferring at kilobits per second for some reason. I watched my netdata= output while running the transfer, along with other data transfers, and it= shows an iowait of well over 25% min during any network transfer.


I backed everything up and wiped the drives. I created a software raid 1= and then luksEncrypted the raid device, formatted and mounted. I skipped t= he LVM this time around. I want to test with a little less overhead and see= how it goes. If this performs better I might leave it. Otherwise, if it st= ill gets slow, I'm going to replace this beloved old A6 with a shiny ne= w Ryzen 3.



On 2020-02-06 19:52, Bob = Elzer wrote:

Well if you create a new filesystem and do an rsync then = there is nothing to compare so the copy should go fast.
=C2=A0
If you have 46k files and they need to be compared before= overwriting then that may take a little longer.
=C2=A0
Try copying a 2gb file across your nfs and see how long t= hat takes. I once had a config error that caused my network copies to run s= lower than they should.
=C2=A0
Also run you rsync a second time to a full tmpfs and chec= k the timing I suspect it will take longer. Not sure how many of your files= change, but you might have to let some change to get a better reading.
=C2=A0
=C2=A0
=C2=A0
=C2=A0



I realize ext4 does not easily fragment= , but when you have a large
volume with lots of files of differing size= , how can you optimize it?

I have a 2TB mirrored array that has hund= reds of thousands of less than
12KB files and hundreds of files that ar= e more than 1MB and of course
lots of movies and such which can be 1 to= 4GB. Over the years it has
gotten really slow.

I have a shell s= cript that basically runs rsync against my home
directory and pushes it= to a specific folder on my file server (part of
this 2TB array).
Typically the script runs in the wee hours when I'm asleep. But the <= br>other day I decided to run it just to watch it and see what happens. It =
was horrendously slow!
I tried timing it. I ran time { rsync -av /ho= me/myuser/.cache/
remote:/backup/dir/.cache/; } and after 75 minutes I = cancelled it. There
are 46k files in that folder and it is roughly 2GB.= .. 75 minutes it
wasn't finished. Now this is running over an NFS l= ink just FYI.

So I created a 4GB tmpfs and mounted it where I needed= and ran my time
backup again and it took 2 minutes and 6 seconds. Obvi= ously my network
is not the issue.

So today I'm trying to fi= nd places to store 2TB of data so I can
rearrange things, but I'm w= ondering...

Is there a program that watches and optimizes placement = of files on a
hard drive? I know these exist for windows, but linux?---------------------------------------------------
PLUG-discuss mailin= g list - PLUG-discuss@lists.phxlinux.org
To subscribe, = unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/plug-discuss<= /blockquote>


---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
https://lists.phxlinux.org/mailman/listinfo/= plug-discuss
--0000000000001172b1059e504b3c-- --===============1648247267712210422== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClBMVUct ZGlzY3VzcyBtYWlsaW5nIGxpc3QgLSBQTFVHLWRpc2N1c3NAbGlzdHMucGh4bGludXgub3JnClRv IHN1YnNjcmliZSwgdW5zdWJzY3JpYmUsIG9yIHRvIGNoYW5nZSB5b3VyIG1haWwgc2V0dGluZ3M6 Cmh0dHBzOi8vbGlzdHMucGh4bGludXgub3JnL21haWxtYW4vbGlzdGluZm8vcGx1Zy1kaXNjdXNz --===============1648247267712210422==--