cp: Argument list too long

[Bill Earl]

David, thanks for the great suggestions (and thanks also to der.hans and
G.D.!).  I've been temporarily moved onto another challenge, but I'm coming
back to this tomorrow morning.  I'll report back to the list what I find so
the list archive will have a recorded answer to the question after running
some tests.

I wish we could keep down the number of files in a directory, but so far any
method we've tried to break down the files into smaller groups has caused
other more complicated problems. I guess I should say that the files ARE
grouped into smaller groups, but each group of files needs to contain around
250k files!  I'm experimenting with the XFS file system right now on a Red
Hat 7.3 system, and so far it looks like access times are staying pretty
constant as the number of files grows. Actually using the system is pretty
speedy, which surprised me. The main problem is just getting the files INTO
the right location! :)  I have a test system that I'm going to use to try to
push that envelope (1M files in a directory, anyone?).

Along those lines, does anyone have any comments about XFS vs. ReiserFS for
large directory access times?

Again, thanks everyone for the great suggestions!

Bill Earl
Network Admin
Spectrum Data Solutions, LLC
www.bestbill.com


----- Original Message -----
From: "David Mandala" <davidm@them.com>
To: <plug-discuss@lists.plug.phoenix.az.us>
Sent: Monday, December 09, 2002 3:39 PM
Subject: Re: cp: Argument list too long


> You may want to not have so many files in a single directory. Linux
> stores directories in a liner list so accessing large directores is hard
> and slow (as you've noticed). There is a patch for the kernel that adds
> btree capabilities to the directories but you must build the kernel with
> it and you still can get grief with utilities that are not expecting so
> much data on the command line.
>
> Also you may need to just do $(ls) instead of $(ls *.pdf) and add an if
> statement where the mv or cp is so you only cp or mv a pdf file.
>
> here is a corrected version using an if statement, make sure to type the
> spaces correctly in the if statement, the [] braces must have spaces
> before and after them.
>
> for file in $(ls); do filename=$file; if [ ${filename##*.} = "pdf" ];
> then mv $file /newdir/; fi; done
>
>
>
> On Mon, 2002-12-09 at 14:54, David Mandala wrote:
> > This should work from the command line, make sure to replace the
> > /newdir/ with a proper path for you. Can use cp also. Make sure to type
> > this all on one command line.
> >
> > for file in $(ls *.pdf); do  mv $file /newdir/; done
> >
> > Warning I can't test this with as many files as you have but this should
> > work.
> >
> > Cheers,
> >
> > Davidm
> >
> > On Mon, 2002-12-09 at 14:15, Bill Earl wrote:
> > > Hello everyone!
> > >
> > > I need to copy, and occasionally move, some directories that contain
> > > thousands of files, (like around 250,000 pdfs), and the problem I'm
> > > having is that in the Bash shell, both cp and mv commands seem to die
> > > at somewhere around 3,000 files, so that if I enter
> > >
> > > cp /mydir/myfiles/*.pdf  /newdir/
> > >
> > > I get the following error:
> > >
> > > bash: /bin/cp: Argument list too long.
> > >
> > > I noticed that even just trying to list this many files can cause this
> > > error, depending on the syntax used. "ls /mydir/myfiles/*.pdf"
> > > generates the Argument list too long error, but "ls /mydir/myfiles"
> > > displays fine.
> > >
> > > I can get around it using a python script and more fiddling around
> > > than I really want to do. I'd really like to be able to move these
> > > large numbers of files around just through the shell.
> > >
> > > Is anyone aware of a separate file copying or moving (I need to do
> > > both, depending on the situation) utility that can handle very large
> > > numbers of files?  Would switching to a different shell solve this (if
> > > so, which one?), or is it a limit on the cp and mv commands?
> > >
> > > Thanks!
> > >
> > > Bill Earl
> > > Network Admin
> > > Spectrum Data Solutions, LLC
> > > www.bestbill.com
> --
> David IS Mandala
> gpg fingerprint 8932 E7EF CCF5 1B8C 1B5C  A92E C678 795E 45B2 D952
> Phoenix, AZ (480) 460-7546 HP, (602) 321-8277 CP
> http://www.them.com/~davidm/
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change  you mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>