Professional Web Applications Themes

sh: line 1: /bin/cp: Argument list too long - Linux / Unix Administration

Hi all: I try to run the script which include the sentence as following : my $cmdline="cp /home/binye/data/test_perl/*dmp /home/binye/data/data/dmp_txt/"; system($cmdline); However the system told me " sh: line 1: /bin/cp: Argument list too long" It is really weird. I try to run the command like "ls -f -r *.dmp" ALso get the same message from the system, what is wrong with that ? Thanks for any comments. bin...

  1. #1

    Default sh: line 1: /bin/cp: Argument list too long

    Hi all:

    I try to run the script which include the sentence as following :
    my $cmdline="cp /home/binye/data/test_perl/*dmp
    /home/binye/data/data/dmp_txt/";
    system($cmdline);

    However the system told me " sh: line 1: /bin/cp: Argument list too
    long"

    It is really weird. I try to run the command like "ls -f -r *.dmp"
    ALso get the same message from the system, what is wrong with that ?

    Thanks for any comments.

    bin

    yezi Guest

  2. #2

    Default Re: sh: line 1: /bin/cp: Argument list too long


    yezi wrote: 

    The system is trying to tell you that when it expanded the list
    "/home/binye/data/test_perl/*dmp" that the resulting argument list was
    too long to handle. You didn't specify an OS, but some are / were
    quite restrictive. Check the ARG_MAX parameter in
    /usr/include/limits.h.

    If you are close to the limit, then you might try to "cd" to the
    directory and then copy the files w/o the full path. NOTE: even if this
    works, may eventually fail as entries are added.

    A better solution would be something like:

    find /home/binye/data/test_perl -name "*dmp" -exec cp {}
    /home/binye/data/dmt_txt/.;

    which "finds" each file individually and copies it.

    scott.myron@steelox.com Guest

  3. #3

    Default Re: sh: line 1: /bin/cp: Argument list too long

    The system is the linux Fedora

    yezi Guest

  4. #4

    Default Re: sh: line 1: /bin/cp: Argument list too long

    The strange thing is I run the script before,and the number of files is
    same as now. It works very well. I suspect some wrong with the system.

    yezi Guest

  5. #5

    Default Re: sh: line 1: /bin/cp: Argument list too long

    On 6 Feb 2006 13:11:47 -0800, yezi <com> wrote: 

    I suspect that the number of files is _not_ the same as now. Look into
    the find command, specifically with -exec and an xargs.


    Dave Guest

  6. #6

    Default Re: sh: line 1: /bin/cp: Argument list too long

    Try to Check the ARG_MAX parameter in
    /usr/include/limits.h.

    The limits.h exists, but can not find the parameter ARG_MAX.

    yezi Guest

  7. #7

    Default Re: sh: line 1: /bin/cp: Argument list too long

    yezi wrote: 

    There's no way the number of files is the same as before. Suspect
    all you want about the system, but this is a standard topic to be
    learned by SAGE Novice level sysadmins for them to get to
    Junior level. Extremely standard UNIX toolsmithing and
    mentorship stuff. Should be covered by your mentor in the first
    year of an well organized apprenticeship program.

    Command lines are limited by the shell buffer. Standard input is
    not. This is your chance to learn how to combine find and xargs
    to take advantage of the fact that standard input doesn't have that
    sort of limit.

    General process: Make your "ls" not have names on its lines. Use
    "grep" to filter. Use "xargs" to convert back from standard input to
    command line. Use "cp" to do the move as you originally intended.

    Doug Guest

  8. #8

    Default Re: sh: line 1: /bin/cp: Argument list too long

    On 6 Feb 2006 16:24:44 -0800, Doug Freyburger <com> wrote: 
    >
    > There's no way the number of files is the same as before. Suspect
    > all you want about the system, but this is a standard topic to be
    > learned by SAGE Novice level sysadmins for them to get to
    > Junior level. Extremely standard UNIX toolsmithing and
    > mentorship stuff. Should be covered by your mentor in the first
    > year of an well organized apprenticeship program.[/ref]

    Yup. That's why I pointed him to the keywords "find", "exec", and
    "xargs". Let's see if he puts 'em together in the right order, it's one
    good way to solve the problem he's having and to learn a few things at
    the same time.
     

    Yup!

    Dave Guest

  9. #9

    Default Re: sh: line 1: /bin/cp: Argument list too long

    2006-02-6, 09:59(-08), yezi: 
    [...]

    That's a limitation of the execve system call (and the size of
    the area used to store the argument and environment list), not
    of cp in particular.

    The the number of files is not that big, it may be that the
    environment is full.

    $cmdline = "env -i sh -c '
    command -p cp /home/binye/data/test_perl/*dmp \
    /home/binye/data/data/dmp_txt/'"

    might then help. But you should rather investigate why your
    environment is so full.

    Or as someone suggested:

    $cmdline = "
    cd /home/binye/data/test_perl &&
    cp -- *dmp ../data/dmp_txt"

    --
    Stéphane
    Stephane Guest

  10. #10

    Default Re: sh: line 1: /bin/cp: Argument list too long

    2006-02-6, 16:24(-08), Doug Freyburger: 
    >
    > There's no way the number of files is the same as before. Suspect
    > all you want about the system, but this is a standard topic to be
    > learned by SAGE Novice level sysadmins for them to get to
    > Junior level. Extremely standard UNIX toolsmithing and
    > mentorship stuff. Should be covered by your mentor in the first
    > year of an well organized apprenticeship program.
    >
    > Command lines are limited by the shell buffer.[/ref]

    No, they are not, there's no shell I know of with such
    limitations (though I've been told some old csh implementations
    might have had one).

    The limitation is on the execve system call. That's a system
    limitation, not a shell's.
     

    find's default output format is unfortunately not compatible
    with xargs expected input format (unless you post process find
    output format or use the GNU/BSD -print0/xargs -0 non standard
    options).

    --
    Stéphane
    Stephane Guest

  11. #11

    Default Re: sh: line 1: /bin/cp: Argument list too long

    Stephane CHAZELAS wrote: 

    >
    > No, they are not, there's no shell I know of with such
    > limitations (though I've been told some old csh implementations
    > might have had one).
    >
    > The limitation is on the execve system call. That's a system
    > limitation, not a shell's.[/ref]

    Good correction. The difference isn't visible but design
    issues like that matter.
     
    >
    > find's default output format is unfortunately not compatible
    > with xargs expected input format (unless you post process find
    > output format or use the GNU/BSD -print0/xargs -0 non standard
    > options).[/ref]

    Some man pages for xargs claim that the string "{}" gets
    substituted with the arguments from standard in, but it doesn't
    work on Solaris, HP-UX or AIX that I've tried. What's up with
    that? Why is it in the man page but doesn't work? I've tried
    blaskslash escapes, ticks and double quotes all to no avail:

    find . -name stuff -print | xargs cp "{}" target

    I've gone so far as to write a script that detects how many args
    are on its command line and tacks stuff onto the end, but that
    defeats the purpose of using xargs with the bracket subs. It's
    also overkill compared to using the -exec clause since it doesn't
    run fewer images the way xargs normally should.

    Doug Guest

  12. #12

    Default Re: sh: line 1: /bin/cp: Argument list too long

    On 7 Feb 2006 07:40:38 -0800, Doug Freyburger wrote:
    [...] 
    >
    > Some man pages for xargs claim that the string "{}" gets
    > substituted with the arguments from standard in, but it doesn't
    > work on Solaris, HP-UX or AIX that I've tried. What's up with
    > that? Why is it in the man page but doesn't work? I've tried
    > blaskslash escapes, ticks and double quotes all to no avail:
    >
    > find . -name stuff -print | xargs cp "{}" target[/ref]
    [...]

    What I was thinking of is that, xargs expect as input a blank
    separated list of arguments where the single and double quotes
    and the backslash serve as escaping each other. While find
    output is just newline separated list of file paths. find
    doesn't transform (escape) the blank characters or quotes or
    backslashes that are in the file names.

    More over, you don't need xargs to pipe the output of find into
    as find does have the feature that breaks a list of arguments to
    work around the limitation of execve (at least Unix compliant
    implementations of find).

    find . -name stuff -type f -exec sh -c '
    exec cp -- "$" target' inline {} +

    --
    Stephane
    Stephane Guest

  13. #13

    Default Re: sh: line 1: /bin/cp: Argument list too long

    Stephane Chazelas <fr> wrote: 

    When I look at writing something using 'find -exec ...' versus
    'find ... | xargs', I evaluate the choices according to the following
    tradeoffs / scenarios:

    1. find ... | xargs ... will break on whitespace embedded in
    filenames. Only use if you are certain that it won't be a
    problem; one-offs where the list has been checked, or if
    the list is otherwise sanitized.

    2. That limitation can be avoided with 'find ... -print0 | xargs -0',
    but it's non-portable.

    3. find ... -exec ... {} ; avoids both of the above problems, but
    has to fork/exec for each found file, which can add a lot of
    overhead.

    I know that the above isn't a news flash to the regulars, but usually
    one doesn't see the safety/portablility/speed tradeoff explicitly
    mentioned. As is often the case, it requires a certain amount of
    judgement to choose which is the best approach for each individual
    problem.


    Mike

    --
    Michael Zawrotny
    Institute of Molecular Biophysics
    Florida State University | email: fsu.edu
    Tallahassee, FL 32306-4380 | phone: (850) 644-0069
    Michael Guest

  14. #14

    Default Re: sh: line 1: /bin/cp: Argument list too long

    In article <googlegroups.com>,
    "Doug Freyburger" <com> wrote:
     

    Check the man pages again, it probably says that you have to use a
    particular option to xargs to make it do that.

    --
    Barry Margolin, mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***
    Barry Guest

  15. #15

    Default Re: sh: line 1: /bin/cp: Argument list too long

    On 7 Feb 2006 21:56:21 GMT, Michael Zawrotny wrote:
    [...] 
    >
    > When I look at writing something using 'find -exec ...' versus
    > 'find ... | xargs', I evaluate the choices according to the following
    > tradeoffs / scenarios:
    >
    > 1. find ... | xargs ... will break on whitespace embedded in
    > filenames. Only use if you are certain that it won't be a
    > problem; one-offs where the list has been checked, or if
    > the list is otherwise sanitized.
    >
    > 2. That limitation can be avoided with 'find ... -print0 | xargs -0',
    > but it's non-portable.
    >
    > 3. find ... -exec ... {} ; avoids both of the above problems, but
    > has to fork/exec for each found file, which can add a lot of
    > overhead.
    >
    > I know that the above isn't a news flash to the regulars, but usually
    > one doesn't see the safety/portablility/speed tradeoff explicitly
    > mentioned. As is often the case, it requires a certain amount of
    > judgement to choose which is the best approach for each individual
    > problem.[/ref]
    [...]

    As you may have noticed, I didn't use -exec {} \; but -exec {} +
    whose only speed tradeof wrt find | xargs is that find and the
    execed command don't run concurrently (find won't continue
    running until it fills the pipe buffer while the cmd is running
    as in the find+xargs way, that won't be necessarily be slower
    though if the cmd accesses the fs which it is likely to do). But
    as few commands as possible will be run as with xargs.

    "-exec ... {} +" is in every POSIX conformant and Unix conformant
    version of find.

    --
    Stephane
    Stephane Guest

  16. #16

    Default Re: sh: line 1: /bin/cp: Argument list too long

    Stephane Chazelas <fr> wrote: 
    >
    > As you may have noticed, I didn't use -exec {} \; but -exec {} +
    > whose only speed tradeof wrt find | xargs is that find and the
    > execed command don't run concurrently (find won't continue
    > running until it fills the pipe buffer while the cmd is running
    > as in the find+xargs way, that won't be necessarily be slower
    > though if the cmd accesses the fs which it is likely to do). But
    > as few commands as possible will be run as with xargs.
    >
    > "-exec ... {} +" is in every POSIX conformant and Unix conformant
    > version of find.[/ref]

    Nope, I missed that. In fact, it's the first time I can recall seeing
    "find -exec ... +". I'll have to remember that.


    Mike

    --
    Michael Zawrotny
    Institute of Molecular Biophysics
    Florida State University | email: fsu.edu
    Tallahassee, FL 32306-4380 | phone: (850) 644-0069
    Michael Guest

Similar Threads

  1. grep argument list too long...how to get around it?
    By Steve Grazzini in forum PERL Beginners
    Replies: 9
    Last Post: November 1st, 10:05 PM
  2. Replies: 2
    Last Post: April 2nd, 04:32 PM
  3. ls *.txt gives "bash: /bin/ls: Argument list too long" !?
    By Rob Baxter in forum Linux / Unix Administration
    Replies: 16
    Last Post: December 31st, 07:10 AM
  4. Replies: 8
    Last Post: September 20th, 07:59 PM
  5. How to use Command line argument --help or --debug ???
    By magnum_mentor in forum PERL Beginners
    Replies: 0
    Last Post: August 23rd, 01:17 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139