Professional Web Applications Themes

grep argument list too long...how to get around it? - PERL Beginners

On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote: > Are you sure about using ls? We have directory here that has several > thousand files in it and when doing an ls *.whatever-extension we always > get an "argument list too long". > > Any idea what the actual file limit is for grep? It's a system limit (not specific to grep) based on the size-in-bytes of the argument list (not the number of items). -- Steve...

  1. #1

    Default Re: grep argument list too long...how to get around it?

    On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
    > Are you sure about using ls? We have directory here that has several
    > thousand files in it and when doing an ls *.whatever-extension we always
    > get an "argument list too long".
    >
    > Any idea what the actual file limit is for grep?
    It's a system limit (not specific to grep) based on the size-in-bytes
    of the argument list (not the number of items).

    --
    Steve
    Steve Grazzini Guest

  2. #2

    Default RE: grep argument list too long...how to get around it?

    What version of unix are you using?

    It was fixed in a patch on HPUX
    > -----Original Message-----
    > From: Kevin Old [SMTP:koldkold.homelinux.com]
    > Sent: 09 October 2003 17:22
    > To: Dan Muey
    > Cc: [email]beginnersperl.org[/email]
    > Subject: RE: grep argument list too long...how to get around it?
    >
    > On Thu, 2003-10-09 at 12:13, Dan Muey wrote:
    > > > Hello everyone,
    > >
    > > Howdy
    > >
    > > >
    > > > We use the Barracuda Spam appliance (barracudanetworks.com)
    > > > to filter our spam and their web based interface is written
    > > > in Perl. They have a form that allows the user to search
    > > > messages for key words. Evidentally it stores the each
    > > > message in a file in a directory and when trying to search
    > > > several hundred thousand messages for a word the response back
    > > > is:
    > > >
    > > > egrep: argument list too long
    > > >
    > >
    > > If I was trying to grep a zillion files at once and it wouldn't
    > > let me I'd probably grep them one at a time.
    > > For instance via the backtivck execution You might have:
    > >
    > > my matchedfiles = qx(cat `ls /files/` |grep $string);
    > > # a really bad way to do this but for example's sake...
    > >
    > > You could do:
    > >
    > > for(`ls /files/`) {
    > > if(`cat $_ |grep $string`) { push(matchedfiles,$_); }
    > > }
    >
    > Are you sure about using ls? We have directory here that has several
    > thousand files in it and when doing an ls *.whatever-extension we always
    > get an "argument list too long".
    >
    > Any idea what the actual file limit is for grep?
    >
    > >
    > > Then you are only greping one file at a time instead of a list of too
    > many.
    > > Of course what would be better is to use the readdir() functions to list
    > the files
    > > and open() and grep() combo to grep the contents. But the same principle
    > applies.
    > >
    > > Just make sure the barracuda folks says thanks for fixing their problem
    > :)
    >
    > Yeah, we're hoping for a few months of service for free.....:)
    >
    > This was also a personal quest to find the answer for myself. So either
    > way I win.
    >
    > Thanks for your help,
    > Kevin
    > --
    > Kevin Old <koldkold.homelinux.com>
    >
    >
    > --
    > To unsubscribe, e-mail: [email]beginners-unsubscribeperl.org[/email]
    > For additional commands, e-mail: [email]beginners-helpperl.org[/email]

    ------------------------------------------------------------
    The information contained in or attached to this email is
    intended only for the use of the individual or entity to
    which it is addressed. If you are not the intended
    recipient, or a person responsible for delivering it to the
    intended recipient, you are not authorised to and must not
    disclose, copy, distribute, or retain this message or any
    part of it. It may contain information which is confidential
    and/or covered by legal professional or other privilege (or
    other rules or laws with similar effect in jurisdictions
    outside England and Wales).
    The views expressed in this email are not necessarily the
    views of Centrica plc, and the company, its directors,
    officers or employees make no representation or accept any
    liability for its accuracy or completeness unless expressly
    stated to the contrary.

    Gary Egleton Guest

  3. #3

    Default Re: grep argument list too long...how to get around it?

    On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:
    > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
    > > Are you sure about using ls? We have directory here that has several
    > > thousand files in it and when doing an ls *.whatever-extension we always
    > > get an "argument list too long".
    > >
    > > Any idea what the actual file limit is for grep?
    >
    > It's a system limit (not specific to grep) based on the size-in-bytes
    > of the argument list (not the number of items).
    So it's related to my ulimit open files?

    Currently it's set to 1024. Does that mean I'm limited to grepping an
    argument list that is less than 1024?

    Thanks,
    Kevin

    --
    Kevin Old <koldkold.homelinux.com>

    Kevin Old Guest

  4. #4

    Default Re: grep argument list too long...how to get around it?

    On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:
    > On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:
    > > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
    > > > Are you sure about using ls? We have directory here that has several
    > > > thousand files in it and when doing an ls *.whatever-extension we always
    > > > get an "argument list too long".
    > > >
    > > > Any idea what the actual file limit is for grep?
    > >
    > > It's a system limit (not specific to grep) based on the size-in-bytes
    > > of the argument list (not the number of items).
    >
    > So it's related to my ulimit open files?
    No. (It's ARG_MAX...)

    --
    Steve
    Steve Grazzini Guest

  5. #5

    Default Re: grep argument list too long...how to get around it?

    On Fri, Oct 10, 2003 at 12:37:02PM -0400, Kevin Old wrote:
    > On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:
    > > No. (It's ARG_MAX...)
    >
    > I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
    > "unlimited"? If not, what is the default?
    It's not an environment variable. (Check limits.h.)

    And remember, that's not the *number* of files/arguments/whatever.
    ARG_MAX is the total *memory* that can be used for a new process's
    environment and argument list.

    % export FOO=$( perl -le 'print "x" x 2**17' )
    % ls
    bash: /bin/ls: Argument list too long

    --
    Steve
    Steve Grazzini Guest

  6. #6

    Default Re: grep argument list too long...how to get around it?

    On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:
    > On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:
    > > On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:
    > > > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
    > > > > Are you sure about using ls? We have directory here that has several
    > > > > thousand files in it and when doing an ls *.whatever-extension we always
    > > > > get an "argument list too long".
    > > > >
    > > > > Any idea what the actual file limit is for grep?
    > > >
    > > > It's a system limit (not specific to grep) based on the size-in-bytes
    > > > of the argument list (not the number of items).
    > >
    > > So it's related to my ulimit open files?
    >
    > No. (It's ARG_MAX...)
    I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
    "unlimited"? If not, what is the default?

    Is it the same on other *nix systems?

    Thanks,
    Kevin

    --
    Kevin Old <koldkold.homelinux.com>

    Kevin Old Guest

  7. #7

    Default RE: grep argument list too long...how to get around it?

    You can put arguments in a file and then open and p the file inside
    perl.

    -Alex C.

    -----Original Message-----
    From: Steve Grazzini [mailto:grazzpobox.com]
    Sent: Friday, October 10, 2003 12:29 PM
    To: Kevin Old
    Cc: Dan Muey; [email]beginnersperl.org[/email]
    Subject: Re: grep argument list too long...how to get around it?

    On Fri, Oct 10, 2003 at 12:37:02PM -0400, Kevin Old wrote:
    > On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:
    > > No. (It's ARG_MAX...)
    >
    > I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
    > "unlimited"? If not, what is the default?
    It's not an environment variable. (Check limits.h.)

    And remember, that's not the *number* of files/arguments/whatever.
    ARG_MAX is the total *memory* that can be used for a new process's
    environment and argument list.

    % export FOO=$( perl -le 'print "x" x 2**17' )
    % ls
    bash: /bin/ls: Argument list too long

    --
    Steve

    --
    To unsubscribe, e-mail: [email]beginners-unsubscribeperl.org[/email]
    For additional commands, e-mail: [email]beginners-helpperl.org[/email]


    Alexandru Colea Guest

  8. #8

    Default Re: grep argument list too long...how to get around it?

    Kevin Old wrote:
    > On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:
    >> On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:
    >> > On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:
    >> > > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
    >> > > > Are you sure about using ls? We have directory here that has
    >> > > > several thousand files in it and when doing an ls
    >> > > > *.whatever-extension we always get an "argument list too long".
    >> > > >
    >> > > > Any idea what the actual file limit is for grep?
    >> > >
    >> > > It's a system limit (not specific to grep) based on the size-in-bytes
    >> > > of the argument list (not the number of items).
    >> >
    >> > So it's related to my ulimit open files?
    >>
    >> No. (It's ARG_MAX...)
    >
    > I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
    > "unlimited"? If not, what is the default?
    for a linux box, this number is hidden inside:

    [panda]$ grep ARG_MAX /usr/include/linux/limits.h
    #define ARG_MAX 131072 /* # bytes of args + environ for exec() */
    [panda]$
    >
    > Is it the same on other *nix systems?
    >
    no. it's not always the same for all *nix system. openBSD:

    [tiger]# grep ARG_MAX /usr/include/sys/syslimits.h
    #define ARG_MAX (256 * 1024) /* max bytes for an exec function */
    [tiger]#

    this is implementation dependent. if you run into this limit, consider using
    xargs if your system has it. man xargs.

    david
    --
    $_=q,015001450154015401570040016701570162015401440 041,,*,=*|=*_,split+local$";
    map{~$_&1&&{$,<<=1,$#.=qq~\x63\x68\x72\x28_[$_..$||3])=>~}}0..s~.~~g-1;*_=*#,

    goto=>print+eval
    David Guest

  9. #9

    Default RE: grep argument list too long...how to get around it?

    Luke Bakken wrote:
    > Rather than calling
    >
    > egrep REGEX really long list of files ...
    >
    > they should be calling (ksh here, use "echo" instead of "print" for other
    > shells):
    >
    > print really long list of files | xargs egrep REGEX
    IMO, they should be using glob and coding the grep themselves in Perl.
    In addition to items mentioned earlier (faster, safer), it's also more
    portable. Imagine that the code needed to be moved from BSD or Linux
    or other unix-like OS to, say, Mac OS X, or worse, Windows.
    Portability can be a concern as much as anything else (my #1 concern as
    my code must run on about a dozen different OS's).

    And, of course, it'll also work on directories with more files than the
    shell can handle on its commandline. ;->
    Darin McBride Guest

  10. #10

    Default Re: grep argument list too long...how to get around it?

    find . -exec grep {} \; -print
    Unregistered Guest

Similar Threads

  1. Long list of items in my datagrid, editing is a pain!!
    By David Lozzi in forum ASP.NET Data Grid Control
    Replies: 4
    Last Post: November 9th, 01:45 AM
  2. Problem: 32 Argument function parameter list
    By Bradley D. Snobar in forum PostgreSQL / PGSQL
    Replies: 0
    Last Post: January 31st, 04:06 AM
  3. ls *.txt gives "bash: /bin/ls: Argument list too long" !?
    By Rob Baxter in forum Linux / Unix Administration
    Replies: 16
    Last Post: December 31st, 07:10 AM
  4. grep'd list as loop parameter
    By ed in forum PERL Miscellaneous
    Replies: 7
    Last Post: September 3rd, 07:40 PM
  5. long list of tips (alternatives)...
    By Fraggy in forum Macromedia Director Lingo
    Replies: 1
    Last Post: July 13th, 01:20 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139