grep argument list too long...how to get around it?

[Steve Grazzini]

On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:

> Are you sure about using ls? We have directory here that has several
> thousand files in it and when doing an ls *.whatever-extension we always
> get an "argument list too long".
>
> Any idea what the actual file limit is for grep?

It's a system limit (not specific to grep) based on the size-in-bytes
of the argument list (not the number of items).

--
Steve

[Gary Egleton]

What version of unix are you using?

It was fixed in a patch on HPUX

> -----Original Message-----
> From: Kevin Old [SMTP:kold@kold.homelinux.com]
> Sent: 09 October 2003 17:22
> To: Dan Muey
> Cc: [email]beginners@perl.org[/email]
> Subject: RE: grep argument list too long...how to get around it?
>
> On Thu, 2003-10-09 at 12:13, Dan Muey wrote:

> > > Hello everyone,

> >
> > Howdy
> >

> > >
> > > We use the Barracuda Spam appliance (barracudanetworks.com)
> > > to filter our spam and their web based interface is written
> > > in Perl. They have a form that allows the user to search
> > > messages for key words. Evidentally it stores the each
> > > message in a file in a directory and when trying to search
> > > several hundred thousand messages for a word the response back
> > > is:
> > >
> > > egrep: argument list too long
> > >

> >
> > If I was trying to grep a zillion files at once and it wouldn't
> > let me I'd probably grep them one at a time.
> > For instance via the backtivck execution You might have:
> >
> > my @matchedfiles = qx(cat `ls /files/` |grep $string);
> > # a really bad way to do this but for example's sake...
> >
> > You could do:
> >
> > for(`ls /files/`) {
> > if(`cat $_ |grep $string`) { push(@matchedfiles,$_); }
> > }

>
> Are you sure about using ls? We have directory here that has several
> thousand files in it and when doing an ls *.whatever-extension we always
> get an "argument list too long".
>
> Any idea what the actual file limit is for grep?
>

> >
> > Then you are only greping one file at a time instead of a list of too

> many.

> > Of course what would be better is to use the readdir() functions to list

> the files

> > and open() and grep() combo to grep the contents. But the same principle

> applies.

> >
> > Just make sure the barracuda folks says thanks for fixing their problem

> :)
>
> Yeah, we're hoping for a few months of service for free.....:)
>
> This was also a personal quest to find the answer for myself. So either
> way I win.
>
> Thanks for your help,
> Kevin
> --
> Kevin Old <kold@kold.homelinux.com>
>
>
> --
> To unsubscribe, e-mail: [email]beginners-unsubscribe@perl.org[/email]
> For additional commands, e-mail: [email]beginners-help@perl.org[/email]

------------------------------------------------------------
The information contained in or attached to this email is
intended only for the use of the individual or entity to
which it is addressed. If you are not the intended
recipient, or a person responsible for delivering it to the
intended recipient, you are not authorised to and must not
disclose, copy, distribute, or retain this message or any
part of it. It may contain information which is confidential
and/or covered by legal professional or other privilege (or
other rules or laws with similar effect in jurisdictions
outside England and Wales).
The views expressed in this email are not necessarily the
views of Centrica plc, and the company, its directors,
officers or employees make no representation or accept any
liability for its accuracy or completeness unless expressly
stated to the contrary.

[Kevin Old]

On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:

> On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:

> > Are you sure about using ls? We have directory here that has several
> > thousand files in it and when doing an ls *.whatever-extension we always
> > get an "argument list too long".
> >
> > Any idea what the actual file limit is for grep?

>
> It's a system limit (not specific to grep) based on the size-in-bytes
> of the argument list (not the number of items).

So it's related to my ulimit open files?

Currently it's set to 1024. Does that mean I'm limited to grepping an
argument list that is less than 1024?

Thanks,
Kevin

--
Kevin Old <kold@kold.homelinux.com>

[Steve Grazzini]

On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:

> On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:

> > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:

> > > Are you sure about using ls? We have directory here that has several
> > > thousand files in it and when doing an ls *.whatever-extension we always
> > > get an "argument list too long".
> > >
> > > Any idea what the actual file limit is for grep?

> >
> > It's a system limit (not specific to grep) based on the size-in-bytes
> > of the argument list (not the number of items).

>
> So it's related to my ulimit open files?

No. (It's ARG_MAX...)

--
Steve

[Steve Grazzini]

On Fri, Oct 10, 2003 at 12:37:02PM -0400, Kevin Old wrote:

> On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:

> > No. (It's ARG_MAX...)

>
> I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
> "unlimited"? If not, what is the default?

It's not an environment variable. (Check limits.h.)

And remember, that's not the *number* of files/arguments/whatever.
ARG_MAX is the total *memory* that can be used for a new process's
environment and argument list.

% export FOO=$( perl -le 'print "x" x 2**17' )
% ls
bash: /bin/ls: Argument list too long

--
Steve

[Kevin Old]

On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:

> On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:

> > On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:

> > > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
> > > > Are you sure about using ls? We have directory here that has several
> > > > thousand files in it and when doing an ls *.whatever-extension we always
> > > > get an "argument list too long".
> > > >
> > > > Any idea what the actual file limit is for grep?
> > >
> > > It's a system limit (not specific to grep) based on the size-in-bytes
> > > of the argument list (not the number of items).

> >
> > So it's related to my ulimit open files?

>
> No. (It's ARG_MAX...)

I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
"unlimited"? If not, what is the default?

Is it the same on other *nix systems?

Thanks,
Kevin

--
Kevin Old <kold@kold.homelinux.com>

[Alexandru Colea]

You can put arguments in a file and then open and parse the file inside
perl.

-Alex C.

-----Original Message-----
From: Steve Grazzini [mailto:grazz@pobox.com]
Sent: Friday, October 10, 2003 12:29 PM
To: Kevin Old
Cc: Dan Muey; [email]beginners@perl.org[/email]
Subject: Re: grep argument list too long...how to get around it?

On Fri, Oct 10, 2003 at 12:37:02PM -0400, Kevin Old wrote:

> On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:

> > No. (It's ARG_MAX...)

>
> I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
> "unlimited"? If not, what is the default?

It's not an environment variable. (Check limits.h.)

And remember, that's not the *number* of files/arguments/whatever.
ARG_MAX is the total *memory* that can be used for a new process's
environment and argument list.

% export FOO=$( perl -le 'print "x" x 2**17' )
% ls
bash: /bin/ls: Argument list too long

--
Steve

--
To unsubscribe, e-mail: [email]beginners-unsubscribe@perl.org[/email]
For additional commands, e-mail: [email]beginners-help@perl.org[/email]

[David]

Kevin Old wrote:

> On Fri, 2003-10-10 at 11:46, Steve Grazzini wrote:

>> On Fri, Oct 10, 2003 at 09:35:25AM -0400, Kevin Old wrote:

>> > On Fri, 2003-10-10 at 02:44, Steve Grazzini wrote:
>> > > On Thu, Oct 09, 2003 at 12:21:57PM -0400, Kevin Old wrote:
>> > > > Are you sure about using ls? We have directory here that has
>> > > > several thousand files in it and when doing an ls
>> > > > *.whatever-extension we always get an "argument list too long".
>> > > >
>> > > > Any idea what the actual file limit is for grep?
>> > >
>> > > It's a system limit (not specific to grep) based on the size-in-bytes
>> > > of the argument list (not the number of items).
>> >
>> > So it's related to my ulimit open files?

>>
>> No. (It's ARG_MAX...)

>
> I'm running Mandrake 9.0 and my ARG_MAX is not set, so is it
> "unlimited"? If not, what is the default?

for a linux box, this number is hidden inside:

[panda]$ grep ARG_MAX /usr/include/linux/limits.h
#define ARG_MAX 131072 /* # bytes of args + environ for exec() */
[panda]$

>
> Is it the same on other *nix systems?
>

no. it's not always the same for all *nix system. openBSD:

[tiger]# grep ARG_MAX /usr/include/sys/syslimits.h
#define ARG_MAX (256 * 1024) /* max bytes for an exec function */
[tiger]#

this is implementation dependent. if you run into this limit, consider using
xargs if your system has it. man xargs.

david
--
$_=q,015001450154015401570040016701570162015401440 041,,*,=*|=*_,split+local$";
map{~$_&1&&{$,<<=1,$#.=qq~\x63\x68\x72\x28@_[$_..$||3])=>~}}0..s~.~~g-1;*_=*#,

goto=>print+eval

[Darin McBride]

Luke Bakken wrote:

> Rather than calling
>
> egrep REGEX really long list of files ...
>
> they should be calling (ksh here, use "echo" instead of "print" for other
> shells):
>
> print really long list of files | xargs egrep REGEX

IMO, they should be using glob and coding the grep themselves in Perl.
In addition to items mentioned earlier (faster, safer), it's also more
portable. Imagine that the code needed to be moved from BSD or Linux
or other unix-like OS to, say, Mac OS X, or worse, Windows.
Portability can be a concern as much as anything else (my #1 concern as
my code must run on about a dozen different OS's).

And, of course, it'll also work on directories with more files than the
shell can handle on its commandline. ;->

[Unregistered]

find . -exec grep {} \; -print