Perl - HowTo operating with large file?

[Juris]

I have one small problem!
HowTo read from large text file text in binary mode?

if i want read all file, i use this code:

my (@LOG_FILE);
open (FL, "/var/log/maillog");
@LOG_FILE=<FL>;
close (FL);
#After this code execution file contents stored in array @LOG_FILE
@LOG_FILE=grep /$something/, @LOG_FILE

I not want read all file string by struing - it's so slowly, if file is
large and contains more than 1000000 records!
I need read each 500000 bytes, but how?
Please, help!


--
With best regards,
Juris

[Paul William]

why not read the file line by line and then simply match each line with
/$something/, disgarding any lines which do not match /$something/.

If you wanted to could push all matching lines into and array.

Cheers

Paul


On Fri, 2003-09-26 at 18:52, Juris wrote:

> I have one small problem!
> HowTo read from large text file text in binary mode?
>
> if i want read all file, i use this code:
>
> my (@LOG_FILE);
> open (FL, "/var/log/maillog");
> @LOG_FILE=<FL>;
> close (FL);
> #After this code execution file contents stored in array @LOG_FILE
> @LOG_FILE=grep /$something/, @LOG_FILE
>
> I not want read all file string by struing - it's so slowly, if file is
> large and contains more than 1000000 records!
> I need read each 500000 bytes, but how?
> Please, help!
>
>
> --
> With best regards,
> Juris

[Ramprasad A Padmanabhan]

Juris wrote:

> I have one small problem!
> HowTo read from large text file text in binary mode?
>
> if i want read all file, i use this code:
>
> my (@LOG_FILE);
> open (FL, "/var/log/maillog");
> @LOG_FILE=<FL>;
> close (FL);
> #After this code execution file contents stored in array @LOG_FILE
> @LOG_FILE=grep /$something/, @LOG_FILE
>
> I not want read all file string by struing - it's so slowly, if file is
> large and contains more than 1000000 records!
> I need read each 500000 bytes, but how?
> Please, help!
>
>

When you are opening big files never do
@array = <FILE>
This essentially reads the entire file into an array and is very
expensive on memory.

you could do something like
while(<FILE>){
push @arr , $_ if(/$something/);
}

But IMHO this still that may not be the best way.

What I would do is


system("grep $something $filename > $tempfile");
# *Nothing* beats gnu grep when you parse large file

open(FILE,$tempfile);
# Now if you really want the lines in an array
@lines = <FILE>

Ram

[Wiggins D'Anconia]

------------------------------------------------
On Fri, 26 Sep 2003 20:13:21 +0530, Ramprasad A Padmanabhan <ramprasad@netcore.co.in> wrote:

> Juris wrote:

> > I have one small problem!
> > HowTo read from large text file text in binary mode?
> >
> > if i want read all file, i use this code:
> >
> > my (@LOG_FILE);
> > open (FL, "/var/log/maillog");
> > @LOG_FILE=<FL>;
> > close (FL);
> > #After this code execution file contents stored in array @LOG_FILE
> > @LOG_FILE=grep /$something/, @LOG_FILE
> >
> > I not want read all file string by struing - it's so slowly, if file is
> > large and contains more than 1000000 records!
> > I need read each 500000 bytes, but how?
> > Please, help!
> >
> >

>
>
> When you are opening big files never do
> @array = <FILE>
> This essentially reads the entire file into an array and is very
> expensive on memory.
>
> you could do something like
> while(<FILE>){
> push @arr , $_ if(/$something/);
> }
>
> But IMHO this still that may not be the best way.
>
> What I would do is
>
>
> system("grep $something $filename > $tempfile");
> # *Nothing* beats gnu grep when you parse large file
>
> open(FILE,$tempfile);
> # Now if you really want the lines in an array
> @lines = <FILE>

This is NOT production sufficient code, either please handle it fully or disclaim it by stating so. You don't use a full path (or suggest to), you don't check the return value of system, there are all kinds of issues with respect to using '>' to write to a file (permissions, disk space, etc.) Security issues, naturally checking 'open' for success, removing core files if they are generated, "cleaning" the variables for taint safety, etc, etc, etc. Shelling out in this manner is quick and dirty (and has its places I admit) but on a beginners list please disclaim it as so.

The OP may want to check out the read function, use it carefully....

perldoc -f read

[url]http://danconia.org[/url]

[Juris]

Hi!

Thanks, your solution is one way how solve this problem! I found other ...

sub remove_ary_dupes{
my @in=@_;
#Define hash
undef %saw;
@saw{@in} = ();
#Hash owerwrite automaticly duplicated values!
my @out = sort keys %saw; # remove sort if undesired
return @out;
}

sub grep_log_file{
my @sparam=@_;
#if (undef $sparam[0]) {print "No records found with @ARGV\n"; exit 0}
my ($first, $match);
my $block_size=20480;
my ($tmp_block, $pos_corect,@ary_match, @res_ary);

open(F, "$LOG_FILE") or die "Error opening $LOG_FILE: $!";

#Read LOG file block by block
while (read(F,$first,$block_size)) {
$end_pos=rindex($first,"\n");
# $first=~/(.*)\n(.*)$/; #Not used, because so slowly for long lines!
# Compare read line length with default block size!
# If not equal, its last block in cycle
if (length($first) ne $block_size) {
$match=$first; $match = $tmp_block . $first;
$tmp_block=""; $pos_corect=0;
} else {
$match=substr($first,0, $end_pos); $match=$tmp_block .
$match ;
$pos_corect=$block_size-$end_pos;
$tmp_block=substr($first,$end_pos,$block_size-$end_pos);
}
my @log_lines=split("\n",$match);
##Grep all lines
@log_lines=remove_ary_dupes(@log_lines); #Remove duplicated log
lines!
foreach my $j (@sparam) {
push @ary_match, (grep /$j/, @log_lines);
}
#push @res_ary, @ary_match;
}
close (F);
return @ary_match;
}

Try this code, if have similar problem!

On Fri, 26 Sep 2003 20:13:21 +0530, Ramprasad A Padmanabhan
<ramprasad@netcore.co.in> wrote:

> Juris wrote:

>> I have one small problem!
>> HowTo read from large text file text in binary mode?
>>
>> if i want read all file, i use this code:
>>
>> my (@LOG_FILE);
>> open (FL, "/var/log/maillog");
>> @LOG_FILE=<FL>;
>> close (FL);
>> #After this code execution file contents stored in array @LOG_FILE
>> @LOG_FILE=grep /$something/, @LOG_FILE
>>
>> I not want read all file string by struing - it's so slowly, if file is
>> large and contains more than 1000000 records!
>> I need read each 500000 bytes, but how?
>> Please, help!
>>
>>

>
>
> When you are opening big files never do
> @array = <FILE>
> This essentially reads the entire file into an array and is very
> expensive on memory.
>
> you could do something like
> while(<FILE>){
> push @arr , $_ if(/$something/);
> }
>
> But IMHO this still that may not be the best way.
>
> What I would do is
>
>
> system("grep $something $filename > $tempfile");
> # *Nothing* beats gnu grep when you parse large file
>
> open(FILE,$tempfile);
> # Now if you really want the lines in an array
> @lines = <FILE>
>
> Ram

--
Using M2, Opera's revolutionary e-mail client: [url]http://www.opera.com/m2/[/url]

[Ramprasad A Padmanabhan]

On Mon, 2003-09-29 at 16:07, Juris wrote:

> Hi!
>
> Thanks, your solution is one way how solve this problem! I found other ...
>

Gr8 That is the spirit of perl. There is always a different , slightly
better way of doing things in perl.

Ram

[ruqui]

A better solution is to use the Tie::File module.

This url explains how to use it: http://www.perlhowto.com/modify_a_file_in_place