multiline matching

[Jose Malacara]

Can someone explain to me how to do multiline matching? I am trying to extract three consecutive lines from a datafile containing multiple records like this:

Name: Bob
City: Austin
State: Texas
Address: 123 Whatever
Age: 46

Name: Jose
City: Denver
State: Colorado
Address: 118 Mystreet
Age: 28



This is what I have so far, but it doesn't seem to work:

#!/usr/bin/perl -w
open FILE, "<file1" or die "Can't open file!\n";
while (<FILE>){
if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
print "Match found!\n"; # ideally, I want to print the the lines found
}
}
close FILE;

But for some reason, it doesn't seem to like the (\n)'s in the regex. Any help would be appreciated!

This is what I would like to return:

Name: Bob
City: Austin
State: Texas

Name: Jose
City: Denver
State: Colorado


Thanks in advance,
Jose

[Jeff Westman]

Jose Malacara <josem@thrifty.net> wrote:

> Can someone explain to me how to do multiline matching? I am
> trying to extract three consecutive lines from a datafile
> containing multiple records like this:

Check out

perldoc perlre

What you should do is look at the 'm' (multiple line) option.


HTH

JW

> Name: Bob
> City: Austin
> State: Texas
> Address: 123 Whatever
> Age: 46
>
> Name: Jose
> City: Denver
> State: Colorado
> Address: 118 Mystreet
> Age: 28
>
>
>
> This is what I have so far, but it doesn't seem to work:
>
> #!/usr/bin/perl -w
> open FILE, "<file1" or die "Can't open file!\n";
> while (<FILE>){
> if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
> print "Match found!\n"; # ideally, I want to print the the
> lines found
> }
> }
> close FILE;
>
> But for some reason, it doesn't seem to like the (\n)'s in the
> regex. Any help would be appreciated!
>
> This is what I would like to return:
>
> Name: Bob
> City: Austin
> State: Texas
>
> Name: Jose
> City: Denver
> State: Colorado
>
>
> Thanks in advance,
> Jose

__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
[url]http://hotjobs.sweepstakes.yahoo.com/signingbonus[/url]

[Kenton Brede]

On Wed, Jan 14, 2004 at 06:22:58PM -0700, Jose Malacara wrote:

> Can someone explain to me how to do multiline matching? I am trying to extract three consecutive lines from a datafile containing multiple records like this:
>
> Name: Bob
> City: Austin
> State: Texas
> Address: 123 Whatever
> Age: 46
>
> Name: Jose
> City: Denver
> State: Colorado
> Address: 118 Mystreet
> Age: 28
>
>
>
> This is what I have so far, but it doesn't seem to work:
>
> #!/usr/bin/perl -w
> open FILE, "<file1" or die "Can't open file!\n";
> while (<FILE>){
> if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
> print "Match found!\n"; # ideally, I want to print the the lines found
> }
> }
> close FILE;
>
> But for some reason, it doesn't seem to like the (\n)'s in the regex. Any help would be appreciated!
>
> This is what I would like to return:
>
> Name: Bob
> City: Austin
> State: Texas
>
> Name: Jose
> City: Denver
> State: Colorado

I'm sure others will have better solutions but this works -

#!/usr/bin/perl
use warnings;
use strict;

while (<DATA>) {
print if /^Name:/;
print if /^City:/;
print "$_\n" if /^State:/;
}

__DATA__
Name: Bob
City: Austin
State: Texas
Address: 123 Whatever
Age: 46

Name: Jose
City: Denver
State: Colorado
Address: 118 Mystreet
Age: 28

hth,
Kent

--
"I am always doing that which I can not do,
in order that I may learn how to do it." --Pablo Picasso

[Jose Malacara]

Thanks, Jeff. I read over the reference, but I guess I am missing the
syntax as I cannot get it to work properly. Can you point me to any
examples?

I found this example, but am having trouble translating it to my while
loop:

perl -e '$_ = "{a\n{bb}\n{c\n\nc}\na\}"; m({.*})m; print $&'


Thanks again,
Jose

On Wed, 2004-01-14 at 18:37, Jeff Westman wrote:

> Jose Malacara <josem@thrifty.net> wrote:
>

> > Can someone explain to me how to do multiline matching? I am
> > trying to extract three consecutive lines from a datafile
> > containing multiple records like this:

>
> Check out
>
> perldoc perlre
>
> What you should do is look at the 'm' (multiple line) option.
>
>
> HTH
>
> JW
>
>

> > Name: Bob
> > City: Austin
> > State: Texas
> > Address: 123 Whatever
> > Age: 46
> >
> > Name: Jose
> > City: Denver
> > State: Colorado
> > Address: 118 Mystreet
> > Age: 28
> >
> >
> >
> > This is what I have so far, but it doesn't seem to work:
> >
> > #!/usr/bin/perl -w
> > open FILE, "<file1" or die "Can't open file!\n";
> > while (<FILE>){
> > if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
> > print "Match found!\n"; # ideally, I want to print the the
> > lines found
> > }
> > }
> > close FILE;
> >
> > But for some reason, it doesn't seem to like the (\n)'s in the
> > regex. Any help would be appreciated!
> >
> > This is what I would like to return:
> >
> > Name: Bob
> > City: Austin
> > State: Texas
> >
> > Name: Jose
> > City: Denver
> > State: Colorado
> >
> >
> > Thanks in advance,
> > Jose

>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
> [url]http://hotjobs.sweepstakes.yahoo.com/signingbonus[/url]

--
Jose Malacara
[email]josem@thrifty.net[/email]

[Daniel Staal]

--As off Wednesday, January 14, 2004 6:22 PM -0700, Jose Malacara is
alleged to have said:

> Can someone explain to me how to do multiline matching? I am trying
> to extract three consecutive lines from a datafile containing
> multiple records like this:
>
> Name: Jose
> City: Denver
> State: Colorado
> Address: 118 Mystreet
> Age: 28
>
>
>
> This is what I have so far, but it doesn't seem to work:
>
># !/usr/bin/perl -w
> open FILE, "<file1" or die "Can't open file!\n";
> while (<FILE>){
> if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
> print "Match found!\n"; # ideally, I want to print the the
> lines found }
> }
> close FILE;
>
> But for some reason, it doesn't seem to like the (\n)'s in the
> regex. Any help would be appreciated!

--As for the rest, it is mine.

You actually have two seperate problems here... First off, you
aren't reading more than one line at a time, since the readline
operator stops at a newline. Then, a regrex normally stops at a
newline too, unless you specifically tell it to continue past those.
( The 'm' operator, mentioned in other posts.)

The first problem is actually the bigger one: The obvious solution of
reading the whole file to an array/scalar first has preformance
problems, especially with large files. (Worst case: A file bigger
than avalible memory...)

Kent's solution neatly sidesteps that, if you don't care that the
program has no idea of which records go together. (His solution just
prints each record line, assuming they are in the correct order
already.) Otherwise you are going to need some sort of temporary
storage where you put the records back together, or read the file in
chunks.

Daniel T. Staal

---------------------------------------------------------------
This email copyright the author. Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes. This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

[James Edward Gray II]

On Jan 14, 2004, at 7:22 PM, Jose Malacara wrote:

> Can someone explain to me how to do multiline matching? I am trying to
> extract three consecutive lines from a datafile containing multiple
> records like this:
>
> Name: Bob
> City: Austin
> State: Texas
> Address: 123 Whatever
> Age: 46
>
> Name: Jose
> City: Denver
> State: Colorado
> Address: 118 Mystreet
> Age: 28
>
>
>
> This is what I have so far, but it doesn't seem to work:
>
> #!/usr/bin/perl -w
> open FILE, "<file1" or die "Can't open file!\n";
> while (<FILE>){
> if ( /^Name: (.*)\nCity: (.*)\nState: (.*)/) {
> print "Match found!\n"; # ideally, I want to print the the lines
> found
> }
> }
> close FILE;
>
> But for some reason, it doesn't seem to like the (\n)'s in the regex.
> Any help would be appreciated!
>
> This is what I would like to return:
>
> Name: Bob
> City: Austin
> State: Texas
>
> Name: Jose
> City: Denver
> State: Colorado

Since you've already been shown the super easy way, I'll dare to be a
little different:

#!/usr/bin/perl

use strict;
use warnings;

$/ = ''; # enter "paragraph" mode
while (<>) { # call with: perl script_name file1
my %contact = map { /^(\w+):\s*(.+)$/ } split /\n/, $_;
print "$_: $contact{$_}\n" foreach qw(Name City State);
print "\n"
}

__END__

The first way your were shown is probably a little easier, but this
method is probably better if you want to do anything more complicated
than simple printing, since you have the whole hash to play with. It's
a different way of thinking about the problem at least.

Good luck.

James

[Kenton Brede]

On Wed, Jan 14, 2004 at 09:57:51PM -0600, James Edward Gray II (james@grayproductions.net) wrote:

> On Jan 14, 2004, at 7:22 PM, Jose Malacara wrote:

<snip>

> Since you've already been shown the super easy way, I'll dare to be a
> little different:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> $/ = ''; # enter "paragraph" mode
> while (<>) { # call with: perl script_name file1
> my %contact = map { /^(\w+):\s*(.+)$/ } split /\n/, $_;
> print "$_: $contact{$_}\n" foreach qw(Name City State);
> print "\n"
> }
>
> __END__
>
> The first way your were shown is probably a little easier, but this
> method is probably better if you want to do anything more complicated
> than simple printing, since you have the whole hash to play with. It's
> a different way of thinking about the problem at least.

Thanks for posting this. My first thought was a hash, thinking in terms
of key-item but I couldn't figure out how to populate the hash. This
code will give me something to analyze. The "map" function looks like
voodoo to me:)
Kent

--
"I am always doing that which I can not do,
in order that I may learn how to do it." --Pablo Picasso

[Tim Johnson]

Just for the sake of showing another solution (even though I'm not doing any multi-line matching), how about:

##############################

use strict;
use warnings;
open(INFILE,"myfile.txt") || die "Couldn't open myfile.txt for writing!\n";
while(<INFILE>){
my %person;
$person{name} = $_;
$person{city} = <INFILE>;
$person{state} = <INFILE>;
$person{address} = <INFILE>;
$person{age} = <INFILE>;
#do whatever you want with your lines here
}

##############################

This gets you the whole record in a hash that goes out of scope after each record so that you don't have to slurp the file, but you have the flexibility to do whatever you want with the data. You can replace the city and state lines with just <INFILE>; on its own line if you don't need the data for anything. Of course I'm assuming the file is in order, but I figured that was implied by the "three consecutive lines" part of your question.



-----Original Message-----
From: Jose Malacara [mailto:josem@thrifty.net]
Sent: Wednesday, January 14, 2004 5:23 PM
To: [email]beginners@perl.org[/email]
Subject: multiline matching


Can someone explain to me how to do multiline matching? I am trying to extract three consecutive lines from a datafile containing multiple records like this:

Name: Bob
City: Austin
State: Texas
Address: 123 Whatever
Age: 46

Name: Jose
City: Denver
State: Colorado
Address: 118 Mystreet
Age: 28

[drowl@23.me.uk]

> On Wed, Jan 14, 2004 at 06:22:58PM -0700, Jose Malacara wrote:

>> Can someone explain to me how to do multiline matching? I am trying to
>> extract three consecutive lines from a datafile containing multiple
>> records like this:
>>

Hi All well i was reading through and i have a similar problem
i have code that "works" but its all slow and nasty =(
wonder if any one can help...
the data is stored in a text file in the format below,
we could have up to 40000 entry's so its a big file (but running on a E450
with 4Gig ram so ;) )
the number of lines can vary but will always start with...
segment "name" {
agentAddress "IP"
more "info"
}

any (semi constructive) ideas suggestions welcome

RichT

database format-------------------------------------
segment "Birmingham-WAN0" {
agentAddress "127.0.0.1"
uniqueDeviceId "SomeHeXNumbers (123445)"
mibTranslationFile "ciscoMib2.mtf"
index "2"
parentName "Birmingham-RH"
deviceSpeed "64000.0"
deviceSpeed2 "128000.0"
discoverMtf "ciscoMib2.mtf"
community "SupperPassword"
sysDescr "long text telling you the Cisco ver"
sysName "ThisOniSTheRouterName"
sysLoc "Birmingham"
ifDescr "Serial0"
ifType "frame-relay"
aliasName "Cust12345"
nmsName "Birmingham-WAN0"
nmsKey "ThisOniSTheRouterName link Serial0"
enterpriseId "9"
possibleLatencySources "concord, ciscoPing"
fullDuplex "1"
mediaSpeed "128000.0"
mediaSpeed1 "128000.0"
statistics "1"
}
End The database format is like this -------------------------------------


My Current code ----------------------------------------------
sub checkResults {
my($NH_HOME,$i,@pollerCfgLine,@foundElements)=("/nethealth1","0","","");
#setup main vars
my($elementName,$elementSpeed,$elementRouterName,$ elementAlias,$elementIP,$elementMIB);
# setting up names from poller
my($siteIPAdd)=$_[0];
open(POLLER, "$NH_HOME/poller/poller.cfg")|| die "can not open : $!"; #
open poller.cfg file , inport and close
my(@pollerCfg)=<POLLER>;
close POLLER;
foreach (@pollerCfg) { #start main loop throught all poller cfg lines
@pollerCfgLine=split(/"/,$_,3); #"split up type / value
if ($pollerCfgLine[0] =~ /^ +segment/) { #if we are on first line for
element key word "segment"
$elementName=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +agentAddress/){
$elementIP=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +mibTranslationFile/){
$elementMIB=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +deviceSpeed2/){
$elementSpeed=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +sysName/){
$elementRouterName=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +aliasName/){
$elementAlias=$pollerCfgLine[1];
}elsif ($pollerCfgLine[0] =~ /^ +\}$/){
if ($elementIP eq $siteIPAdd) {
$foundElements[$i]=("$elementName,$elementSpeed,$elementRouterName,$ elementAlias,$elementMIB\n");
$i++;
}
($elementName,$elementSpeed,$elementRouterName,$el ementAlias,$elementIP,$elementMIB)=("","","","","" ,"");
}
}

open(DISCOVERLOG, ">>tmp/discover.log.csv");
print DISCOVERLOG
"elementName,elementSpeed,elementRouterName,elemen tAlias,elementIP,elementMIB\n";
print DISCOVERLOG "@foundElements\n";

}

End My Current code ----------------------------------------------

[James Edward Gray II]

On Jan 14, 2004, at 10:28 PM, Kenton Brede wrote:

> On Wed, Jan 14, 2004 at 09:57:51PM -0600, James Edward Gray II
> (james@grayproductions.net) wrote:

>> On Jan 14, 2004, at 7:22 PM, Jose Malacara wrote:

> <snip>
>

>> Since you've already been shown the super easy way, I'll dare to be a
>> little different:
>>
>> #!/usr/bin/perl
>>
>> use strict;
>> use warnings;
>>
>> $/ = ''; # enter "paragraph" mode
>> while (<>) { # call with: perl script_name file1
>> my %contact = map { /^(\w+):\s*(.+)$/ } split /\n/, $_;
>> print "$_: $contact{$_}\n" foreach qw(Name City State);
>> print "\n"
>> }
>>
>> __END__
>>
>> The first way your were shown is probably a little easier, but this
>> method is probably better if you want to do anything more complicated
>> than simple printing, since you have the whole hash to play with.
>> It's
>> a different way of thinking about the problem at least.

>
> Thanks for posting this. My first thought was a hash, thinking in
> terms
> of key-item but I couldn't figure out how to populate the hash. This
> code will give me something to analyze. The "map" function looks like
> voodoo to me:)

It's not so scary, let's look at it in a longer form:

my %contact; # create the hash to populate
my @lines = split /\n/, $_; # break paragraph into lines
for my $line (@lines) { # walk the lines from the paragraph we read
if ($line =~ /^(\w+):\s*(.+)$/) { # find hash key and value
$contact{$1} = $2; # assign to hash
}
}

I was just lazy and didn't want to type that much, so I cheated:

# read the following comments from the bottom up
my %contact = # stick all that here, assumes Key, Value, Key,
Value...
map { /^(\w+):\s*(.+)$/ } # match each line and return what we capture
in parens
split /\n/, $_; # break paragraph into lines and hand them all to map

If you want more info, try:

perldoc -f split
perldoc -f map

Hope that clears it up a little.

James

[Kenton Brede]

On Thu, Jan 15, 2004 at 08:38:11AM -0600, James Edward Gray II (james@grayproductions.net) wrote:

> On Jan 14, 2004, at 10:28 PM, Kenton Brede wrote:
>

> >On Wed, Jan 14, 2004 at 09:57:51PM -0600, James Edward Gray II
> >(james@grayproductions.net) wrote:

> >>On Jan 14, 2004, at 7:22 PM, Jose Malacara wrote:

> ><snip>
> >

> >>Since you've already been shown the super easy way, I'll dare to be a
> >>little different:
> >>
> >>#!/usr/bin/perl
> >>
> >>use strict;
> >>use warnings;
> >>
> >>$/ = ''; # enter "paragraph" mode
> >>while (<>) { # call with: perl script_name file1
> >> my %contact = map { /^(\w+):\s*(.+)$/ } split /\n/, $_;
> >> print "$_: $contact{$_}\n" foreach qw(Name City State);
> >> print "\n"
> >>}
> >>
> >>__END__
> >>
> >>The first way your were shown is probably a little easier, but this
> >>method is probably better if you want to do anything more complicated
> >>than simple printing, since you have the whole hash to play with.
> >>It's
> >>a different way of thinking about the problem at least.

> >
> >Thanks for posting this. My first thought was a hash, thinking in
> >terms
> >of key-item but I couldn't figure out how to populate the hash. This
> >code will give me something to analyze. The "map" function looks like
> >voodoo to me:)

>
> It's not so scary, let's look at it in a longer form:
>
> my %contact; # create the hash to
> populate
> my @lines = split /\n/, $_; # break paragraph
> into lines
> for my $line (@lines) { # walk the lines
> from the paragraph we read
> if ($line =~ /^(\w+):\s*(.+)$/) { # find hash key and
> value
> $contact{$1} = $2; # assign to hash
> }
> }
>

<snip>

Thanks for the great explanation. I've been working with the code above
since I need more hash practice, trying to keep on track with the
original poster's question. The code below works fine except I can't
figure out how to put one "\n" between the two records like -

Name: Bob
City: Austin
State: Texas

Name: Jose
City: Denver
State: Colorado

If I place "print "\n";" after the print line I get double spaces
between all lines. If I place it outside the last "for" loop I get
double spaces between the two records. What I have below just prints
them in one block. Hope that all made some sense:)
Thanks for any help,
Kent

#!/usr/bin/perl
use warnings;
use strict;

while (<DATA>) {
my @lines = (split /\n/, $_);
my %contact;
for my $line (@lines) {
if ($line =~ /^(\w+):\s*(.+)$/) {
$contact{$1} = $2;
for (keys %contact) {
if (/^Name/ or /^City/ or /^State/) {
print "$_: $contact{$_}\n";
}
}
}
}
}

__DATA__
Name: Bob
City: Austin
State: Texas
Address: 123 Whatever
Age: 46

Name: Jose
City: Denver
State: Colorado
Address: 118 Mystreet
Age: 28
__END__


--
"I am always doing that which I can not do,
in order that I may learn how to do it." --Pablo Picasso

[James Edward Gray II]

On Jan 15, 2004, at 8:30 PM, Kenton Brede wrote:

> Thanks for the great explanation. I've been working with the code
> above
> since I need more hash practice, trying to keep on track with the
> original poster's question. The code below works fine except I can't
> figure out how to put one "\n" between the two records like -
>
> Name: Bob
> City: Austin
> State: Texas
>
> Name: Jose
> City: Denver
> State: Colorado
>
> If I place "print "\n";" after the print line I get double spaces
> between all lines. If I place it outside the last "for" loop I get
> double spaces between the two records. What I have below just prints
> them in one block. Hope that all made some sense:)

I'm with you. See below.

> Thanks for any help,
> Kent
>
> #!/usr/bin/perl
> use warnings;
> use strict;

You forgot a very important line from my example right here:

local $/ = ''; # enter "paragraph" mode

Your code was reading line by line, then working on just that line.
You never had more than one entry in the hash at a time.

My example reads until it sees one or more blanks lines (paragraph
mode), then works with an entire contact at once.

That almost fixes you up, but we have to do a little more.

> while (<DATA>) {
> my @lines = (split /\n/, $_);
> my %contact;
> for my $line (@lines) {
> if ($line =~ /^(\w+):\s*(.+)$/) {
> $contact{$1} = $2;

The for loop below is in the wrong spot. Here we're processing each
line of the hash, so the loop would get run once for each and every
line, giving us erroneous output. We need to move it down a bit...

> for (keys %contact) {
> if (/^Name/ or /^City/ or /^State/) {

Minor complaint about the line above. Don't use a Regex to test
equality. Save the big guns for when you actually need pattern
matching. It should read:

if ($_ eq 'Name' || $_ eq 'City' || $_ eq 'State') {

> print "$_: $contact{$_}\n";
> }
> }
> }
> }

Move that for loop I mentioned above to here and add a:

print "\n";

after the loop. That "fixes" your program. Let's look into it a
little more though...

> }

Putting everything I've said so far together, we get the code:

#!/usr/bin/perl

use warnings;
use strict;

local $/ = '';

while (<DATA>) {
my @lines = (split /\n/, $_);
my %contact;
for my $line (@lines) {
if ($line =~ /^(\w+):\s*(.+)$/) {
$contact{$1} = $2;
}
}
for (keys %contact) {
if ($_ eq 'Name' || $_ eq 'City' || $_ eq 'State') {
print "$_: $contact{$_}\n";
}
}
print "\n";
}

__DATA__
Name: Bob
City: Austin
State: Texas
Address: 123 Whatever
Age: 46

Name: Jose
City: Denver
State: Colorado
Address: 118 Mystreet
Age: 28

Go ahead and paste that into a file and run it a few times for some
interesting results.

Do you see it? The order of the output changes with subsequent runs if
you're running a fairly recent version of perl.

I don't think it was ever said if order is significant in this problem,
but the behavior is annoying so let's fix it too. The "problem" is in
the output loop:

for (keys %contact) {
if ($_ eq 'Name' || $_ eq 'City' || $_ eq 'State') {
print "$_: $contact{$_}\n";
}
}

That loop fetches all the keys of the hash and then prints the Name,
City and State as it finds them. Trouble is, perl isn't guaranteed to
return those keys in a set order and indeed it doesn't. The loop is
also wasteful as it has to go through keys we don't care about. Let's
change it to ask for what we want instead:

for (qw(Name City State)) {
print "$_: $contact{$_}\n";
}

That does it. The program now returns what we expect every time it's
run.

Hope that clears things up for you.

James