Can Perl do this?

[CountryLover]

First off, I know next to nothing about Perl, other than a few scripts that I
have used (written by others).

A project that I am working on requires the following steps:

1. Parse a directory containing text files, potentially thousands
2. Open each text file and read in the first 25 lines or so
3. Write these lines to a temporary file
4. Delete each original file & replace with the temp file, using same filename
5. Parse the truncated files and find a specific keyword
6. Create directories based on the value of the keywords found
7. Move the truncated files to their respective directories.
8. Optionally, create a log file of all operations

Is this something that Perl could be persuaded to do?

Any suggestions or code snippets would be much appreciated!

[Vlad Tepes]

CountryLover <crazygonuts@pcgeek.invalid> wrote:

>
> First off, I know next to nothing about Perl, other than a few scripts that I
> have used (written by others).
>
> A project that I am working on requires the following steps:
>
> 1. Parse a directory containing text files, potentially thousands

my @files = </path/to/dir/*>;

> 2. Open each text file and read in the first 25 lines or so
> 3. Write these lines to a temporary file
> 4. Delete each original file & replace with the temp file, using same filename

FILE: foreach my $filename ( @files ) {
open $fh, $filename or do {
warn "Can't open $filename: $!";
next FILE;
};
while ( <$fh> ) {

}

> 5. Parse the truncated files and find a specific keyword
> 6. Create directories based on the value of the keywords found
> 7. Move the truncated files to their respective directories.
> 8. Optionally, create a log file of all operations
>
> Is this something that Perl could be persuaded to do?
>
> Any suggestions or code snippets would be much appreciated!
>
>

--
Vlad

[John Bokma]

CountryLover wrote:

> First off, I know next to nothing about Perl, other than a few scripts that I
> have used (written by others).
>
> A project that I am working on requires the following steps:
>
> 1. Parse a directory containing text files, potentially thousands

Parse as "traverse"? Sure. Hmmm, with just one directory it means
getting all text files.

> 2. Open each text file and read in the first 25 lines or so
> 3. Write these lines to a temporary file
> 4. Delete each original file & replace with the temp file, using same filename
> 5. Parse the truncated files and find a specific keyword
> 6. Create directories based on the value of the keywords found
> 7. Move the truncated files to their respective directories.
> 8. Optionally, create a log file of all operations
>
> Is this something that Perl could be persuaded to do?

Yes, quite easy. I do similar things often with Perl :-)

Probably a *nix hacker could do most, if not all with GNU tools like
find (the txt files), head (to get the first 25), mv, grep/sed, mkdir,
mv and such. Probably in a one-liner.

> Any suggestions or code snippets would be much appreciated!

:-) I could be persuaded to write it for you ;-)

--
Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
virtual home: [url]http://johnbokma.com/[/url] ICQ: 218175426
John web site hints: [url]http://johnbokma.com/websitedesign/[/url]

[Vlad Tepes]

Vlad Tepes <minceme@start.no> wrote:

> CountryLover <crazygonuts@pcgeek.invalid> wrote:

(
Sorry about my previous reply. I started writing an answer, but
changed my mind. I really have to go to bed, and in my tired state
I pressed the wrong button. :-(
)

--
Vlad [ Zzzz...... ]

[CountryLover]

On Mon, 8 Sep 2003 01:30:01 +0000 (UTC), Vlad Tepes <minceme@start.no> wrote:

>CountryLover <crazygonuts@pcgeek.invalid> wrote:
>

>>
>> First off, I know next to nothing about Perl, other than a few scripts that I
>> have used (written by others).
>>
>> A project that I am working on requires the following steps:
>>
>> 1. Parse a directory containing text files, potentially thousands

>
>my @files = </path/to/dir/*>;
>

>> 2. Open each text file and read in the first 25 lines or so
>> 3. Write these lines to a temporary file
>> 4. Delete each original file & replace with the temp file, using same filename

>
>FILE: foreach my $filename ( @files ) {
> open $fh, $filename or do {
> warn "Can't open $filename: $!";
> next FILE;
> };
> while ( <$fh> ) {
>
>}
>
>
>

>> 5. Parse the truncated files and find a specific keyword
>> 6. Create directories based on the value of the keywords found
>> 7. Move the truncated files to their respective directories.
>> 8. Optionally, create a log file of all operations
>>
>> Is this something that Perl could be persuaded to do?
>>
>> Any suggestions or code snippets would be much appreciated!
>>
>>

Thanks for the reply, Vlad, I really appreciate it. I've been doing alot of
this manually for several weeks, and it's gettin old fast :)

I'll try out your suggestions tonite.

Thanks again!

[James E Keenan]

"CountryLover" <crazygonuts@pcgeek.invalid> wrote in message
news:7ilnlv48qm5raprrfmi7e1gfd1tvbn7guc@4ax.com...

>
> First off, I know next to nothing about Perl, other than a few scripts

that I

> have used (written by others).
>
> A project that I am working on requires the following steps:
>
> 1. Parse a directory containing text files, potentially thousands
> 2. Open each text file and read in the first 25 lines or so
> 3. Write these lines to a temporary file
> 4. Delete each original file & replace with the temp file, using same

filename

> 5. Parse the truncated files and find a specific keyword
> 6. Create directories based on the value of the keywords found
> 7. Move the truncated files to their respective directories.
> 8. Optionally, create a log file of all operations
>
> Is this something that Perl could be persuaded to do?

Yes.

>
> Any suggestions or code snippets would be much appreciated!
>

Buy a good introductory Perl textbook such as Randal Schwartz's "Learning
Perl." Work through it. By the time you're 2/3 of the way through it,
you'll know enough Perl to accomplish everything you describe above.

[CountryLover]

On Mon, 08 Sep 2003 03:33:15 +0200, John Bokma <postmaster@castleamber.com>
wrote:

>CountryLover wrote:
>

>> First off, I know next to nothing about Perl, other than a few scripts that I
>> have used (written by others).
>>
>> A project that I am working on requires the following steps:
>>
>> 1. Parse a directory containing text files, potentially thousands

>
>Parse as "traverse"? Sure. Hmmm, with just one directory it means
>getting all text files.
>

>> 2. Open each text file and read in the first 25 lines or so
>> 3. Write these lines to a temporary file
>> 4. Delete each original file & replace with the temp file, using same filename
>> 5. Parse the truncated files and find a specific keyword
>> 6. Create directories based on the value of the keywords found
>> 7. Move the truncated files to their respective directories.
>> 8. Optionally, create a log file of all operations
>>
>> Is this something that Perl could be persuaded to do?

>
>Yes, quite easy. I do similar things often with Perl :-)
>
>Probably a *nix hacker could do most, if not all with GNU tools like
>find (the txt files), head (to get the first 25), mv, grep/sed, mkdir,
>mv and such. Probably in a one-liner.
>

>> Any suggestions or code snippets would be much appreciated!

>
>:-) I could be persuaded to write it for you ;-)

Would a "pretty please" be sufficient to persuade you? :)

Thing is, I'm on a tight timeline, and starting with some ready-made code would
sure beat starting from scratch to learn Perl.

Pretty Please?!

[CountryLover]

On Mon, 8 Sep 2003 01:38:39 +0000 (UTC), Vlad Tepes <minceme@start.no> wrote:

>Vlad Tepes <minceme@start.no> wrote:

>> CountryLover <crazygonuts@pcgeek.invalid> wrote:

>
>(
> Sorry about my previous reply. I started writing an answer, but
> changed my mind. I really have to go to bed, and in my tired state
> I pressed the wrong button. :-(
>)

No prob, Vlad. I appreciate your effort nonetheless.

Have a good nite!

[John Bokma]

CountryLover wrote:

> Would a "pretty please" be sufficient to persuade you? :)

An O'Reilly book would :-)

> Thing is, I'm on a tight timeline, and starting with some ready-made code would
> sure beat starting from scratch to learn Perl.
>
> Pretty Please?!

Two books would make me write it now or tomorrow :-D

--
Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
virtual home: [url]http://johnbokma.com/[/url] ICQ: 218175426
John web site hints: [url]http://johnbokma.com/websitedesign/[/url]

[Jeff 'japhy' Pinyan]

On Mon, 8 Sep 2003, CountryLover wrote:

>1. Parse a directory containing text files, potentially thousands
>2. Open each text file and read in the first 25 lines or so
>3. Write these lines to a temporary file
>4. Delete each original file & replace with the temp file, using same filename

That can be done in Perl without a temporary file. Roughly:

open FILE, "<+ $filename" or
die "can't open $filename for read/write: $!";
<FILE> for 1 .. 25;
truncate FILE, tell FILE;

>5. Parse the truncated files and find a specific keyword

seek FILE, 0, 0;
while (<FILE>) {
# look for keywords
}

That could even be done in the previous chunk of code, stopping at the
25th line and truncating.

>6. Create directories based on the value of the keywords found
>7. Move the truncated files to their respective directories.

Do you mean that you want a copy of each file in each directory that the
file has the keyword of? So that foo.txt, with keywords 'yucca', 'root',
and 'beer' would be in the yucca/, root/, and beer/ directories?

>8. Optionally, create a log file of all operations

Simple.

--
Jeff Pinyan RPI Acacia Brother #734 2003 Rush Chairman
"And I vos head of Gestapo for ten | Michael Palin (as Heinrich Bimmler)
years. Ah! Five years! Nein! No! | in: The North Minehead Bye-Election
Oh. Was NOT head of Gestapo AT ALL!" | (Monty Python's Flying Circus)

[Mike Flannigan]

Perl would be great to do what you want. Problem is, you are
unlikely to get this done just by using somebody else's code.
I mean, you are probably going to need to upgrade and refine
the program as you use it and see it's shortcomings.

Simple things can often be done just by asking for an
answer on this list. In your case I highly recommend
taking the time to learn Perl. It will probably take you
only 2 or 3 weeks or so - maybe less.


Mike


CountryLover wrote:

> Would a "pretty please" be sufficient to persuade you? :)
>
> Thing is, I'm on a tight timeline, and starting with some ready-made code would
> sure beat starting from scratch to learn Perl.
>
> Pretty Please?!

[John Bokma]

Mike Flannigan wrote:

> Perl would be great to do what you want. Problem is, you are
> unlikely to get this done just by using somebody else's code.
> I mean, you are probably going to need to upgrade and refine
> the program as you use it and see it's shortcomings.
>
> Simple things can often be done just by asking for an
> answer on this list. In your case I highly recommend
> taking the time to learn Perl. It will probably take you
> only 2 or 3 weeks or so - maybe less.

With "Learning Perl" and the "Cook book" I guess it could be done within
a week or less just by copy & paste and trail & error.

However Learning the basics of Perl takes months and months if not more.
I use Perl for almost 10 years and I learn quite often new things.

Worse, when I see scripts that are a few years old I sometimes wonder
how I ever came up with that solution.

--
Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
virtual home: [url]http://johnbokma.com/[/url] ICQ: 218175426
John web site hints: [url]http://johnbokma.com/websitedesign/[/url]

[CountryLover]

On Mon, 08 Sep 2003 04:07:20 +0200, John Bokma <postmaster@castleamber.com>
wrote:

>CountryLover wrote:
>

>> Would a "pretty please" be sufficient to persuade you? :)

>
>An O'Reilly book would :-)
>

>> Thing is, I'm on a tight timeline, and starting with some ready-made code would
>> sure beat starting from scratch to learn Perl.
>>
>> Pretty Please?!

>
>Two books would make me write it now or tomorrow :-D

Thanks, but if I'm gonna buy a book or two, I'll just use `em myself. :)

[CountryLover]

On Mon, 08 Sep 2003 02:32:37 GMT, Mike Flannigan <mikeflan@earthlink.net>
wrote:

>Perl would be great to do what you want. Problem is, you are
>unlikely to get this done just by using somebody else's code.
>I mean, you are probably going to need to upgrade and refine
>the program as you use it and see it's shortcomings.
>
>Simple things can often be done just by asking for an
>answer on this list. In your case I highly recommend
>taking the time to learn Perl. It will probably take you
>only 2 or 3 weeks or so - maybe less.

Agreed. However, I just don't have alot of time to devote to it right now on
top of the stuff they pay me to do. :)

I'm going to grab a few books and see what I can whip up....

[John Bokma]

CountryLover wrote:

>>Two books would make me write it now or tomorrow :-D

>
> Thanks, but if I'm gonna buy a book or two, I'll just use `em myself. :)

:-D. Which I surely recommend. There are two kind of people: the book
readers and people who hire book readers.

I can recommend the Perl Cook Book and make sure you get the latest
edition. To learn Perl either pick learning Perl or Programming Perl.
The latter expects quite some programming experience and is too hard for
most people to learn Perl I guess.

And where can you find a program, well documented and tested within a
few hours for just two O'Reilly books :-D.

--
Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
virtual home: [url]http://johnbokma.com/[/url] ICQ: 218175426
John web site hints: [url]http://johnbokma.com/websitedesign/[/url]

[John W. Krahn]

CountryLover wrote:

>
> First off, I know next to nothing about Perl, other than a few scripts that I
> have used (written by others).
>
> A project that I am working on requires the following steps:
>
> 1. Parse a directory containing text files, potentially thousands
> 2. Open each text file and read in the first 25 lines or so
> 3. Write these lines to a temporary file
> 4. Delete each original file & replace with the temp file, using same filename
> 5. Parse the truncated files and find a specific keyword
> 6. Create directories based on the value of the keywords found
> 7. Move the truncated files to their respective directories.
> 8. Optionally, create a log file of all operations
>
> Is this something that Perl could be persuaded to do?
>
> Any suggestions or code snippets would be much appreciated!

#!/usr/bin/perl
use warnings;
use strict;

# UNTESTED !!

my $log_file = '/home/cl/log_file';
open my $log, '>>', $log_file or die "Cannot open $log_file: $!";
select( ( select( $log ), $| = 1 )[ 0 ] );

my $dir = '/home/cl/textfiles';
my $lines_to_read = 25;

opendir my $dh, $dir or die "Cannot open $dir: $!";

while ( defined( my $file = readdir $dh ) ) {
next unless -T "$dir/$file";
print $log "Found file: $dir/$file\n";

open my $fh, '<', "$dir/$file" or die "Cannot open $dir/$file: $!";
my ( $data, $keyword );
while ( <$fh> ) {
last if $. > $lines_to_read;
$keyword = $1 if /keyword: (\w+)/;
$data .= $_;
}
close $fh;

next unless defined $keyword;
print $log "Found keyword: $keyword\n";

unless ( -d "$dir/$keyword" ) {
mkdir "$dir/$keyword" or die "Cannot mkdir $dir/$keyword: $!";
}
open my $fh, '>', "$dir/$keyword/$file" or die "Cannot open $dir/$keyword/$file: $!";
print $fh $data or die "Cannot write to $dir/$keyword/$file: $!";
close $fh;
print $log "New file $dir/$keyword/$file created.\n";

unlink "$dir/$file" or warn "Cannot delete $dir/$file: $!";
}

__END__



John
--
use Perl;
program
fulfillment

[Helgi Briem]

On Mon, 08 Sep 2003 01:23:09 GMT, CountryLover
<crazygonuts@pcgeek.invalid> wrote:

>A project that I am working on requires the following steps:
>
>1. Parse a directory containing text files, potentially thousands

perldoc perlrun
perldoc warnings
perldoc strict
perldoc -f opendir
perldoc -f grep
perldoc -f readdir
perldoc -f die
perldoc perlvar ($!)
perldoc File::Find (if there are subdirectories)

>2. Open each text file and read in the first 25 lines or so

perldoc -f open
perldoc perlvar ( $. )
perldoc perlsyn (while)

>3. Write these lines to a temporary file

perldoc -f open
perldoc -f print
perldoc -f close

>4. Delete each original file & replace with the temp file, using same filename

perldoc -f unlink

>5. Parse the truncated files and find a specific keyword

perldoc perlre (or perldoc perlrequick)

>6. Create directories based on the value of the keywords found

perldoc -f mkdir

>7. Move the truncated files to their respective directories.

perldoc -f rename

>8. Optionally, create a log file of all operations

perldoc -f open
perldoc -f print

>Is this something that Perl could be persuaded to do?

Of course. What have you tried so far?

[Tim Hammerquist]

John Bokma graced us by uttering:

> CountryLover wrote:

<snip>

> > Is this something that Perl could be persuaded to do?

>
> Yes, quite easy. I do similar things often with Perl :-)
>
> Probably a *nix hacker could do most, if not all with GNU tools
> like find (the txt files), head (to get the first 25), mv,
> grep/sed, mkdir, mv and such. Probably in a one-liner.

Yes, it's possible with standard Unix tools. But as it will most
likely be extended and expanded, I'd recommend Perl anyway. :)

Tim Hammerquist
--
Every man is guilty of all the good he didn't do.
-- Voltaire

[John Bokma]

Tim Hammerquist wrote:

> John Bokma graced us by uttering:
>

>>CountryLover wrote:

>
> <snip>
>

>>>Is this something that Perl could be persuaded to do?

>>
>>Yes, quite easy. I do similar things often with Perl :-)
>>
>>Probably a *nix hacker could do most, if not all with GNU tools
>>like find (the txt files), head (to get the first 25), mv,
>>grep/sed, mkdir, mv and such. Probably in a one-liner.

>
> Yes, it's possible with standard Unix tools. But as it will most
> likely be extended and expanded, I'd recommend Perl anyway. :)

Indeed. I use Perl to forgot all the horrible experiences with sed, awk
and friends :-)

--
Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
virtual home: [url]http://johnbokma.com/[/url] ICQ: 218175426
John web site hints: [url]http://johnbokma.com/websitedesign/[/url]

[Bart Lateur]

CountryLover wrote:

>First off, I know next to nothing about Perl, other than a few scripts that I
>have used (written by others).
>
>A project that I am working on requires the following steps:
>
>1. Parse a directory containing text files, potentially thousands
>2. Open each text file and read in the first 25 lines or so
>3. Write these lines to a temporary file
>4. Delete each original file & replace with the temp file, using same filename
>5. Parse the truncated files and find a specific keyword
>6. Create directories based on the value of the keywords found
>7. Move the truncated files to their respective directories.
>8. Optionally, create a log file of all operations
>
>Is this something that Perl could be persuaded to do?

Yes.

In fact, given such a task, Perl would be my first choice. (Perl's not
the only language I use...)

BTW instead of copying the first N lines of a file and then overwriting
the original file with the truncated one, you can just as well use the
truncate() keyword... you may have to use seek() for this to take
effect.

Or you can use the -i command line switch, and stop after reading and
printing 25 lines.

To move a file, use the File::Copy module.

--
Bart.