Final "Flocking" Script

[\Dandy\ Randy]

Thanx you to everyone who has been helping with my flocking issue. After
reading several perdocs from [url]www.perdoc.com[/url] I have come up with the
following script. This script simply advances the second value of a text
file number by 1 each time it is run. The only quirk I have found is with
"use strict" and "use warnings" In my script these lines are blocked out ...
it is the only way the sript runs correctly.

#!/usr/bin/perl

#use strict;
#use warnings;
use 5.004;
use Fcntl qw(:DEFAULT :flock);

open (FH, "<data.txt") or die "Can't open file: $!";
flock (FH, LOCK_EX) or die "Can't lock file: $!";
$data=<FH>;
chomp ($data);
($total,$opened,$followed)=split(/\|/,$data);
close(FH);

$opened = $opened + 1;

sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
$!";
flock (FH, LOCK_EX) or die "can't lock filename: $!";
truncate (FH, 0) or die "can't truncate filename: $!";
print FH "$total|$opened|$followed\n";
close FH;

print "Content-type: text/html \n\n";
print "Done\n";
exit;

The end result, the number advances correctly. This script is basically a
counter for email ... one of my other programs sends out several emails, and
in the email code, there are instructions to run this script ... it simply
tells me how many emails were actually opened compared to how many sent. I
am hoping the flocking script i'm using above is the correct one for my
"cause". If it's not, please advise. thanx to everyone for the help;

Randy

P.S.

In case any of you are wondering about the strict or warning issues, when I
unblock these commands, the server returns the following text:

--------------------------------------
Internal Server Error
The server encountered an internal error or misconfiguration and was unable
to complete your request.
Please contact the server administrator and inform them of the time the
error occurred, and anything you might have done that may have caused the
error.

More information about this error may be available in the server error log.
--------------------------------------
Apache/1.3.27 Server at [url]www.awebsite.com[/url] Port 80

[Tad McClellan]

\"Dandy\" Randy <ducott@hotmail.com> wrote:

> sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
> $!";
> flock (FH, LOCK_EX) or die "can't lock filename: $!";
> truncate (FH, 0) or die "can't truncate filename: $!";

There should be a seek(FH, 0, 0) here in case some other process
advanced the file pointer between your sysopen() and your flock().

> In case any of you are wondering about the strict or warning issues, when I
> unblock these commands, the server returns the following text:
>
> --------------------------------------

[snip]

> More information about this error may be available in the server error log.
> --------------------------------------

So then, what did it say in the server error log?


--
Tad McClellan SGML consulting
[email]tadmc@augustmail.com[/email] Perl programming
Fort Worth, Texas

[Tad McClellan]

Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:

> Put the following at the beginning of the script:
>
> use CGI::Carp 'fatalsToBrowser';

And be sure to *remove it* for production!


--
Tad McClellan SGML consulting
[email]tadmc@augustmail.com[/email] Perl programming
Fort Worth, Texas

[Gunnar Hjalmarsson]

Tad McClellan wrote:

> Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:

>>Put the following at the beginning of the script:
>>
>> use CGI::Carp 'fatalsToBrowser';

>
> And be sure to *remove it* for production!

What makes you say that, Tad? The CGI::Carp docs suggests that the
carpout() routine shouldn't be used for production for *performance*
reasons, but I don't think there is such a comment as regards
fatalsToBrowser(). Or is there any other reason for your advise?

--
Gunnar Hjalmarsson
Email: [url]http://www.gunnar.cc/cgi-bin/contact.pl[/url]

[Alan J. Flavell]

On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:

> Tad McClellan wrote:

> > And be sure to *remove it* for production!

>
> What makes you say that, Tad? The CGI::Carp docs suggests that the
> carpout() routine shouldn't be used for production for *performance*
> reasons, but I don't think there is such a comment as regards
> fatalsToBrowser(). Or is there any other reason for your advise?

I don't know Tad's motivation, but: it can expose details of the
internals of your scripts to an intruder. The same goes for exposing
warnings within the browser window.

Of course, it also gives an unprofessional impression if it happens to
a bona fide user. Ideally you should cover all known failure modes at
development time, and supply user-friendly responses for them, rather
than exposing the end-users to what are _supposed_ to be your internal
diagnostics.

But indeed, during development these features can be _very_ useful.

--
"In the hardest of cases, I show them the W3C membership list: as
soon as the customer sees that MS is a member, then everything the
W3C says is good, and typically they even demand conformance with
W3C written into the contract" - (Matthias Esken, freely translated)

[Matija Papec]

X-Ftn-To: Tad McClellan

[email]tadmc@augustmail.com[/email] (Tad McClellan) wrote:

>> sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
>> $!";
>> flock (FH, LOCK_EX) or die "can't lock filename: $!";
>> truncate (FH, 0) or die "can't truncate filename: $!";

>
>There should be a seek(FH, 0, 0) here in case some other process
>advanced the file pointer between your sysopen() and your flock().

Hi,
isn't a file pointer something that belongs to(and only to) process itself?


--
Matija

[Chas Friedman]

__________
You wrote:

>Thanx you to everyone who has been helping with my flocking issue.
>.................
>#!/usr/bin/perl
>
>#use strict;
>#use warnings;
>use 5.004;
>use Fcntl qw(:DEFAULT :flock);
>
>open (FH, "<data.txt") or die "Can't open file: $!";
>flock (FH, LOCK_EX) or die "Can't lock file: $!";
>$data=<FH>;
>chomp ($data);
>($total,$opened,$followed)=split(/\|/,$data);
>close(FH);
>
>$opened = $opened + 1;
>
>sysopen(FH, "data.txt", O_WRONLY | O_CREAT) or die "can't open filename:
>$!";
>flock (FH, LOCK_EX) or die "can't lock filename: $!";

..........

I always worried about having the open and lock statements separate. I had
the idea (possibly wrong) that there could be some time lag between
execution of the 2 statements, and the file could change during that time.
(In the particular example you consider, this might be unlikely.)
I usually do something like:

..............
use Fcntl ':flock'; # import LOCK_* constants
use Fcntl ':DEFAULT';
use POSIX;
sub lock {
flock F, LOCK_EX;
# and, in case someone appended <-----
# while we were waiting... <------ (This is from perl docs.)
seek F, 0, 2; #<-----------
}

sub unlock {
flock F, LOCK_UN;
}

if (sysopen(F,$file, O_WRONLY)){
lock();}
..............

That way, the open and lock is "atomic". I wonder if this is actually
necessary. The perl docs show some examples where the open and lock are
in separate statements, so I thought that should be OK. But I once did
some tests of locking in which I appended, say, 100 times to a file with
a script using open and lock separately and then in a single statement, and
it seemed to make some difference - there were occasionally some errors
when the open and lock were in separate statements. However, this could have
been for some other reason...I was never totally sure...
I'd be interested in any comments about this!
chas

[Tad McClellan]

Chas Friedman <friedman@math.utexas.edu> wrote:

> You wrote:

Who is "you"?

Please provide a proper attribution when you quote someone.

>>Thanx you to everyone who has been helping with my flocking issue.

>>open (FH, "<data.txt") or die "Can't open file: $!";
>>flock (FH, LOCK_EX) or die "Can't lock file: $!";

> I always worried about having the open and lock statements separate. I had
> the idea (possibly wrong) that there could be some time lag between
> execution of the 2 statements, and the file could change during that time.

Eh?

The very purspose of doing file locking in the first place is
because of such a "time lag", most commonly called a
"race condition".

So your idea is perfectly right. (so far)

> I usually do something like:
>
> .............
> use Fcntl ':flock'; # import LOCK_* constants
> use Fcntl ':DEFAULT';
> use POSIX;
> sub lock {
> flock F, LOCK_EX;
> # and, in case someone appended <-----
> # while we were waiting... <------ (This is from perl docs.)
> seek F, 0, 2; #<-----------
> }
>
> sub unlock {
> flock F, LOCK_UN;
> }

Uh oh, if this gets called, then you have _introduced_ a race,
exactly the opposite of what you're trying to do.

If you're worried about the open/flock timing why not worry
about the unlock/close timing as well?

seek()ing can overcome the former, but the later is a problem.

Do not unlock the file, just close() it. close() will unlock
it in a way that _is_ atomic.

> if (sysopen(F,$file, O_WRONLY)){
> lock();}
> .............
>
> That way, the open and lock is "atomic".

No they're not.

Why do you think that they are atomic in your code?

Your process could _still_ do the open() and lose its time slice
before doing the flock().

In fact, it makes the time for the race to bite you _longer_ due
to the overhead of calling a subroutine.

You made it a little worse, but that's OK since you're seek()ing anyway.

> I wonder if this is actually
> necessary.

If "this" means your code above, then no, it is not necessary.

> The perl docs show some examples where the open and lock are
> in separate statements, so I thought that should be OK.

Putting them in the same statement will not affect the possibility
of a race either.

> But I once did
> some tests of locking in which I appended, say, 100 times to a file with
> a script using open and lock separately and then in a single statement, and
> it seemed to make some difference

Must have been a statistical anomaly.

> - there were occasionally some errors
> when the open and lock were in separate statements. However, this could have
> been for some other reason...

Like the race between the unlock and the close. :-)


--
Tad McClellan SGML consulting
[email]tadmc@augustmail.com[/email] Perl programming
Fort Worth, Texas

[Matija Papec]

X-Ftn-To: Steve Grazzini

Steve Grazzini <grazz@pobox.com> wrote:

>The problem would occur when another process writes to the
>file between the append-open() and the flock(). This could
>invalidate your seek pointer -- i.e. you're prepared to write
>at the end of the file, and the end of the file *moves* so
>you end up writing in the wrong place.
>
>This only applies to appending, though, and the FAQ suggests
>that it only applies to Windows. Maybe somebody can list some

You mean faq from perldoc or some other faq?
It seems that under Linux you can freely change $file while script sleeps,


open FH, "+>>$file" or die $!;
sleep 20;
flock(FH, 2); #exclusive_lock
print FH "writing to EOF\n";


--
Matija

[Brian McCauley]

Matija Papec <mpapec@yahoo.com> writes:

> X-Ftn-To: Steve Grazzini
>
> Steve Grazzini <grazz@pobox.com> wrote:

> >The problem would occur when another process writes to the
> >file between the append-open() and the flock(). This could
> >invalidate your seek pointer -- i.e. you're prepared to write
> >at the end of the file, and the end of the file *moves* so
> >you end up writing in the wrong place.

Unless, of course, you are not using an OS that doesn't correctly
implement append-open().

Or, to put it another way (wihtout the tripple negative)...

Only if your OS has broken append-open.

> >
> >This only applies to appending, though, and the FAQ suggests
> >that it only applies to Windows.

I don't know if it _only_ applies to Windows but it doesn't apply on
OSs with working append-open.

I must admit I'd thought 5.8 was going to have a work-round for this bug
by putting an a seek-to-end before each write to a filehandle opened
for append on Windows. Guess it never made it.

> It seems that under Linux you can freely change $file while script sleeps,

You can but it doesn't matter, because append-open isn't broken on Linux.

--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\

[Steve Grazzini]

Matija Papec <mpapec@yahoo.com> wrote:

> Steve Grazzini <grazz@pobox.com> wrote:

> >The problem would occur when another process writes to the
> >file between the append-open() and the flock().
> >
> >This only applies to appending, though, and the FAQ suggests
> >that it only applies to Windows.

>
> You mean faq from perldoc or some other faq?

The Perl FAQ.

% perldoc -q "do i still have to use locking"

I was referring to this:

If you know you are only going to use a system that does
correctly implement appending (i.e. not Win32) then you
can omit the seek() from the above code.

--
Steve

[Purl Gurl]

Steve Grazzini wrote:

> Matija Papec wrote:

> > Steve Grazzini wrote:

(snipped)

> > You mean faq from perldoc or some other faq?

> The Perl FAQ.

> % perldoc -q "do i still have to use locking"

> I was referring to this:

> If you know you are only going to use a system that does
> correctly implement appending (i.e. not Win32) then you
> can omit the seek() from the above code.

I have posted a number of articles highlighting how incorrect
is this FAQ. Perl FAQ maintainers have known for a number of
years this Perl FAQ is dead wrong but refuse to correct it.

Place little faith in Perl FAQs. A good percentage disseminate
poor advice or totally incorrect information. Test all methods
presented in Perl FAQs before blindly believing them true; you
are reading material presented by those who firmly believe
a whole number is a decimal number.

Most Perl FAQs are correct, many are dead wrong if not idiocy.


Purl Gurl

[Purl Gurl]

Purl Gurl wrote:

> Steve Grazzini wrote:

> > Matija Papec wrote:

> > > Steve Grazzini wrote:

(snipped)

> > The Perl FAQ.

> > If you know you are only going to use a system that does
> > correctly implement appending (i.e. not Win32) then you
> > can omit the seek() from the above code.

> I have posted a number of articles highlighting how incorrect
> is this FAQ. Perl FAQ maintainers have known for a number of
> years this Perl FAQ is dead wrong but refuse to correct it.

Win32 appending seriously discrediting the author this FAQ.


Purl Gurl
--

#!perl

print "BEFORE APPEND:\n\n";

open (READ, "test.txt");
while (<READ>)
{ print $_; }
close (READ);

open (APPEND, ">>test.txt");
print APPEND "This Is Appended Text.";
close (APPEND);

print "\n\nAFTER APPEND:\n\n";

open (NEW, "test.txt");
while (<NEW>)
{ print $_; }
close (NEW);

PRINTED RESULTS:
________________

BEFORE APPEND:

Original Line 1
Original Line 2
Appending Will Follow:


AFTER APPEND:

Original Line 1
Original Line 2
Appending Will Follow:
This Is Appended Text.

[ctcgag@hotmail.com]

"Alan J. Flavell" <flavell@mail.cern.ch> wrote:

> On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:
>

> > Tad McClellan wrote:

>

> > > And be sure to *remove it* for production!

> >
> > What makes you say that, Tad? The CGI::Carp docs suggests that the
> > carpout() routine shouldn't be used for production for *performance*
> > reasons, but I don't think there is such a comment as regards
> > fatalsToBrowser(). Or is there any other reason for your advise?

>
> I don't know Tad's motivation, but: it can expose details of the
> internals of your scripts to an intruder. The same goes for exposing
> warnings within the browser window.

In my particular case, all the users of the scripts are inside our
firewall. Any of those users could just walk into my office and shove
an icepick in my ear. So if they've gone bad and are on the attack, CGI
scripts are the least of my worries. :)

Of course, I'm usually one of the major users of my own scripts, so
at least part of the time that the user gets a ugly error message, the
user is someone in a position to do something about it.

> Of course, it also gives an unprofessional impression if it happens to
> a bona fide user. Ideally you should cover all known failure modes at
> development time, and supply user-friendly responses for them, rather
> than exposing the end-users to what are _supposed_ to be your internal
> diagnostics.

I've had a group decide that the error messages looked unprofessional.
But they weren't willing to spend the time making user-friendly responses,
so instead they just yanked the fatalsToBrowser, and let the scripts
die silently. No ugly error messages anymore, just pretty half-rendered
web pages with no clue as to what is wrong. The veneer of professionalism,
without the substance. It just boggled my mind.

>
> But indeed, during development these features can be _very_ useful.

Yes, and everything I do is always in development. :)

Xho

--
-------------------- [url]http://NewsReader.Com/[/url] --------------------
Usenet Newsgroup Service New Rate! $9.95/Month 50GB

[Steve Grazzini]

[email]ctcgag@hotmail.com[/email] wrote:

> Steve Grazzini <grazz@pobox.com> wrote:

>>
>> [ error-checking omitted ]

>
> Which kind of screwed my up, because you don't
> use Fcntl ':flock';

Sigh.

I actually ran it that way and didn't notice -- demonstrating
either that the OP's flock() didn't do anything useful, or that
I'm a careless fool, or very possibly: both.

--
Steve

[Gunnar Hjalmarsson]

Alan J. Flavell wrote:

> On Tue, Aug 5, Gunnar Hjalmarsson inscribed on the eternal scroll:

>> Alan J. Flavell wrote:

>>> Of course, it also gives an unprofessional impression if it
>>> happens to a bona fide user. Ideally you should cover all known
>>> failure modes at development time, and supply user-friendly
>>> responses for them, rather than exposing the end-users to what
>>> are _supposed_ to be your internal diagnostics.

>>
>> I'm distributing a CGI script to largely novise users (mostly on
>> shared hosting accounts without access to the error logs...),

>
> I'm afraid you misinterpreted my meaning of the term 'user' here.

Yes, so it seems.

> Naturally, those who are installing the script should get these
> kind of diagnostics until the scripts are working.

Okay, then we are talking about adding the recommendation to the
installation instructions that the fatalsToBrowser() routine is
commented out when the script is up and running. I take it as a
_warning_ to be _considered_. :)

You mentioned the risk for helping possible intruders as a reason to
do so.

I can see two reasons against it:

- Web hosts offering shared accounts often have the bad habit of
changing things without informing their customers. When that happens,
it's useful to see the resulting error messages.

- It would make the installation a little more difficult. (In my case,
fatalsToBrowser() is called from each one of 15+ *.pl scripts.)

So, basically we are talking about security vs. convenience. So far, I
have concluded that - _in my particular case_ - the convenience aspect
carries greater weight.

--
Gunnar Hjalmarsson
Email: [url]http://www.gunnar.cc/cgi-bin/contact.pl[/url]

[Tad McClellan]

Alan J. Flavell <flavell@mail.cern.ch> wrote:

> we had a hacker into our system for a couple of months
> before we became aware of him (and, as far as we could diagnose
> afterwards, we only became aware of him as an accident

The "Star Wars secrets for cocaine" hackers of the 1980s were
tripped up due to a $0.75 discrepency in billing for computer time.

The book about that incident, "The Cuckoo's Egg" (Clifford Stoll),
is a very good read. Kept me up until 3am two nights...


--
Tad McClellan SGML consulting
[email]tadmc@augustmail.com[/email] Perl programming
Fort Worth, Texas