illegal seek

[Dave Saville]

I was debugging a totally unrelated problem yesterday and put in a
quick and dirty print line with $! in it. Now as it happened the bit of
code I was trying to debug was not giving the error I thought it was
and so $! of course had a previous value.

What printed out was "Illegal Seek" - looking backwards I found a "seek
HANDLE,0,0; Now I have never seen any code where the return from seek
is checked although I guess it should be :-)

The actual code is as follows:

open LIST, "<$clean_username" or die "$clean_username $!";
flock LIST, $LOCK_EX;
seek LIST, 0, 0;

I had read that one should re seek after flock in case another process
modified the file between you opening it and flock coming back with the
exclusive lock.

Or does seek give an error if it is already where one wants to seek to?

TIA


Regards

Dave Saville

NB switch saville for nospam in address

[news@roaima.freeserve.co.uk]

Dave Saville <dave.nospam@ntlworld.com> wrote:

> What printed out was "Illegal Seek" - looking backwards I found a "seek
> HANDLE,0,0

The error value is only set on an error: it's never implicitly reset. So,
can you be sure that this is the relevant seek that generated the
error? What happened when you /did/ check the return value from it?

> The actual code is as follows:

> open LIST, "<$clean_username" or die "$clean_username $!";

The "<" is implicit, and so can be omitted. This leaves a quoted simple
variable, so the quotes can now also be stripped:

open LIST, $clean_username or die "can't open $clean_username: $!";

> flock LIST, $LOCK_EX;

Depending on the underlying implementation of flock and your cross
platform requirements, you cannot /guarantee/ an exclusive lock with
read-only file access. (Mostly it will work, but just be aware...)

> seek LIST, 0, 0;

> I had read that one should re seek after flock in case another process
> modified the file between you opening it and flock coming back with the
> exclusive lock.

You usually need a seek (even if it's to the current position, ick!) when
changing from reading to writing on the same file, as this flushes the
I/O buffers. However, if you've not read anything then there's no point
in rewinding the stream as it's already there. A flock is applied to the
entire file, not just to a section, so when you get the lock you've got
the file in its entirety.

> Or does seek give an error if it is already where one wants to seek to?

No, it's quite happy with that. What it does object to is trying to seek
on a pipe (e.g. stdin) or non-seekable device (e.g. tape). Maybe your
file $clean_username is actually a named pipe?

Cheers,
Chris
--
@s=split(//,"Je,\nhn ersloak rcet thuarP");$k=$l=@s;for(;$k;$k--){$i=($i+1)%$l
until$s[$i];$c=$s[$i];print$c;undef$s[$i];$i=($i+(ord$c))%$l}

[Anno Siegel]

Dave Saville <dave.nospam@ntlworld.com> wrote in comp.lang.perl.misc:

> I was debugging a totally unrelated problem yesterday and put in a
> quick and dirty print line with $! in it. Now as it happened the bit of
> code I was trying to debug was not giving the error I thought it was
> and so $! of course had a previous value.
>
> What printed out was "Illegal Seek" - looking backwards I found a "seek
> HANDLE,0,0;

$! is meaningless, unless immediately after the return from an
failed system call. The "Illegal seek" value may or may not come
from the call in your code, nobody knows.

> Now I have never seen any code where the return from seek
> is checked although I guess it should be :-)

Then you haven't seen enough code. Of course you must check seek() if
it's important. You also ought to check the flock() call.

> The actual code is as follows:
>
> open LIST, "<$clean_username" or die "$clean_username $!";
> flock LIST, $LOCK_EX;

"$LOCK_EX" looks fishy. The constant exported by Fcntl.pm is "LOCK_EX".
Are you sure this is literally your code?

Anyway, you don't want an exclusive lock on a read-only file. Many
systems (Solaris, for instance) won't even give you one. That's one
of the reasons why it's important to check the call to flock().

> seek LIST, 0, 0;
>
> I had read that one should re seek after flock in case another process
> modified the file between you opening it and flock coming back with the
> exclusive lock.

That's okay, though your code has bigger problems.

> Or does seek give an error if it is already where one wants to seek to?

No. But, as noted, forget about the error code, it can come from
anywhere.

Anno

[Dave Saville]

On Mon, 8 Sep 2003 10:01:43 +0100, [email]news@roaima.freeserve.co.uk[/email] wrote:

>Dave Saville <dave.nospam@ntlworld.com> wrote:

>> What printed out was "Illegal Seek" - looking backwards I found a "seek
>> HANDLE,0,0

>
>The error value is only set on an error: it's never implicitly reset. So,
>can you be sure that this is the relevant seek that generated the
>error? What happened when you /did/ check the return value from it?
>

Because it was just before - I have not got around to changing anything
before I fully understood what was going on.

>> The actual code is as follows:

>

>> open LIST, "<$clean_username" or die "$clean_username $!";

>
>The "<" is implicit, and so can be omitted. This leaves a quoted simple
>variable, so the quotes can now also be stripped:
>
> open LIST, $clean_username or die "can't open $clean_username: $!";
>

>> flock LIST, $LOCK_EX;

>
>Depending on the underlying implementation of flock and your cross
>platform requirements, you cannot /guarantee/ an exclusive lock with
>read-only file access. (Mostly it will work, but just be aware...)

I am now *really* confused - I do not understand about flock and read
only - surely the use of a lock is to protect from concurrent usage - I
am taking an exclusive lock because it is entirely possible that
another program could be updating the same file when I want to read it
- and of course I need to process it in a consistent state.

I was taking an exclusive lock on reading <$file and writing - either

>$file or >>$file

So how *should* I be doing it?

Regards

Dave Saville

NB switch saville for nospam in address

[Anno Siegel]

Dave Saville <dave.nospam@ntlworld.com> wrote in comp.lang.perl.misc:

> On Mon, 8 Sep 2003 10:01:43 +0100, [email]news@roaima.freeserve.co.uk[/email] wrote:

> >Dave Saville <dave.nospam@ntlworld.com> wrote:

[...]

> >Depending on the underlying implementation of flock and your cross
> >platform requirements, you cannot /guarantee/ an exclusive lock with
> >read-only file access. (Mostly it will work, but just be aware...)

>
> I am now *really* confused - I do not understand about flock and read
> only - surely the use of a lock is to protect from concurrent usage - I
> am taking an exclusive lock because it is entirely possible that
> another program could be updating the same file when I want to read it
> - and of course I need to process it in a consistent state.

That's fine. If the updating process gets itself an exclusive lock,
it will have to wait till the reading process gives up its shared lock.
That's how file locking works.

> I was taking an exclusive lock on reading <$file and writing - either

> >$file or >>$file

>
> So how *should* I be doing it?

Get shared locks for reading and exclusive locks for writing.

Anno

[Alan J. Flavell]

On Mon, Sep 8, Anno Siegel inscribed on the eternal scroll:

> Get shared locks for reading and exclusive locks for writing.

Indeed. However, AIUI anyone who's reading a file in order to
subsequently update it will need an exclusive lock from the start.

cheers

[Dave Saville]

On 8 Sep 2003 13:13:44 GMT, Anno Siegel wrote:

>> I was taking an exclusive lock on reading <$file and writing - either

>> >$file or >>$file

>>
>> So how *should* I be doing it?

>
>Get shared locks for reading and exclusive locks for writing.
>
>Anno

Thank's - I was used to locking on other languages/systems and there
you took exclusive all round.

Regards

Dave Saville

NB switch saville for nospam in address

[Anno Siegel]

Dave Saville <dave.nospam@ntlworld.com> wrote in comp.lang.perl.misc:

> On 8 Sep 2003 13:13:44 GMT, Anno Siegel wrote:
>

> >> I was taking an exclusive lock on reading <$file and writing - either
> >> >$file or >>$file
> >>
> >> So how *should* I be doing it?

> >
> >Get shared locks for reading and exclusive locks for writing.
> >
> >Anno

>
> Thank's - I was used to locking on other languages/systems and there
> you took exclusive all round.

So multiple reads on a locked file aren't possible in those languages?
That sounds less than optimal.

Anno

[Abigail]

Dave Saville (dave.nospam@ntlworld.com) wrote on MMMDCLX September
MCMXCIII in <URL:news:qnirfnivyyragyjbeyqpbz.hkwrp36.pminews@t ext.news.ntlworld.com>:
$$ On 8 Sep 2003 13:13:44 GMT, Anno Siegel wrote:
$$
$$ >> I was taking an exclusive lock on reading <$file and writing - either
$$ >> >$file or >>$file
$$ >>
$$ >> So how *should* I be doing it?
$$ >
$$ >Get shared locks for reading and exclusive locks for writing.
$$ >
$$ >Anno
$$
$$ Thank's - I was used to locking on other languages/systems and there
$$ you took exclusive all round.


While it's entirely possible to have a degenerate language that
doesn't give you access to shared locking, whether exclusively
locking is possible on a file open for reading is an OS matter,
not a language matter.

If the OS doesn't allow exclusive locks on file open for reading,
not all the King's languages can get such a lock.


Abigail
--
perl -e '$_ = q *4a75737420616e6f74686572205065726c204861636b65720 a*;
for ($*=******;$**=******;$**=******) {$**=*******s*..*qq}
print chr 0x$& and q
qq}*excess********}'

[news@roaima.freeserve.co.uk]

Alan J. Flavell <flavell@mail.cern.ch> wrote:

>> Get shared locks for reading and exclusive locks for writing.

> Indeed. However, AIUI anyone who's reading a file in order to
> subsequently update it will need an exclusive lock from the start.

Yes - but in this case you'd open the file for updating (with "<+ $file"),
so it's effectively an exclusive write lock that you'd need here anyway.

Chris
--
@s=split(//,"Je,\nhn ersloak rcet thuarP");$k=$l=@s;for(;$k;$k--){$i=($i+1)%$l
until$s[$i];$c=$s[$i];print$c;undef$s[$i];$i=($i+(ord$c))%$l}

[Alan J. Flavell]

On Tue, Sep 9, [email]news@roaima.freeserve.co.uk[/email] inscribed on the eternal scroll:

> Alan J. Flavell <flavell@mail.cern.ch> wrote:

> >> Get shared locks for reading and exclusive locks for writing.

>

> > Indeed. However, AIUI anyone who's reading a file in order to
> > subsequently update it will need an exclusive lock from the start.

>
> Yes - but in this case you'd open the file for updating (with "<+ $file"),
> so it's effectively an exclusive write lock that you'd need here anyway.

Yes, I agree: but I've heard people saying that it would take them a
long time to read their file, but adding a new record at the end would
be fast, so can't they take a shared lock to read the file, and then
somehow upgrade it to an exclusive lock to add the new record?
Sounds a plausible idea, but I don't think it can be done in those
terms.

cheers

[news@roaima.freeserve.co.uk]

Alan J. Flavell <flavell@mail.cern.ch> wrote:

> Yes, I agree: but I've heard people saying that it would take them a
> long time to read their file, but adding a new record at the end would
> be fast, so can't they take a shared lock to read the file, and then
> somehow upgrade it to an exclusive lock to add the new record?

Hmm (puts thinking hat on). What about this (with dies omitted):

open (F, "<+ somefile");
flock F, LOCK_SH;
# Do lots of reads

flock F, LOCK_EX;
seek F, END_OF_FILE; # Can't remember symbolic offhand
# Do lots of writes at EOF

close F;

Chris
--
@s=split(//,"Je,\nhn ersloak rcet thuarP");$k=$l=@s;for(;$k;$k--){$i=($i+1)%$l
until$s[$i];$c=$s[$i];print$c;undef$s[$i];$i=($i+(ord$c))%$l}

[Anno Siegel]

Alan J. Flavell <flavell@mail.cern.ch> wrote in comp.lang.perl.misc:

> On Tue, Sep 9, [email]news@roaima.freeserve.co.uk[/email] inscribed on the eternal scroll:
>

> > Alan J. Flavell <flavell@mail.cern.ch> wrote:

> > >> Get shared locks for reading and exclusive locks for writing.

> >

> > > Indeed. However, AIUI anyone who's reading a file in order to
> > > subsequently update it will need an exclusive lock from the start.

> >
> > Yes - but in this case you'd open the file for updating (with "<+ $file"),
> > so it's effectively an exclusive write lock that you'd need here anyway.

>
> Yes, I agree: but I've heard people saying that it would take them a
> long time to read their file, but adding a new record at the end would
> be fast, so can't they take a shared lock to read the file, and then
> somehow upgrade it to an exclusive lock to add the new record?
> Sounds a plausible idea, but I don't think it can be done in those
> terms.

That's a good description of the scenario where a lock upgrade looks
tempting. But, as you say, it can't be done. To get a write lock, you'd
have to give up the read lock, which means someone else can change the
file before you do. End of story, until an OS gives us LOCK_UPGD.

Anno

[Alan J. Flavell]

On Tue, Sep 9, [email]news@roaima.freeserve.co.uk[/email] inscribed on the eternal scroll:

> Hmm (puts thinking hat on). What about this (with dies omitted):
>
> open (F, "<+ somefile");
> flock F, LOCK_SH;
> # Do lots of reads
>
> flock F, LOCK_EX;
> seek F, END_OF_FILE; # Can't remember symbolic offhand

[...]

First off, it seems one can't actually do that. I don't see it
clearly ruled out in the Perl documentation, and I can't testify to it
from any personal knowledge of the internals, but if I try a man page
for flock (i.e the underlying OS implementation) it says:

A single file may not simultaneously have both shared and
exclusive locks.

So, on *that* rather popular unix-ish OS, which also claims
conformance to BSD, you'd have to let-go the shared lock before you
could get the exclusive lock, thus blowing the scheme out of the
water.

Secondly: even if it _was_ feasible, I believe there's a very high
risk of creating a deadlock situation if two simultaneous processes
apply the proposed technique. Looks to me as if it would need a
back-off strategy in most practical situations - just possibly if
you're certain that although there can be many readers there will only
ever be one read/writer at a time, the plan might work - but in view
of the problem with flock itself, I think you'd need an alternative
mechanism, maybe based on semaphore file(s)(?)

But I certainly could be wrong, I can assure you that I've made
mistakes with locking before :-}

[Vlad Tepes]

Anno Siegel <anno4000@lublin.zrz.tu-berlin.de> wrote:

....

> But, as you say, it can't be done. To get a write lock, you'd
> have to give up the read lock, which means someone else can change the
> file before you do. End of story, until an OS gives us LOCK_UPGD.

Are you sure about this? I've tried Chris's idea. It seems to be
possible to upgrade the lock from LOCK_SH to LOCK_EX without unlocking
first, but I could be missing something.

Here are the two scripts I tried, followed by output from a small
command line session:

lockupgrade.pl:

#!/usr/bin/perl -w

use strict;
use Fcntl qw( :flock :seek );
$|++;

open F, "+<", "lock.txt" or die "$0 open : $!";
print "\n$0: waiting for shared lock...";
flock F, LOCK_SH or die "$0 flocksh : $!";
print "\n$0: got shared lock\n";
sleep 10;

print "$0: waiting for exclusive lock...\n";
flock F, LOCK_EX or die "$0 flockex: $!";
print "$0: got exclusive lock!\n";
seek F, SEEK_END, 2 or die "$0 seekend: $!";
print F "\nwritten by $0 at @{[ scalar localtime ]}\n";
flock F, LOCK_UN or die "$0 cannot unlock: $!";
close F or die "$0 closeex: $!";

__END__


locksh.pl:

#!/usr/bin/perl

use warnings;
use strict;
use Fcntl qw( :flock :seek );

open F, "lock.txt" or die "open : $!";
print "$0: waiting for shared lock..\n";
flock F, LOCK_SH or die "$0 flocksh: $!";
print "$0: got shared lock!....\n";
sleep 10;
flock F, LOCK_UN or die "$0 unlock: $!";
print "$0: released readlock!....\n";
close F;

__END__

Here's what I did, together with output:

$ perl lockupg.pl &; sleep 2; perl locksh.pl ; sleep 3

[1] 5265
lockupg.pl: waiting for shared lock...
lockupg.pl: got shared lock
locksh.pl: waiting for shared lock..
locksh.pl: got shared lock!....
lockupg.pl: waiting for exclusive lock...
locksh.pl: released readlock!....
lockupg.pl: got exclusive lock!
[1] + 5265 done perl lockupg.pl
$ cat lock.txt
la
lala
lalala
lalalala

written by lockupg.pl at Tue Sep 9 19:33:44 2003
$

--
Vlad

[Anno Siegel]

Vlad Tepes <minceme@start.no> wrote in comp.lang.perl.misc:

> Anno Siegel <anno4000@lublin.zrz.tu-berlin.de> wrote:
>
> ...
>

> > But, as you say, it can't be done. To get a write lock, you'd
> > have to give up the read lock, which means someone else can change the
> > file before you do. End of story, until an OS gives us LOCK_UPGD.

>
> Are you sure about this? I've tried Chris's idea. It seems to be
> possible to upgrade the lock from LOCK_SH to LOCK_EX without unlocking
> first, but I could be missing something.
>
> Here are the two scripts I tried, followed by output from a small
> command line session:
>
> lockupgrade.pl:
>
> #!/usr/bin/perl -w
>
> use strict;
> use Fcntl qw( :flock :seek );
> $|++;
>
> open F, "+<", "lock.txt" or die "$0 open : $!";
> print "\n$0: waiting for shared lock...";
> flock F, LOCK_SH or die "$0 flocksh : $!";
> print "\n$0: got shared lock\n";
> sleep 10;
>
> print "$0: waiting for exclusive lock...\n";
> flock F, LOCK_EX or die "$0 flockex: $!";
> print "$0: got exclusive lock!\n";
> seek F, SEEK_END, 2 or die "$0 seekend: $!";
> print F "\nwritten by $0 at @{[ scalar localtime ]}\n";
> flock F, LOCK_UN or die "$0 cannot unlock: $!";
> close F or die "$0 closeex: $!";
>
> __END__

[mode code snipped]

This looks like an upgrade, and it is even called an upgrade in the
flock man page, but it isn't an upgrade in the sense that the process
seamlessly keeps a lock on the file:

A shared lock may be upgraded to an exclusive lock, and vice versa, sim-
ply by specifying the appropriate lock type; this results in the previous
lock being released and the new lock applied (possibly after other pro-
cesses have gained and released the lock).

Still end of story, no LOCK_UPGD yet.

Anno

[news@roaima.freeserve.co.uk]

Anno Siegel <anno4000@lublin.zrz.tu-berlin.de> wrote:

> That's a good description of the scenario where a lock upgrade looks
> tempting. But, as you say, it can't be done. To get a write lock, you'd
> have to give up the read lock, which means someone else can change the
> file before you do. End of story, until an OS gives us LOCK_UPGD.

Hmm. It gets worse - and not just in perl but also in native C (I've
tested both). When you "upgrade" a flock from LOCK_SH to LOCK_EX the
kernel actually releases the lock transiently whilst *appearing* to
succeed atomically. Ouch.

The issue here (as far as I'm concerned) is that flock *appears* to let
you upgrade atomically but it doesn't really do so. Can this warning be
added explicitly to perldoc for flock?

(This is on GNU/Linux kernel 2.4.20. I haven't checked Solaris.)

Chris

[Anno Siegel]

<chris-usenet@roaima.co.uk> wrote in comp.lang.perl.misc:

> Anno Siegel <anno4000@lublin.zrz.tu-berlin.de> wrote:

> > That's a good description of the scenario where a lock upgrade looks
> > tempting. But, as you say, it can't be done. To get a write lock, you'd
> > have to give up the read lock, which means someone else can change the
> > file before you do. End of story, until an OS gives us LOCK_UPGD.

>
> Hmm. It gets worse - and not just in perl but also in native C (I've
> tested both). When you "upgrade" a flock from LOCK_SH to LOCK_EX the
> kernel actually releases the lock transiently whilst *appearing* to
> succeed atomically. Ouch.
>
> The issue here (as far as I'm concerned) is that flock *appears* to let
> you upgrade atomically but it doesn't really do so. Can this warning be
> added explicitly to perldoc for flock?

This doesn't belong in perldoc -- it depends on the underlying OS and
isn't a property of Perl. The documentation of all system calls in
Perl must be read in conjunction with the man page of that call.

In the case of flock(), the man page is quite explicit that "lock
upgrading" implies giving up the previous lock and acquiring a new one
(with the consequence that other processes can get access to the file
in between). The choice of the term "upgrading" is certainly unfortunate
for that action.

Anno

[Alan J. Flavell]

On Wed, Sep 10, Anno Siegel inscribed on the eternal scroll:

> <chris-usenet@roaima.co.uk> wrote in comp.lang.perl.misc:

> > The issue here (as far as I'm concerned) is that flock *appears* to let
> > you upgrade atomically but it doesn't really do so. Can this warning be
> > added explicitly to perldoc for flock?

>
> This doesn't belong in perldoc -- it depends on the underlying OS and
> isn't a property of Perl.

With respect, that's an over-purist approach to documentation. As
we've seen in discussion, this was a very tempting approach that some
programmers believe in good faith is feasible: at the very least, the
Perl documentation which the Perl programmer is supposed to consult
should draw attention to the potential issue, and direct them to look
at the portability documentation, where they should then be able to
find relevant details for their particular OS.

But anyone with an eye to portable programming (and this _is_ one of
Perl's strengths, after all) should be able to select a portable
programming technique without being forced to trawl through the
portability documentation for umpteen OSes with which they're
otherwise totally unfamiliar. Am I being unreasonable?

> The documentation of all system calls in
> Perl must be read in conjunction with the man page of that call.

Sorry, but I stand by my point. Someone amongst the perlporters is in
a position to know about the OSes that are unfamiliar to me as a
programmer: I want to be able to select a portable programming
technique "at the point of service", i.e in this case starting out
from Perl's own documentation relating to file locking, based on their
advice, without having to do my own research on numerous unfamiliar
OSes. Whether it's from perldoc -f flock, or from perlopentut, or
from the relevant FAQ, in a practical sense I want to be directed to
any specific caveats that I need. Telling me to read the man pages
for umpteen OSes that I don't have and am completely unfamiliar with
does not answer that requirement, do you see what I mean?

best regards

[Anno Siegel]

Alan J. Flavell <flavell@mail.cern.ch> wrote in comp.lang.perl.misc:

> On Wed, Sep 10, Anno Siegel inscribed on the eternal scroll:
>

> > <chris-usenet@roaima.co.uk> wrote in comp.lang.perl.misc:

>

> > > The issue here (as far as I'm concerned) is that flock *appears* to let
> > > you upgrade atomically but it doesn't really do so. Can this warning be
> > > added explicitly to perldoc for flock?

> >
> > This doesn't belong in perldoc -- it depends on the underlying OS and
> > isn't a property of Perl.

>
> With respect, that's an over-purist approach to documentation. As
> we've seen in discussion, this was a very tempting approach that some
> programmers believe in good faith is feasible: at the very least, the
> Perl documentation which the Perl programmer is supposed to consult
> should draw attention to the potential issue, and direct them to look
> at the portability documentation, where they should then be able to
> find relevant details for their particular OS.
>
> But anyone with an eye to portable programming (and this _is_ one of
> Perl's strengths, after all) should be able to select a portable
> programming technique without being forced to trawl through the
> portability documentation for umpteen OSes with which they're
> otherwise totally unfamiliar. Am I being unreasonable?

Unreasonable, you? You must be kidding! You are a rock of reason
in the sea of Usenet.

> > The documentation of all system calls in
> > Perl must be read in conjunction with the man page of that call.

>
> Sorry, but I stand by my point.

It is, however, how the Perl documentation proper deals with system
calls. "perldoc -f exec" basically points you at the man pages for
execvp(3) and sh(1), which must be frustrating when your system has
neither. The get* chapters do the same, quite summarily.

> Someone amongst the perlporters is in
> a position to know about the OSes that are unfamiliar to me as a
> programmer: I want to be able to select a portable programming
> technique "at the point of service", i.e in this case starting out
> from Perl's own documentation relating to file locking, based on their
> advice, without having to do my own research on numerous unfamiliar
> OSes. Whether it's from perldoc -f flock, or from perlopentut, or
> from the relevant FAQ, in a practical sense I want to be directed to
> any specific caveats that I need. Telling me to read the man pages
> for umpteen OSes that I don't have and am completely unfamiliar with
> does not answer that requirement, do you see what I mean?

Oh, I do see what you mean. File locking is one of the functions that
suffer most from portability issues. A nice matrix that shows flock()
features against Perl ports would be a meritorious project, but someone
has to do it, and maintain it.

At the moment, I'd stick to the most elementary locking methods if
portability is an issue. It doesn't pay to cut corners, even if it
"works" on one system. A false sense of security may be the result.

If that it unbearably slow for some reason, I'd try to optimize it
for the system(s) it is running on, turning to the relevant basic
man pages.

Anno