Unique file names.

[John McKown]

I have a bit of a problem and I'm wondering if there is a better solution.
I have a Perl program (of course) which reads STDIN and breaks it into
separate files based on a "report separator". I can construct an
appropriate file name based on information in the "trailing" separator.
I.e. I don't know the name of the report until the report is complete.
What I am doing at present is using the routine 'tempfile' in the
File::Temp package to create a unique file in the output directory. Once
I get the report name from the "trailer" separator, I want to rename this
file to the correct name. However, it is possible that there will be
multiple, different reports coming in which have the same name. So, what I
want to do is put a "sequence number" as the last node in the file name.
This means that I must generate a test file name, then see if that file
already exists. If it does, increment the sequence number, generate a new
test file name and loop. Oh, did I mention that it is possible for this
program to have several copies all running at the same time, possibly
producing different reports with identical names? So I must worry about
"race" conditions. My code, so far, looks like:

# $fn contains the file name generated by 'tempname'
# $report contains the report name
my ($number,$test);
for ($number=1;;$number++) {
$test = "$report.$number";
last if sysopen JUNK, $test, O_WRONLY|O_EXCL|O_CREAT;
}
close JUNK;
mv($fn,$test);

Using sysopen with those parameters, especially O_EXCL, is the only way
that I can think to ensure that the name in $test does not exist and is
created "atomically" so that no other incarnation of this program which
happens to be running at the same time will exit the for loop with the
same value in $test. I then close this file (now a 0 length file). I then
use the "mv" function from File::Copy to rename my real data file to the
new name, replacing the 0 length file.

This is running on Linux. I don't know if the sysopen can guarantee the
"atomicity" of sysopen on other platforms.

All comments gratefully received.

--
Maranatha!
John McKown

[Randy W. Sims]

On 2/12/2004 9:53 PM, John McKown wrote:

> I have a bit of a problem and I'm wondering if there is a better solution.
> I have a Perl program (of course) which reads STDIN and breaks it into
> separate files based on a "report separator". I can construct an
> appropriate file name based on information in the "trailing" separator.
> I.e. I don't know the name of the report until the report is complete.
> What I am doing at present is using the routine 'tempfile' in the
> File::Temp package to create a unique file in the output directory. Once
> I get the report name from the "trailer" separator, I want to rename this
> file to the correct name. However, it is possible that there will be
> multiple, different reports coming in which have the same name. So, what I
> want to do is put a "sequence number" as the last node in the file name.
> This means that I must generate a test file name, then see if that file
> already exists. If it does, increment the sequence number, generate a new
> test file name and loop. Oh, did I mention that it is possible for this
> program to have several copies all running at the same time, possibly
> producing different reports with identical names? So I must worry about
> "race" conditions.

If your code is the only code accessing the file, the standard way to
handle this is to create a lock file. Try to write a file to a specific
place as a lock file; if it fails, another process has already created
it, so you poll until you can create it. Then run your routine to get a
filename, move the file, and delete the lock file so that the next
process can run.

Randy.

[Rob Dixon]

John McKown wrote:

>
> I have a bit of a problem and I'm wondering if there is a better solution.
> I have a Perl program (of course) which reads STDIN and breaks it into
> separate files based on a "report separator". I can construct an
> appropriate file name based on information in the "trailing" separator.
> I.e. I don't know the name of the report until the report is complete.
> What I am doing at present is using the routine 'tempfile' in the
> File::Temp package to create a unique file in the output directory. Once
> I get the report name from the "trailer" separator, I want to rename this
> file to the correct name. However, it is possible that there will be
> multiple, different reports coming in which have the same name. So, what I
> want to do is put a "sequence number" as the last node in the file name.
> This means that I must generate a test file name, then see if that file
> already exists. If it does, increment the sequence number, generate a new
> test file name and loop. Oh, did I mention that it is possible for this
> program to have several copies all running at the same time, possibly
> producing different reports with identical names? So I must worry about
> "race" conditions. My code, so far, looks like:
>
> # $fn contains the file name generated by 'tempname'
> # $report contains the report name
> my ($number,$test);
> for ($number=1;;$number++) {
> $test = "$report.$number";
> last if sysopen JUNK, $test, O_WRONLY|O_EXCL|O_CREAT;
> }
> close JUNK;
> mv($fn,$test);
>
> Using sysopen with those parameters, especially O_EXCL, is the only way
> that I can think to ensure that the name in $test does not exist and is
> created "atomically" so that no other incarnation of this program which
> happens to be running at the same time will exit the for loop with the
> same value in $test. I then close this file (now a 0 length file). I then
> use the "mv" function from File::Copy to rename my real data file to the
> new name, replacing the 0 length file.
>
> This is running on Linux. I don't know if the sysopen can guarantee the
> "atomicity" of sysopen on other platforms.

As far as I know 'rename' is atomic, so:

my $number;

do {
$number++;
my $test = "$report.$number";
} until rename $fn, $test;

HTH,

Rob

[John McKown]

On Fri, 13 Feb 2004, Rob Dixon wrote:

>
> As far as I know 'rename' is atomic, so:
>
> my $number;
>
> do {
> $number++;
> my $test = "$report.$number";
> } until rename $fn, $test;
>
> HTH,
>
> Rob

My Perl book indicates that if a file with the name in $test exists, it is
destroyed and replaced. I don't want to replace an existing file. Every
other method other than sysopen seems to have this effect as well. I.e.
rename, File::Copy "move", etc.

--
Maranatha!
John McKown