Devel::Cover failing with 'bizarre copy of hash in leave' error

[Jim Keenan]

(This message was posted to the perl.qa list on October 13, but for
some reason has not yet appeared.)

I would like to report a problem I am having applying Devel::Cover to
two of my CPAN distributions.

The two distributions in question, Data::Presenter and
Mail::Digest::Tools, have repeatedly passed the automated testing
processes provided by testers.cpan.org. Moreover, ActiveState and
others have been able to make 'ppm' and 'rpm' versions. So the code
tests satisfactorily -- but not when running through the Devel::Cover
module for coverage analysis. I want Devel::Cover to work well
because I want to use it as part of the Phalanx project
([url]http://qa.perl.org/phalanx[/url]).

##### GENERAL PROBLEM #####

Data::Presenter and Mail::Digest::Tools pass all their tests in the
normal installation sequence:

perl Makefile.PL; make; make test

But both distributions fail -- with exactly the same error reported --
when I follow the sequence advised in the Devel::Cover docs for
testing the coverage provided by my test suites:

perl Makefile.PL; make
cover -delete
HARNESS_PERL_SWITCHES=-MDevel::Cover make test

In the case of Data::Presenter, the code fails in each of the five
tests in the test suite with this message:

Bizarre copy of HASH in leave at
~/Data-Presenter-0.66/blib/lib/Data/Presenter.pm line 321.

In the case of Mail::Digest::Tools, the code fails in two of the three
tests in the test suite with this message:

Bizarre copy of HASH in leave at
~/Mail-Digest-Tools-2.1/blib/lib/Mail/Digest/Tools.pm line 542.

According to perldiag, that message is a particular instance of a more
general error message:

Bizarre copy of %s in %s: (P) Perl detected an attempt to copy an
internal value that is not copyable.

perldiag notes that a class (P) error is "An internal error you should
never see (trappable)."

I know what a hash is. I'm not sure what a 'leave' is in this
context, though perlcall has a reference to a leave in connection with
the C-Perl API. And I clearly have no idea what a "bizarre copy" of a
hash is. And please note, this error only appears when testing with
Devel::Cover. If I take the same 'make-d' code and just call 'make
test', everything is A-OK.

##### DETAILED DISCUSSION OF PROBLEMS #####

In Data::Presenter, the coverage/testing process fails in each of five
cases at the same method call, which looks like this:

@columns_selected = ('timeslot', 'instructor', 'ward',
'groupname', 'room', 'groupid');
$sorted_data = $dp->sort_by_column(\@columns_selected);
ok( (1 == sdtest(\@columns_selected, $sorted_data)), 'valid sorted
data hash');# 230

The code is failing *before* (not in) test 230. It's failing inside
method sort_by_column(). Specifically, it's failing inside a private
function called inside method sort_by_column() known as
_key_constructor():

sub _key_constructor {
my ($self, $argsref) = @_;
my @args = @$argsref; # for convenience
my @keys = ();
foreach my $k (keys %$self) {
unless ($reserved{$k}) {
my @temp = ();
foreach my $i (@args) {
push @temp, ${%$self}{$k}->[$fieldlabels{$i}];
### ABOVE IS LINE 321 of Presenter.pm
}
push @temp, $k;
push(@keys, \@temp);
}
}
return @keys;
}

In the case of Data::Presenter's failure, the hash whose copy is
'bizarre' is reported as being in line 321.

In Mail::Digest::Tools, the coverage/testing process fails in two of
the three files in the test suite at the same function call, which
looks like this:

process_new_digests(\%pbml_config_in, \%pbml_config_out);

The point at which the code is failing is, once again, in a private
function inside the public function. In this case, the private
function is called _update_digest_logs() and looks like this:

sub _update_digests_log {
my ($hashlog_ref, $logfile) = @_;
my ($logstring);
foreach ( sort keys %$hashlog_ref ) {
$logstring .= $_ . ';' . ${%$hashlog_ref}{$_}[0] . ';' .
${%$hashlog_ref}{$_}[1]. "\n";
### ABOVE IS LINE 542 in Mail::Digest::Tools.pm
}
open(LOG, ">$logfile")
|| die "cannot open $logfile for writing: $!";
print LOG $logstring;
close(LOG) || die "cannot close $logfile: $!";
}

In the case of Mail::Digest::Tool's failure, the hash whose copy is
'bizarre' is reported as being in line 542.

I could go on, but this is enough for a first posting.

##### MAIN QUESTION #####

Has anyone else had their utilization of Devel::Cover fail with this
'bizarre
copy of hash in leave' error?

Does anyone have a clue as to what is going on?

Thanks in advance.
Jim Keenan

[Jim Keenan]

Jim Keenan wrote:

> I would like to report a problem I am having applying Devel::Cover to
> two of my CPAN distributions.
>
> [snip]
>
> Data::Presenter and Mail::Digest::Tools pass all their tests in the
> normal installation sequence:
>
> perl Makefile.PL; make; make test
>
> But both distributions fail -- with exactly the same error reported --
> when I follow the sequence advised in the Devel::Cover docs for
> testing the coverage provided by my test suites:
>
> perl Makefile.PL; make
> cover -delete
> HARNESS_PERL_SWITCHES=-MDevel::Cover make test
>
> In the case of Data::Presenter, the code fails in each of the five
> tests in the test suite with this message:
>
> Bizarre copy of HASH in leave at
> ~/Data-Presenter-0.66/blib/lib/Data/Presenter.pm line 321.
>
> In the case of Mail::Digest::Tools, the code fails in two of the three
> tests in the test suite with this message:
>
> Bizarre copy of HASH in leave at
> ~/Mail-Digest-Tools-2.1/blib/lib/Mail/Digest/Tools.pm line 542.
>

Once upon a time, a hacker, young in Perl years but not in human years,
wrote some code, largely for his own benefit and use at home and on the
job. The code consistently gave the results intended and, after a
couple of years of use, the hacker modularized the code and put it up on
CPAN for all to use. The modules repeatedly passed cpan-testing. The
hacker even gave a talk about one of those modules at a Perl conference.

Then one day the hacker decided to see how thoroughly his test suites
tested the code in the modules. To do so, he used Paul Johnson's
Devel::Cover module. He had heard Paul talk about Devel::Cover in Paris
in 2003 and heard Andy Lester in Buffalo in 2004 urge its use as well.
Much to the hacker's surprise, his code blew up when tested with
Devel::Cover -- blew up in ways the the normally soft-spoken Mr Perldiag
described as "bizarre."

The hacker put the problem aside for a couple of months, but
subsequently posted messages about his problem. The hacker was more
concerned for what these problems said about Devel::Cover than what they
said about his own modest modules. Sadly, his messages attracted no
response.

Fortunately, the noble knight Sir Google came to the hacker's rescue.
Sir Google revealed that this error message was well discussed by the
sages in an earlier millennium and in more recent times as well. A
discussion thread beginning on July 16, 2002, proved particularly
fruitful. Sage Uri remarked that there existed code which would run and
test correctly under normal circumstances but would blow up under
special circumstances, such as when being run through the Perl debugger.
Sage Benjamin agreed that reports of "bizarre copies" of hashes and
arrays suggested a bug in Perl, but went on to add:

BG>> While it's true that the bug is an internal error,
BG>> it's *provoked* by the user doing something which
BG>> should be a syntax error.

BG>> Fix the syntax error, and the error message goes away.

The hacker, aided by copious quantities of legal mind-altering liquids,
decided to stare at his code some more. Lo and behold, there, in the
very sections of code he posted on comp.lang.perl.modules, lay the problem.

> push @temp, ${%$self}{$k}->[$fieldlabels{$i}];
> ### ABOVE IS LINE 321 of Presenter.pm

> $logstring .= $_ . ';' . ${%$hashlog_ref}{$_}[0] . ';' .
> ${%$hashlog_ref}{$_}[1]. "\n";
> ### ABOVE IS LINE 542 in Mail::Digest::Tools.pm

The hacker, recalling how confused he was by Perl's "line noise" in his
younger years, now realized that he had superfluous '%' characters at
three places in the above code and that the code should have been written:

push @temp, ${$self}{$k}[$fieldlabels{$i}]; # in Data::Presenter

and

$logstring .= $_ . ';' . ${$hashlog_ref}{$_}[0] . ';' .
${$hashlog_ref}{$_}[1]. "\n"; # in Mail::Digest::Tools

The hacker concluded that his earlier code fit into the categories
described by Sages Uri and Benjamin. It *should* have caused a syntax
error, but it didn't, at least not until it was run under unusual
conditions which entailed digging deep into Perl's internals. The
hacker corrected his code and tested it while running Devel::Cover. No
longer did his code blow up. All his tests passed and he got his
coverage analysis from Devel::Cover. In gratitude, he posted updated
versions of his modules to CPAN and composed the note you are now
reading. He concluded by warning all his fellow Perl hackers to beware
of the following constructions:

${%$some_hashref}{$x}
${@$some_arrayref}[$x]
${%$some_hashref}{$x}
${@$some_arrayref}[$x]

Those '%' and '@' characters are superfluous. Your code may compile,
run and test correctly under normal conditions, but it will fall apart
under more unusual conditions, including testing with Devel::Cover.

And here our tale ends.

Jim Keenan