#40749 [NEW]: pack and unpack erroneous behavior on 64bits hosts

[ben at ateor dot com]

From: ben at ateor dot com
Operating system: OpenBSD amd64 and sparc64
PHP version: 5.2.1
PHP Bug Type: Reproducible crash
Bug description: pack and unpack erroneous behavior on 64bits hosts

Description:
------------
This is a follow-up on #40543 ([url]http://bugs.php.net/bug.php?id=40543[/url],
since
that bug is closed, I can't add comments).
Please note : it's not identical to #4053 (other weird behaviors
are demonstrated).

Iliaa,
Not sure why you suggest to use little endian or host conversions
routines,
but in my standpoint if you reverse twice a number's byte ordering
then you should get the original number back (assuming the number don't
overflows php's internals).

At least, that's the standard behavior for perl and python.

Beside, I can't see why php should handles those endianness conversions
differently on an i386 (32 bits) and on an x86_64 (64 bits), both
having the same byte order.

The following on a 64bit, little endian host :
x86_64$ uname -mprsv
OpenBSD 4.0 GENERIC#0 amd64 AMD Sempron(tm) Processor 3400+

x86_64$ perl -e 'print unpack("N", pack("N", 41445)) ."\n"'
41445

x86_64$ python
Python 2.4.3 (#1, Sep 6 2006, 20:33:08)
[GCC 3.3.5 (propolice)] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.

>>> from struct import *
>>> unpack('>L', pack('>L', 41445))

(41445L,)

And, indeed :
#include <stdio.h>
#include <sys/types.h>
int main(void)
{
u_int32_t x, y, z; /* 32 bits unsigned longs */
x = 41445;
/* conv host (little) to network (big endian) long : pack("N",
41445) */
y = htonl(x);
/* conv network (big endian) to host (little) long : unpack("N",
....) */
z = ntohl(y);
printf("Host : %li\nBig : %li\nHost : %li\n", x, y, z);
return 0;
}

x86_64$ gcc conv.c -o conv ; ./conv
Host : 41445
Big : -442433536
Host : 41445

But still (PHP 5.2.2-dev (cli) (built: Feb 27 2007 22:10:11)) :
x86_64$ php -r 'print_r(unpack("N", pack("N", 41445)));'
Array
(
[1] => -2147442203
)

While on a plain old x86 little endian host (PHP 4.4.0), we get
a different result :
i386_32$ uname -mprsv
OpenBSD 3.9 GENERIC#0 i386 Intel(R) Pentium(R) 4 CPU 1.70GHz
("GenuineIntel" 686-class)
i386_32$ php -r 'print_r(unpack("N", pack("N", 41445)));'
Array
(
[1] => 41445
)

Still on the 64 bits little endian host :
x86_64$ php -r '$a = unpack("N", pack("N", 65536)); printf("$a[1]\n");'
65536 # Ok
x86_64$ php -r '$a = unpack("N", pack("N", 65535)); printf("$a[1]\n");'
-2147418113 # Weird

x86_64$ php -r '$a = unpack("N", pack("N", 0x1234)); printf("0x%x\n",
$a[1]);'
0x1234 # Ok
x86_64$ php -r '$a = unpack("L", pack("N", 0x1234)); printf("0x%x\n",
$a[1]);'
0x34120000 # Ok
x86_64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffff # Ok
x86_64$ php -r '$a = unpack("N", pack("N", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffffffff8000ffff # The doc says "N" gives you "always 32 bit", and we
get
# 8 bytes. No wonder why we overflow.
x86_64$ php -r '$a = unpack("N", pack("N", 0xff )); printf("0x%x\n",
$a[1]);'
0xffffffff800000ff # Same. Don't tell me 0xff is too large.

And now, all the following tests are on a 64 bits _big endian_ host
(sparc64, running php-5.2.1) :
sparc64$ uname -mprsv
OpenBSD 3.8 GENERIC#607 sparc64 SUNW,UltraSPARC-IIi @ 440 MHz, version 0
FPU
sparc64$ php -r '$a = unpack("N", pack("N", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffff # Ok
# The same, but prefixing 0000 to the argument :
sparc64$ php -r '$a = unpack("N", pack("N", 0x0000ffff)); printf("0x%x\n",
$a[1]);'
0xffffffff8000ffff
# Weird (and with "N", we stayed on the host byte order this time).
# Shouldn't 0xffff == 0x0000ffff, even on big endian ? Apparently, yes :
sparc64$ php -r 'printf("0x%x\n", 0x0000ffff);'
0xffff

# Also, look at this :
sparc64$ php -r '$a = unpack("N", pack("N", 41445)); printf("$a[1]\n");'
41445
# And now let's just remove the line feed (\n) from the above printf :
sparc64$ php -r '$a = unpack("N", pack("N", 41445)); printf("$a[1]");'
-2147442203

# Same for 2^16 -1 / 65535 / 0xfff :
sparc64$ php -r '$a = unpack("N", pack("N", 65535)); printf("$a[1]\n");'
65535
sparc64$ php -r '$a = unpack("N", pack("N", 65535)); printf("$a[1]");'
-2147418113

# We get the opposite (bogus with \n, correct without) when converting
# to little endian and back to host :
sparc64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf( $a[1]);'
65535
sparc64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf(
$a[1]."\n");'
-2147418113


This doesn't help :
SKIP Generic pack()/unpack() tests [ext/standard/tests/strings/pack.phpt]
reason: 32bit test only




--
Edit bug report at [url]http://bugs.php.net/?id=40749&edit=1[/url]
--
Try a CVS snapshot (PHP 4.4): [url]http://bugs.php.net/fix.php?id=40749&r=trysnapshot44[/url]
Try a CVS snapshot (PHP 5.2): [url]http://bugs.php.net/fix.php?id=40749&r=trysnapshot52[/url]
Try a CVS snapshot (PHP 6.0): [url]http://bugs.php.net/fix.php?id=40749&r=trysnapshot60[/url]
Fixed in CVS: [url]http://bugs.php.net/fix.php?id=40749&r=fixedcvs[/url]
Fixed in release: [url]http://bugs.php.net/fix.php?id=40749&r=alreadyfixed[/url]
Need backtrace: [url]http://bugs.php.net/fix.php?id=40749&r=needtrace[/url]
Need Reproduce Script: [url]http://bugs.php.net/fix.php?id=40749&r=needscript[/url]
Try newer version: [url]http://bugs.php.net/fix.php?id=40749&r=oldversion[/url]
Not developer issue: [url]http://bugs.php.net/fix.php?id=40749&r=support[/url]
Expected behavior: [url]http://bugs.php.net/fix.php?id=40749&r=notwrong[/url]
Not enough info: [url]http://bugs.php.net/fix.php?id=40749&r=notenoughinfo[/url]
Submitted twice: [url]http://bugs.php.net/fix.php?id=40749&r=submittedtwice[/url]
register_globals: [url]http://bugs.php.net/fix.php?id=40749&r=globals[/url]
PHP 3 support discontinued: [url]http://bugs.php.net/fix.php?id=40749&r=php3[/url]
Daylight Savings: [url]http://bugs.php.net/fix.php?id=40749&r=dst[/url]
IIS Stability: [url]http://bugs.php.net/fix.php?id=40749&r=isapi[/url]
Install GNU Sed: [url]http://bugs.php.net/fix.php?id=40749&r=gnused[/url]
Floating point limitations: [url]http://bugs.php.net/fix.php?id=40749&r=float[/url]
No Zend Extensions: [url]http://bugs.php.net/fix.php?id=40749&r=nozend[/url]
MySQL Configuration Error: [url]http://bugs.php.net/fix.php?id=40749&r=mysqlcfg[/url]

[windeler at mediafinanz dot de]

ID: 40749
Comment by: windeler at mediafinanz dot de
Reported By: ben at ateor dot com
Status: Open
Bug Type: Unknown/Other Function
Operating System: OpenBSD amd64 and sparc64
PHP Version: 5.2.1
New Comment:

Here is another example of a problem with unpack on 64bit systems. It
worked in 5.1.6, but with 5.2.1 the results are bogus.

The expected value from the file content is 200, but PHP says
-2147483448 when I echo $a['i'].

<?php
$f = fopen('test.pdf','rb');
//Read a 4-byte integer from file
$a = unpack('Ni',fread($f,4));
echo $a['i'];
fclose($f);
?>


Previous Comments:
------------------------------------------------------------------------

[2007-03-07 17:12:58] ben at ateor dot com

Description:
------------
This is a follow-up on #40543 ([url]http://bugs.php.net/bug.php?id=40543[/url],
since
that bug is closed, I can't add comments).
Please note : it's not identical to #4053 (other weird behaviors
are demonstrated).

Iliaa,
Not sure why you suggest to use little endian or host conversions
routines,
but in my standpoint if you reverse twice a number's byte ordering
then you should get the original number back (assuming the number
don't
overflows php's internals).

At least, that's the standard behavior for perl and python.

Beside, I can't see why php should handles those endianness
conversions
differently on an i386 (32 bits) and on an x86_64 (64 bits), both
having the same byte order.

The following on a 64bit, little endian host :
x86_64$ uname -mprsv
OpenBSD 4.0 GENERIC#0 amd64 AMD Sempron(tm) Processor 3400+

x86_64$ perl -e 'print unpack("N", pack("N", 41445)) ."\n"'
41445

x86_64$ python
Python 2.4.3 (#1, Sep 6 2006, 20:33:08)
[GCC 3.3.5 (propolice)] on openbsd4
Type "help", "copyright", "credits" or "license" for more information.

>>> from struct import *
>>> unpack('>L', pack('>L', 41445))

(41445L,)

And, indeed :
#include <stdio.h>
#include <sys/types.h>
int main(void)
{
u_int32_t x, y, z; /* 32 bits unsigned longs */
x = 41445;
/* conv host (little) to network (big endian) long : pack("N",
41445) */
y = htonl(x);
/* conv network (big endian) to host (little) long :
unpack("N", ...) */
z = ntohl(y);
printf("Host : %li\nBig : %li\nHost : %li\n", x, y, z);
return 0;
}

x86_64$ gcc conv.c -o conv ; ./conv
Host : 41445
Big : -442433536
Host : 41445

But still (PHP 5.2.2-dev (cli) (built: Feb 27 2007 22:10:11)) :
x86_64$ php -r 'print_r(unpack("N", pack("N", 41445)));'
Array
(
[1] => -2147442203
)

While on a plain old x86 little endian host (PHP 4.4.0), we get
a different result :
i386_32$ uname -mprsv
OpenBSD 3.9 GENERIC#0 i386 Intel(R) Pentium(R) 4 CPU 1.70GHz
("GenuineIntel" 686-class)
i386_32$ php -r 'print_r(unpack("N", pack("N", 41445)));'
Array
(
[1] => 41445
)

Still on the 64 bits little endian host :
x86_64$ php -r '$a = unpack("N", pack("N", 65536));
printf("$a[1]\n");'
65536 # Ok
x86_64$ php -r '$a = unpack("N", pack("N", 65535));
printf("$a[1]\n");'
-2147418113 # Weird

x86_64$ php -r '$a = unpack("N", pack("N", 0x1234)); printf("0x%x\n",
$a[1]);'
0x1234 # Ok
x86_64$ php -r '$a = unpack("L", pack("N", 0x1234)); printf("0x%x\n",
$a[1]);'
0x34120000 # Ok
x86_64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffff # Ok
x86_64$ php -r '$a = unpack("N", pack("N", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffffffff8000ffff # The doc says "N" gives you "always 32 bit", and we
get
# 8 bytes. No wonder why we overflow.
x86_64$ php -r '$a = unpack("N", pack("N", 0xff )); printf("0x%x\n",
$a[1]);'
0xffffffff800000ff # Same. Don't tell me 0xff is too large.

And now, all the following tests are on a 64 bits _big endian_ host
(sparc64, running php-5.2.1) :
sparc64$ uname -mprsv
OpenBSD 3.8 GENERIC#607 sparc64 SUNW,UltraSPARC-IIi @ 440 MHz, version
0 FPU
sparc64$ php -r '$a = unpack("N", pack("N", 0xffff)); printf("0x%x\n",
$a[1]);'
0xffff # Ok
# The same, but prefixing 0000 to the argument :
sparc64$ php -r '$a = unpack("N", pack("N", 0x0000ffff));
printf("0x%x\n", $a[1]);'
0xffffffff8000ffff
# Weird (and with "N", we stayed on the host byte order this time).
# Shouldn't 0xffff == 0x0000ffff, even on big endian ? Apparently, yes
:
sparc64$ php -r 'printf("0x%x\n", 0x0000ffff);'
0xffff

# Also, look at this :
sparc64$ php -r '$a = unpack("N", pack("N", 41445));
printf("$a[1]\n");'
41445
# And now let's just remove the line feed (\n) from the above printf :
sparc64$ php -r '$a = unpack("N", pack("N", 41445)); printf("$a[1]");'
-2147442203

# Same for 2^16 -1 / 65535 / 0xfff :
sparc64$ php -r '$a = unpack("N", pack("N", 65535));
printf("$a[1]\n");'
65535
sparc64$ php -r '$a = unpack("N", pack("N", 65535)); printf("$a[1]");'
-2147418113

# We get the opposite (bogus with \n, correct without) when converting
# to little endian and back to host :
sparc64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf( $a[1]);'
65535
sparc64$ php -r '$a = unpack("L", pack("L", 0xffff)); printf(
$a[1]."\n");'
-2147418113


This doesn't help :
SKIP Generic pack()/unpack() tests
[ext/standard/tests/strings/pack.phpt] reason: 32bit test only





------------------------------------------------------------------------


--
Edit this bug report at [url]http://bugs.php.net/?id=40749&edit=1[/url]