pickle broken: can't handle NaN or Infinity under win32

[Grant Edwards]

I finally figured out why one of my apps sometimes fails under
Win32 when it always works fine under Linux: Under Win32, the
pickle module only works with a subset of floating point
values. In particular the if you try to dump/load an infinity
or nan value, the load operation chokes:
Under Linux:

$ python
Python 2.3.4 (#2, Feb 9 2005, 14:22:48)
[GCC 3.4.1 (Mandrakelinux 10.1 3.4.1-4mdk)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

$ python pickletest.py
(inf, nan) (inf, nan)
Under Win32:

$ python
ActivePython 2.3.4 Build 233 (ActiveState Corp.) based on
Python 2.3.4 (#53, Oct 18 2004, 20:35:07) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.

$ python pickletest.py
Traceback (most recent call last):
File "pickletest.py" , line 8, in ?
d = pickle.loads(s)
File "C:\PYTHON23\li b\pickle.py", line 1394, in loads
return Unpickler(file) .load()
File "C:\PYTHON23\li b\pickle.py", line 872, in load
dispatch[key](self)
File "C:\PYTHON23\li b\pickle.py", line 968, in load_float
self.append(flo at(self.readlin e()[:-1]))
ValueError: invalid literal for float(): 1.#INF

I realize that this is probably due to underlying brokenness in
the Win32 libc implimentation, but should the pickle module
hide such platform-dependancies from the user?

Best case, it would be nice if pickle could handle all floats
in a portable way.

Worst case, shouldn't the pickle module documentation mention
that pickling floats non-portable or only partially implimented?

On a more immediate note, are there hooks in pickle to allow
the user to handle types that pickle can't deal with? Or, do I
have to throw out pickle and write something from scratch?

[NaN and Infinity are prefectly valid (and extremely useful)
floating point values, and not using them would require huge
complexity increases in my apps (not using them would probably
at least triple the amount of code required in some cases).]

--
Grant Edwards grante Yow! Yow!
at
visi.com

[Scott David Daniels]

Grant Edwards wrote:

On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote:

Several issues:

(1) The number of distinct NaNs varies among platforms.

According to the IEEE standard, there are exactly two:
signalling and quiet, and on platforms that don't impliment
floating point exceptions (probably in excess of 99.9% of
python installations), the difference between the two is moot.

But it does not specify the representation of such NaNs.
Negative 0 isn't a NaN, it's just negative 0. Right, but it is hard to construct in standard C.

(2) There is no standard-conforming way to create these values.

What standard are you looking at? My copy of the IEEE 754
standard is pretty clear.

I was talking about the C89 standard. In the absence of a C
standard way to create, manipulate, and test for these values,
you must implement and test platform-by-platform.
The bit patterns are defined by the IEEE 754 standard. Perhaps this is right and I misunderstand the standard, but my
understanding is that the full bit pattern is not, in fact,
defined.
If there are Python-hosting platoforms that don't use IEEE 754 as
the floating point representation, then that can be dealt with. There are.
Python has _tons_ of platform-specific code in it. But the _tons_ are written in C89 C.
Why all of a sudden is it taboo for Python to impliment
something that's not universally portable and defined in a
standard? Where's the standard defining Python?

It is not taboo. I am trying to explain why it is not a
trivial task, but a substantial effort. If you are willing
to perform the substantial effort, good on you, and I'll help.
If you simply want to implement on the two platforms you use,
and want everyone else to implement the interface you choose,
that seems to me an unreasonable request.

--Scott David Daniels
Sc***********@A cm.Org

[Paul Rubin]

Scott David Daniels <Sc***********@ Acm.Org> writes:

Negative 0 isn't a NaN, it's just negative 0.

Right, but it is hard to construct in standard C.

Huh? It's just a hex constant.

[Scott David Daniels]

Paul Rubin wrote:

Scott David Daniels <Sc***********@ Acm.Org> writes:

Negative 0 isn't a NaN, it's just negative 0.

Right, but it is hard to construct in standard C.

Huh? It's just a hex constant.

Well, -0.0 doesn't work, and (double)0x80000 000 doesn't work,
and.... I think you have to use quirks of a compiler to create
it. And I don't know how to test for it either, x < 0.0 is
not necessarily true for negative 0.

I am not trying to say there is no way to do this. I am
trying to say it takes thought and effort on every detail,
in the definition, implementations , and unit tests.

--Scott David Daniels
Sc***********@A cm.Org

[Paul Rubin]

Scott David Daniels <Sc***********@ Acm.Org> writes:

Negative 0 isn't a NaN, it's just negative 0.

Right, but it is hard to construct in standard C.

Huh? It's just a hex constant.

Well, -0.0 doesn't work, and (double)0x80000 000 doesn't work,
and.... I think you have to use quirks of a compiler to create
it. And I don't know how to test for it either, x < 0.0 is
not necessarily true for negative 0.

Aren't we talking about IEEE 754 arithmetic? There's some specific
bit pattern(s) for -0.0 and you can assign a float variable to
such a pattern.

[Tim Peters]

[with the start of US summer comes the start of 754 ranting season]

[Grant Edwards]

Negative 0 isn't a NaN, it's just negative 0.
[Scott David Daniels] Right, but it is hard to construct in standard C.
[Paul Rubin]

Huh? It's just a hex constant.

[Scott David Daniels] Well, -0.0 doesn't work,
C89 doesn't define the result of that, but "most" C compilers these
days will create a negative 0.
and (double)0x80000 000 doesn't work,
In part because that's an integer <wink>, and in part because it's
only 32 bits. It requires representation casting tricks (not
conversion casting tricks like the above), knowledge of the platform
endianness, and knowledge of the platform integer sizes. Assuming the
platform uses 754 bit layout to begin with, of course.
and.... I think you have to use quirks of a compiler to create
it.
You at least need platform knowledge. It's really not hard, if you
can assume enough about the platform.
And I don't know how to test for it either, x < 0.0 is
not necessarily true for negative 0.
If it's a 754-conforming C compiler, that's necessarily false (+0 and
-0 compare equal in 754). Picking the bits apart is again the closest
thing to a portable test. Across platforms with a 754-conforming
libm, the most portable way is via using atan2(!):
pz = 0.0
mz = -pz
from math import atan2
atan2(pz, pz) 0.0 atan2(mz, mz) -3.1415926535897 931

It's tempting to divide into 1, then check the sign of the infinity,
but Python stops you from doing that:
1/pz

Traceback (most recent call last):
File "<stdin>", line 1, in ?
ZeroDivisionErr or: float division

That can't be done at the C level either, because _some_ people run
Python with their 754 HW floating-point zero-division, overflow, and
invalid operation traps enabled, and then anything like division by 0
causes the interpreter to die. The CPython implementation is
constrained that way.

Note that Python already has Py_IS_NAN and Py_IS_INFINITY macros in
pyport.h, and the Windows build maps them to appropriate
Microsoft-specific library functions. I think it's stuck waiting on
others to care enough to supply them for other platforms. If a
platform build doesn't #define them, a reasonable but cheap attempt is
made to supply "portable" code sequences for them, but, as the
pyport.h comments note, they're guaranteed to do wrong things in some
cases, and may not work at all on some platforms. For example, the
default

#define Py_IS_NAN(X) ((X) != (X))

is guaranteed never to return true under MSVC 6.0.
I am not trying to say there is no way to do this. I am
trying to say it takes thought and effort on every detail,
in the definition, implementations , and unit tests.

It's par for the course -- everyone thinks "this must be easy" at
first, and everyone who persists eventually gives up. Kudos to
Michael Hudson for persisting long enough to make major improvements
here in pickle, struct and marshal for Python 2.5!

[Grant Edwards]

On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote:

Grant Edwards wrote:

On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote:

Several issues:

(1) The number of distinct NaNs varies among platforms.

According to the IEEE standard, there are exactly two:
signalling and quiet, and on platforms that don't impliment
floating point exceptions (probably in excess of 99.9% of
python installations), the difference between the two is moot.

But it does not specify the representation of such NaNs.

Yes, it does. It specifies it exactly: certain bits are ones,
certain other bits are zeros. I don't know how much more exactly
the representation can be defined.

The bit patterns are defined by the IEEE 754 standard. Perhaps this is right and I misunderstand the standard, but my
understanding is that the full bit pattern is not, in fact,
defined.
The represntation of NaNs, infinities, normalized numbers and
denormal numbers are all completely defined by the standard.
If there are Python-hosting platoforms that don't use IEEE 754 as
the floating point representation, then that can be dealt with. There are.
That's where it gets nasty.
Python has _tons_ of platform-specific code in it.

But the _tons_ are written in C89 C.
True.
It is not taboo. I am trying to explain why it is not a
trivial task, but a substantial effort.
It's trivial for platforms that obey the IEEE 754 standard.
If you are willing to perform the substantial effort, good on
you, and I'll help. If you simply want to implement on the two
platforms you use, and want everyone else to implement the
interface you choose, that seems to me an unreasonable
request.

I would think that implimenenting things according to the IEEE
standard and letting non-standard platforms figure out what to
do for themselves would seem a reasonable approach.

--
Grant Edwards grante Yow! Now I understand the
at meaning of "THE MOD SQUAD"!
visi.com

[Grant Edwards]

On 2005-06-22, Paul Rubin <http> wrote:

Negative 0 isn't a NaN, it's just negative 0.

Right, but it is hard to construct in standard C.

Huh? It's just a hex constant.

Yup. There are two ways to construct a NaN. One is to do
something like (1e300*1e300)/(1e300*1e300) and hope for the
best. The other is to assume IEEE 754 just use 7f800000 or
7fc00000 depending on whether you want a signalling or quiet
NaN.

--
Grant Edwards grante Yow! Don't hit me!! I'm in
at the Twilight Zone!!!
visi.com

[Grant Edwards]

On 2005-06-23, Paul Rubin <http> wrote:

Scott David Daniels <Sc***********@ Acm.Org> writes:

>>>Negative 0 isn't a NaN, it's just negative 0.
>>
>>Right, but it is hard to construct in standard C.
> Huh? It's just a hex constant. Well, -0.0 doesn't work, and (double)0x80000 000 doesn't work,
and.... I think you have to use quirks of a compiler to create
it. And I don't know how to test for it either, x < 0.0 is
not necessarily true for negative 0.

Aren't we talking about IEEE 754 arithmetic?

Mainly, yes.
There's some specific bit pattern(s) for -0.0 and you can
assign a float variable to such a pattern.

Yup.

--
Grant Edwards grante Yow! My Aunt MAUREEN was
at a military advisor to IKE &
visi.com TINA TURNER!!

[Grant Edwards]

On 2005-06-23, Tim Peters <ti********@gma il.com> wrote:

C89 doesn't define the result of that, but "most" C compilers these
days will create a negative 0.

and (double)0x80000 000 doesn't work,
I think you meant something like

float f;
*((uint32_t*)&d ) = 0xNNNNNNNN;
And I don't know how to test for it either, x < 0.0 is not
necessarily true for negative 0.

If it's a 754-conforming C compiler, that's necessarily false (+0 and
-0 compare equal in 754). Picking the bits apart is again the closest
thing to a portable test. Across platforms with a 754-conforming
libm, the most portable way is via using atan2(!):

[brain-bending example elided]

It's probobly because of the domains in which I work, but I
don't think I've ever cared whether a zero is positive or
negative. I understand why it's easier to impliment things
that way, but I don't see why anybody would care. OTOH, NaNs
and Infinities are indisposable for real-world stuff.

I am not trying to say there is no way to do this. I am
trying to say it takes thought and effort on every detail, in
the definition, implementations , and unit tests.

It's par for the course -- everyone thinks "this must be easy"
at first, and everyone who persists eventually gives up.
Kudos to Michael Hudson for persisting long enough to make
major improvements here in pickle, struct and marshal for
Python 2.5!

I would think it doable if one assumed IEEE-754 FP (famous last
words). I suppose there are still a few VAX machines around.
And there are things like TI DSPs that don't use IEEE-754.

--
Grant Edwards grante Yow! RELAX!!... This
at is gonna be a HEALING
visi.com EXPERIENCE!! Besides,
I work for DING DONGS!

[Grant Edwards]

On 2005-06-23, Grant Edwards <gr****@visi.co m> wrote:

On 2005-06-23, Tim Peters <ti********@gma il.com> wrote:

C89 doesn't define the result of that, but "most" C compilers these
days will create a negative 0.

and (double)0x80000 000 doesn't work,

I think you meant something like

float f;
*((uint32_t*)&d ) = 0xNNNNNNNN;

*((uint32_t*)&f ) = 0xNNNNNNNN;

It doesn't matter how many times one proofreads things like
that...

--
Grant Edwards grante Yow! I will establish
at the first SHOPPING MALL in
visi.com NUTLEY, New Jersey...