I finally figured out why one of my apps sometimes fails under
Win32 when it always works fine under Linux: Under Win32, the
pickle module only works with a subset of floating point
values. In particular the if you try to dump/load an infinity
or nan value, the load operation chokes:
Under Linux:
$ python
Python 2.3.4 (#2, Feb 9 2005, 14:22:48)
[GCC 3.4.1 (Mandrakelinux 10.1 3.4.1-4mdk)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.
$ python pickletest.py
(inf, nan) (inf, nan)
Under Win32:
$ python
ActivePython 2.3.4 Build 233 (ActiveState Corp.) based on
Python 2.3.4 (#53, Oct 18 2004, 20:35:07) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.
$ python pickletest.py
Traceback (most recent call last):
File "pickletest.py" , line 8, in ?
d = pickle.loads(s)
File "C:\PYTHON23\li b\pickle.py", line 1394, in loads
return Unpickler(file) .load()
File "C:\PYTHON23\li b\pickle.py", line 872, in load
dispatch[key](self)
File "C:\PYTHON23\li b\pickle.py", line 968, in load_float
self.append(flo at(self.readlin e()[:-1]))
ValueError: invalid literal for float(): 1.#INF
I realize that this is probably due to underlying brokenness in
the Win32 libc implimentation, but should the pickle module
hide such platform-dependancies from the user?
Best case, it would be nice if pickle could handle all floats
in a portable way.
Worst case, shouldn't the pickle module documentation mention
that pickling floats non-portable or only partially implimented?
On a more immediate note, are there hooks in pickle to allow
the user to handle types that pickle can't deal with? Or, do I
have to throw out pickle and write something from scratch?
[NaN and Infinity are prefectly valid (and extremely useful)
floating point values, and not using them would require huge
complexity increases in my apps (not using them would probably
at least triple the amount of code required in some cases).]
--
Grant Edwards grante Yow! Yow!
at
visi.com 28 2678
On 2005-06-21, Grant Edwards <gr****@visi.co m> wrote: I finally figured out why one of my apps sometimes fails under Win32 when it always works fine under Linux
[...]
Oh, I forgot, here's pickletest.py:
#!/usr/bin/python
import pickle
f1 = (1e300*1e300)
f2 = f1/f1
o = (f1,f2)
s = pickle.dumps(o)
d = pickle.loads(s)
print o,d
$ python pickletest.py (inf, nan) (inf, nan)
Under Win32:
$ python pickletest.py Traceback (most recent call last): File "pickletest.py" , line 8, in ? d = pickle.loads(s) File "C:\PYTHON23\li b\pickle.py", line 1394, in loads return Unpickler(file) .load() File "C:\PYTHON23\li b\pickle.py", line 872, in load dispatch[key](self) File "C:\PYTHON23\li b\pickle.py", line 968, in load_float self.append(flo at(self.readlin e()[:-1])) ValueError: invalid literal for float(): 1.#INF
--
Grant Edwards grante Yow! Here I am in 53
at B.C. and all I want is a
visi.com dill pickle!!
Grant Edwards wrote: I finally figured out why one of my apps sometimes fails under Win32 when it always works fine under Linux: Under Win32, the pickle module only works with a subset of floating point values. In particular the if you try to dump/load an infinity or nan value, the load operation chokes:
There is no completely portable way to do this. Any single platform
can have a solution, but (since the C standards don't address how
NaNs and Infs are represented) there is not a good portable way to do
the pickle / unpickle. It is nice the exception is raised, since at
one point it was not (and a simple 1.0 was returned).
See explanations in article 654866: http://sourceforge.net/tracker/index...70&atid=105470
$ python pickletest.py Traceback (most recent call last): ... File "C:\PYTHON23\li b\pickle.py", line 968, in load_float self.append(flo at(self.readlin e()[:-1])) ValueError: invalid literal for float(): 1.#INF
I realize that this is probably due to underlying brokenness in the Win32 libc implimentation, but should the pickle module hide such platform-dependancies from the user?
As mentioned above, there is no C standard-accessible way to
predictably build or represent NaNs, negative zeroes, or Infinities.
[NaN and Infinity are prefectly valid (and extremely useful) floating point values, and not using them would require huge complexity increases in my apps (not using them would probably at least triple the amount of code required in some cases).]
You could check to see if the Python 2.5 pickling does a better
job. Otherwise, you've got your work cut out for you.
-Scott David Daniels Sc***********@A cm.Org
On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote: I finally figured out why one of my apps sometimes fails under Win32 when it always works fine under Linux: Under Win32, the pickle module only works with a subset of floating point values. In particular the if you try to dump/load an infinity or nan value, the load operation chokes: There is no completely portable way to do this.
Python deals with all sorts of problems for which there is no
completely portable solution. Remember: "practicali ty beats purity."
Any single platform can have a solution, but (since the C standards don't address how NaNs and Infs are represented) there is not a good portable way to do the pickle / unpickle.
Likewise, there is no completely portable python
implimentation. Any single platform can have a Python
implimentation, but since the C standards don't address a
universal standard for "a computer" there is not a good
portable way to do Python. I guess we'd better give up on
Python. :)
It is nice the exception is raised, since at one point it was not (and a simple 1.0 was returned).
That would be even worse. [NaN and Infinity are prefectly valid (and extremely useful) floating point values, and not using them would require huge complexity increases in my apps (not using them would probably at least triple the amount of code required in some cases).]
You could check to see if the Python 2.5 pickling does a better job. Otherwise, you've got your work cut out for you.
Fixing it is really quite trivial. It takes less than a dozen
lines of code. Just catch the exception and handle it.
def load_float(self ):
s = self.readline()[:-1]
try:
f = float(s)
except ValueError:
s = s.upper()
if s in ["1.#INF", "INF"]:
f = 1e300*1e300
elif s in ["-1.#INF", "-INF"]:
f = -1e300*1e300
elif s in ["NAN","1.#QNAN" ,"QNAN","1.#IND ","IND","-1.#IND"]:
f = -((1e300*1e300)/(1e300*1e300))
else:
raise ValueError, "Don't know what to do with "+`s`
self.append(f)
Obviously the list of accepted string values should be expanded
to include other platforms as needed. The above example
handles Win32 and glibc (e.g. Linux).
Even better, add that code to float().
--
Grant Edwards grante Yow! Is the EIGHTIES
at when they had ART DECO
visi.com and GERALD McBOING-BOING
lunch boxes??
Grant Edwards wrote: On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote:...Under Win32, the pickle module only works with a subset of floating point values. In particular ... infinity or nan ...
There is no completely portable way to do this.
Python deals with all sorts of problems for which there is no completely portable solution. Remember: "practicali ty beats purity."
Any single platform can have a solution, but (since the C standards don't address how NaNs and Infs are represented) there is not a good portable way to do the pickle / unpickle. ... Fixing it is really quite trivial. It takes less than a dozen lines of code. Just catch the exception and handle it.
Since you know it is quite trivial, and I don't, why not submit a
patch resolving this issue. Be sure to include tests for all
supported Python platforms.
--Scott David Daniels Sc***********@A cm.Org
On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote: Fixing it is really quite trivial. It takes less than a dozen lines of code. Just catch the exception and handle it.
Since you know it is quite trivial, and I don't, why not submit a patch resolving this issue. Be sure to include tests for all supported Python platforms.
I'm working on it. I should have said it's trivial if you have
access to the platforms to be supported. I've tested a fix
that supports pickle streams generated under Win32 and glibc.
That's using the "native" string representation of a NaN or
Inf.
A perhaps simpler approach would be to define a string
representation for Python to use for NaN and Inf. Just because
something isn't defined by the C standard doesn't mean it can't
be defined by Python.
--
Grant Edwards grante Yow! I'm shaving!! I'M
at SHAVING!!
visi.com
Grant Edwards wrote: I'm working on it. I should have said it's trivial if you have access to the platforms to be supported. I've tested a fix that supports pickle streams generated under Win32 and glibc. That's using the "native" string representation of a NaN or Inf.
Several issues:
(1) The number of distinct NaNs varies among platforms. There are
quiet and signaling NaNs, negative 0, the NaN that Windows VC++
calls "Indeterminate, " and so on.
(2) There is no standard-conforming way to create these values.
(3) There is no standard-conforming way to detect these values.
--Scott David Daniels Sc***********@A cm.Org
"Grant Edwards" <gr****@visi.co m> wrote in message
news:11******** *****@corp.supe rnews.com... I'm working on it. I should have said it's trivial if you have access to the platforms to be supported. I've tested a fix that supports pickle streams generated under Win32 and glibc. That's using the "native" string representation of a NaN or Inf.
A perhaps simpler approach would be to define a string representation for Python to use for NaN and Inf. Just because something isn't defined by the C standard doesn't mean it can't be defined by Python.
I believe that changes have been made to marshal/unmarshal in 2.5 CVS with
respect to NAN/INF to eliminate annoying/surprising behavior differences
between corresponding .py and .pyc files. Perhaps these revisions would be
relevant to pickle changes.
TJR
On 2005-06-22, Scott David Daniels <Sc***********@ Acm.Org> wrote: I'm working on it. I should have said it's trivial if you have access to the platforms to be supported. I've tested a fix that supports pickle streams generated under Win32 and glibc. That's using the "native" string representation of a NaN or Inf. Several issues:
(1) The number of distinct NaNs varies among platforms.
According to the IEEE standard, there are exactly two:
signalling and quiet, and on platforms that don't impliment
floating point exceptions (probably in excess of 99.9% of
python installations), the difference between the two is moot.
There are quiet and signaling NaNs, negative 0,
Negative 0 isn't a NaN, it's just negative 0.
the NaN that Windows VC++ calls "Indeterminate, " and so on.
That's just Microsoft's way of spelling "signalling NaN."
(2) There is no standard-conforming way to create these values.
What standard are you looking at? My copy of the IEEE 754
standard is pretty clear.
(3) There is no standard-conforming way to detect these values.
The bit patterns are defined by the IEEE 754 standard. If
there are Python-hosting platoforms that don't use IEEE 754 as
the floating point representation, then that can be dealt with.
Python has _tons_ of platform-specific code in it.
Why all of a sudden is it taboo for Python to impliment
something that's not universally portable and defined in a
standard? Where's the standard defining Python?
--
Grant Edwards grante Yow! ... A housewife
at is wearing a polypyrene
visi.com jumpsuit!!
"Grant Edwards" <gr****@visi.co m> wrote in message
news:11******** *****@corp.supe rnews.com... The bit patterns are defined by the IEEE 754 standard. If there are Python-hosting platoforms that don't use IEEE 754 as the floating point representation, then that can be dealt with.
Python has _tons_ of platform-specific code in it.
More, I believe, than the current maintainers would like. Adding more
would probably require a commitment to maintain the addition (respond to
bug reports) for a few years.
Why all of a sudden is it taboo for Python to impliment something that's not universally portable and defined in a standard?
??
Perhaps you wrote this before reading my last post reporting that some
NaN/Inf changes have already been made for 2.5. I believe that more would
be considered if properly submitted.
Where's the standard defining Python?
The Language and Library Reference Manuals at python.org.
Terry J. Reedy This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Michael Hohn |
last post by:
Hi,
under python 2.2, the pickle/unpickle sequence incorrectly restores
a larger data structure I have.
Under Python 2.3, these structures now give an explicit exception from
Pickle.memoize():
assert id(obj) not in self.memo
I'm shrinking the offending data structure down to find the problem
|
by: simonwittber |
last post by:
I've written a simple module which serializes these python types:
IntType, TupleType, StringType, FloatType, LongType, ListType, DictType
It available for perusal here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/415503
It appears to work faster than pickle, however, the decode process is
much slower (5x) than the encode process. Has anyone got any tips on
|
by: Mike P. |
last post by:
Hi all,
I'm working on a simulation (can be considered a game) in Python where I
want to be able to dump the simulation state to a file and be able to load
it up later. I have used the standard Python pickle module and it works fine
pickling/unpickling from files.
However, I want to be able to use a third party tool like an XML editor (or
other custom tool) to setup the initial state of the simulation, so I have
been playing around...
|
by: Shi Mu |
last post by:
I got a sample code and tested it but really can not understand the
use of pickle and dump:
>>> import pickle
>>> f = open("try.txt", "w")
>>> pickle.dump(3.14, f)
>>> pickle.dump(, f)
>>> f.close()
|
by: Maurice LING |
last post by:
Hi,
I need to look into serialization for python objects, including codes,
recursive types etc etc. Currently, I have no idea exactly what needs to
be serialized, so my scope is to be as wide as possible.
I understand that marshal is extended by pickle to serialize class
instances, shared elements, and recursive data structures
(http://www.effbot.org/librarybook/pickle.htm) but cannot handle code
types. pickle can be used together...
| |
by: John Salerno |
last post by:
Here's what I have:
import pickle
data = open(r'C:\pickle_data.txt', 'w')
image = open(r'C:\peakhell.jpg')
pickle.Pickler(data)
data.dump(image)
data.close()
image.close()
|
by: Bart Ogryczak |
last post by:
Hello,
I´ve got this problem with pickle, it seems it doesn´t handle
correctly infinite values (nor does Python return overflow/underflow
error). What could I do about it? Example code:
<type 'float'>
1.#INF
ValueError: invalid literal for float(): 1.#INF
|
by: Pierre Rouleau |
last post by:
Hi all,
When using Python 2.4.x on a Win32 box,
marshal.loads(marshal.dumps(1e66666)) returns 1.0 instead of infinity
as it should and does under Python 2.5 (also running on Win32 ).
This problem was reported in another thread here by Peter Hansen
http://groups-beta.google.com/group/comp.lang.python/browse_frm/thread/5c2b4b2a88c8df4/f216739705c9304f?lnk=gst&q=simplejson&rnum=5#f216739705c9304f
Is this considered an important enough...
|
by: DwBear75 |
last post by:
I am contemplating the need for a way to handle high speed data
passing between two processes. One process would act as a queue that
would 'buffer' data coming from another processes. Seems that the
easiest way to handle the data would be to just pass pickles. Further,
I'm thinking that using a unix domain socket would make this a simple
way to pass high volumes of pickles. Are there any examples of an
architecture like these, where a...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |