473,847 Members | 1,500 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

pickle, cPickle, & HIGHEST_PROTOCO L

I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCO L of pickle and cPickle. I am getting an error
message and trace backs if the list exceeds eight items. Whether I use
pickle or cPickle does not matter, i.e., the eight number causes a
problem in both modules, although the trace backs are of course
dissimilar.

This pickling and unpickling of the list of dictionaries worked when I
stopped using the HIGHEST_PROTOCO L in both modules (pickle and
cPickle), which got Python to use the ASCII format (I suppose) as I can
read the pickled data.

This behavior was observed in Python 2.3.4 (final), and 2.4 (final) on
Win98.

Any comments?
Regards,
Khalid

# Sample program tester.py begin
!
! import pickle as pkl
! import os
! #-----------------------------
! def test_pickle():
! fn = 'rkeys.txt'
! f = file(fn, 'r')
! lines = f.readlines()
! f.close()
! _test_list = []
! for line in lines:
! sline = line.split(',')
! #print sline
! key, value = sline[0], sline[1].strip()
! _test_dict = {}
! _test_dict[key] = value
! _test_list.appe nd(_test_dict)
!
! # Let's see the contents of our object
! print _test_list
!
! # Then pickle it
! f = file('pkl_' + fn, 'w')
! pkl.dump(_test_ list, f, pkl.HIGHEST_PRO TOCOL)
! f.close()
!
! # Empty it
! _test_list = []
! print _test_list
!
! # Unpickling object here:
! f = file('pkl_' + fn, 'r')
! _test_list = pkl.load(f)
! f.close()
!
! # See contents after loading
! print _test_list
!#-----------------------------
!if __name__ == '__main__':
! test_pickle()
!
!# Sample program end
# Contents of file rkeys.txt (without the triple quotes):
"""
'1','v1'
'2','v2'
'3','v3'
'4','v4'
'5','v5'
'6','v6'
'7','v7'
'8','v8'
'9','v9'
"""
# Output (without the triple quotes)
# Using "import pickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py" , line 41, in ?
! test_pickle()
! File "tester.py" , line 34, in test_pickle
! _test_list = pkl.load(f)
! File "D:\PY23\PYTHON \DIST\SRC\lib\p ickle.py", line 1390, in load
! return Unpickler(file) .load()
! File "D:\PY23\PYTHON \DIST\SRC\lib\p ickle.py", line 872, in load
! dispatch[key](self)
! File "D:\PY23\PYTHON \DIST\SRC\lib\p ickle.py", line 1189, in
load_binput
! i = ord(self.read(1 ))
!TypeError: ord() expected a character, but string of length 0 found
"""
# Output (without the triple quotes)
# Using "import cPickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py" , line 41, in ?
! test_pickle()
! File "tester.py" , line 34, in test_pickle
! _test_list = pkl.load(f)
!EOFError
"""
# Output (without the triple quotes)
# Using "import cPickle as pkl", or "import pickle as pkl"
# but _not_ using the HIGHEST_PROTOCO L:
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
"""

Jul 18 '05 #1
1 3512
[A.B., Khalid]
I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCO L of pickle and cPickle. .... ... on Win98.


Pickles are binary data. Therefore you should open pickle files in
binary mode on all platforms, and must open them in binary mode on
Windows. That is, replace your lines:

f = file('pkl_' + fn, 'w')
f = file('pkl_' + fn, 'r')

with:

f = file('pkl_' + fn, 'wb')
f = file('pkl_' + fn, 'rb')

It's true that you can get away with opening pickle files in text mode
on Windows if you stick to the default pickle protocol (0), although
in that case your pickle files won't be portable across platforms.
For that reason, don't try to be clever: always use binary-mode files
to store pickles, regardless of pickle protocol in use.
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2045
by: Guenter Walser | last post by:
Hello, When using the codeline: pickle.dump(self, file, pickle.HIGHEST_PROTOCOL) my program runs perfectly fine: --------------------------- Testprotocol -----------------------------
1
6351
by: Jesse Bloom | last post by:
I keep running into a problem when I use pickle or cPickle to unpickle a python object I have serialized from the database. I read the string for the serialized object. Apparently I am missing an argument? Here's what happens: cPickle.loads(s) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: __new__() takes exactly 2 arguments (1 given)
3
4024
by: Michael Hohn | last post by:
Hi, under python 2.2, the pickle/unpickle sequence incorrectly restores a larger data structure I have. Under Python 2.3, these structures now give an explicit exception from Pickle.memoize(): assert id(obj) not in self.memo I'm shrinking the offending data structure down to find the problem
8
8149
by: Hans Georg Krauthaeuser | last post by:
Dear all, I have a long running application (electromagnetic compatibility measurements in mode-stirred chambers over GPIB) that use pickle (cPickle) to autosave a class instance with all the measured data from time to time. At the beginning, pickling is quite fast but when the data becomes more and more pickling slows down rapidly.
1
1558
by: Philippe C. Martin | last post by:
Hi, How would one go about extracting the pickle module from Python (ex: to make a .a or a .dll) ? Thanks Philippe
2
2652
by: Boris Borcic | last post by:
Assuming that the items of my_stream share no content (they are dumps of db cursor fetches), is there a simple way to do the equivalent of def pickles(my_stream) : from cPickle import load,dumps while 1 : yield dumps(load(my_stream)) without the overhead associated with unpickling objects
8
2326
by: Victor Kryukov | last post by:
Hello list, I've found the following strange behavior of cPickle. Do you think it's a bug, or is it by design? Best regards, Victor. from pickle import dumps from cPickle import dumps as cdumps
2
6567
by: Victor Lin | last post by:
Hi, I encounter a problem with pickle. I download a html from: http://www.amazon.com/Magellan-Maestro-4040-Widescreen-Navigator/dp/B000NMKHW6/ref=sr_1_2?ie=UTF8&s=electronics&qid=1202541889&sr=1-2 and parse it with BeautifulSoup. This page is very huge. When I use pickle to dump it, a RuntimeError: maximum recursion depth
1
6312
by: Nagu | last post by:
I didn't have the problem with dumping as a string. When I tried to save this object to a file, memory error pops up. I am sorry for the mention of size for a dictionary. What I meant by 65000X50 is that it has 65000 keys and each key has a list of 50 tuples. I was able to save a dictionary object with 65000 keys and a list of 15-tuple values to a file. But I could not do the same when I have a list of 25-tuple values for 65000 keys.
0
9892
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9734
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10995
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10654
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10718
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
1
7888
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7061
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5729
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4130
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.