pickle, cPickle, & HIGHEST_PROTOCOL

[A.B., Khalid]

I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCOL of pickle and cPickle. I am getting an error
message and trace backs if the list exceeds eight items. Whether I use
pickle or cPickle does not matter, i.e., the eight number causes a
problem in both modules, although the trace backs are of course
dissimilar.

This pickling and unpickling of the list of dictionaries worked when I
stopped using the HIGHEST_PROTOCOL in both modules (pickle and
cPickle), which got Python to use the ASCII format (I suppose) as I can
read the pickled data.

This behavior was observed in Python 2.3.4 (final), and 2.4 (final) on
Win98.

Any comments?
Regards,
Khalid

# Sample program tester.py begin
!
! import pickle as pkl
! import os
! #-----------------------------
! def test_pickle():
! fn = 'rkeys.txt'
! f = file(fn, 'r')
! lines = f.readlines()
! f.close()
! _test_list = []
! for line in lines:
! sline = line.split(',')
! #print sline
! key, value = sline[0], sline[1].strip()
! _test_dict = {}
! _test_dict[key] = value
! _test_list.append(_test_dict)
!
! # Let's see the contents of our object
! print _test_list
!
! # Then pickle it
! f = file('pkl_' + fn, 'w')
! pkl.dump(_test_list, f, pkl.HIGHEST_PROTOCOL)
! f.close()
!
! # Empty it
! _test_list = []
! print _test_list
!
! # Unpickling object here:
! f = file('pkl_' + fn, 'r')
! _test_list = pkl.load(f)
! f.close()
!
! # See contents after loading
! print _test_list
!#-----------------------------
!if __name__ == '__main__':
! test_pickle()
!
!# Sample program end
# Contents of file rkeys.txt (without the triple quotes):
"""
'1','v1'
'2','v2'
'3','v3'
'4','v4'
'5','v5'
'6','v6'
'7','v7'
'8','v8'
'9','v9'
"""
# Output (without the triple quotes)
# Using "import pickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py", line 41, in ?
! test_pickle()
! File "tester.py", line 34, in test_pickle
! _test_list = pkl.load(f)
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1390, in load
! return Unpickler(file).load()
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 872, in load
! dispatch[key](self)
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1189, in
load_binput
! i = ord(self.read(1))
!TypeError: ord() expected a character, but string of length 0 found
"""
# Output (without the triple quotes)
# Using "import cPickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py", line 41, in ?
! test_pickle()
! File "tester.py", line 34, in test_pickle
! _test_list = pkl.load(f)
!EOFError
"""
# Output (without the triple quotes)
# Using "import cPickle as pkl", or "import pickle as pkl"
# but _not_ using the HIGHEST_PROTOCOL:
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
"""

[Tim Peters]

[A.B., Khalid]

I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCOL of pickle and cPickle. .... ... on Win98.

Pickles are binary data. Therefore you should open pickle files in
binary mode on all platforms, and must open them in binary mode on
Windows. That is, replace your lines:

f = file('pkl_' + fn, 'w')
f = file('pkl_' + fn, 'r')

with:

f = file('pkl_' + fn, 'wb')
f = file('pkl_' + fn, 'rb')

It's true that you can get away with opening pickle files in text mode
on Windows if you stick to the default pickle protocol (0), although
in that case your pickle files won't be portable across platforms.
For that reason, don't try to be clever: always use binary-mode files
to store pickles, regardless of pickle protocol in use.