pickle alternative

[simonwittber]

I've written a simple module which serializes these python types:

IntType, TupleType, StringType, FloatType, LongType, ListType, DictType

It available for perusal here:

http://aspn.activestate.com/ASPN/Coo.../Recipe/415503

It appears to work faster than pickle, however, the decode process is
much slower (5x) than the encode process. Has anyone got any tips on
ways I might speed this up?
Sw.

[Andrew Dalke]

simonwittber wrote:

I've written a simple module which serializes these python types:

IntType, TupleType, StringType, FloatType, LongType, ListType, DictType
For simple data types consider "marshal" as an alternative to "pickle".
It appears to work faster than pickle, however, the decode process is
much slower (5x) than the encode process. Has anyone got any tips on
ways I might speed this up?

def dec_int_type(da ta):
value = int(unpack('!i' , data.read(4))[0])
return value

That 'int' isn't needed -- unpack returns an int not a string
representation of the int.

BTW, your code won't work on 64 bit machines.

def enc_long_type(o bj):
return "%s%s%s" % ("B", pack("!L", len(str(obj))), str(obj))

There's no need to compute str(long) twice -- for large longs
it takes a lot of work to convert to base 10. For that matter,
it's faster to convert to hex, and the hex form is more compact.

Every decode you do requires several function calls. While
less elegant, you'll likely get better performance (test it!)
if you minimize that; try something like this

def decode(data):
return _decode(StringI O(data).read)

def _decode(read, unpack = struct.unpack):
code = read(1)
if not code:
raise IOError("reache d the end of the file")
if code == "I":
return unpack("!i", read(4))[0]
if code == "F":
return unpack("!f", read(4))[0]
if code == "L":
count = unpack("!i", read(4))
return [_decode(read) for i in range(count)]
if code == "D":
count = unpack("!i", read(4))
return dict([_decode(read) for i in range(count)]
...

Andrew
da***@dalkescie ntific.com

[simonwittber]

> For simple data types consider "marshal" as an alternative to "pickle".

From the marhal documentation: Warning: The marshal module is not intended to be secure against
erroneous or maliciously constructed data. Never unmarshal data
received from an untrusted or unauthenticated source.
BTW, your code won't work on 64 bit machines.
Any idea how this might be solved? The number of bytes used has to be
consistent across platforms. I guess this means I cannot use the struct
module?
There's no need to compute str(long) twice -- for large longs
it takes a lot of work to convert to base 10. For that matter,
it's faster to convert to hex, and the hex form is more compact.

Thanks for the tip.

Sw.

[Andrew Dalke]

simonwittber wrote:

From the marhal documentation: Warning: The marshal module is not intended to be secure against
erroneous or maliciously constructed data. Never unmarshal data
received from an untrusted or unauthenticated source.

Ahh, I had forgotten that. Though I can't recall what an attack
might be, I think it's because the C code hasn't been fully vetted
for unexpected error conditions.
Any idea how this might be solved? The number of bytes used has to be
consistent across platforms. I guess this means I cannot use the struct
module?

How do you want to solve it? Should a 64 bit machine be able to read
a data stream made on a 32 bit machine? What about vice versa? How
are floats interconverted?

You could preface the output stream with a description of the encoding
used: version number, size of float, size of int (which should always
be sizeof float these days, I think). Read these then use that
information to figure out which decode/dispatch function to use.

Andrew
da***@dalkescie ntific.com

[simonwittber]

Ahh, I had forgotten that. Though I can't recall what an attack
might be, I think it's because the C code hasn't been fully vetted
for unexpected error conditions.

I tried out the marshal module anyway.

marshal can serialize small structures very qucikly, however, using the
below test value:

value = [r for r in xrange(1000000)] +
[{1:2,3:4,5:6},{ "simon":"wittbe r"}]

marshal took 7.90 seconds to serialize it into a 5000061 length string.
decode took 0.08 seconds.

The aforementioned recipe took 2.53 seconds to serialize it into a
5000087 length string. decode took 5.16 seconds, which is much longer
than marshal!!

Sw.

[Andrew Dalke]

simonwittber wrote:

marshal can serialize small structures very qucikly, however, using the
below test value:

value = [r for r in xrange(1000000)] +
[{1:2,3:4,5:6},{ "simon":"wittbe r"}]

marshal took 7.90 seconds to serialize it into a 5000061 length string.
decode took 0.08 seconds.

Strange. Here's what I found:

value = [r for r in xrange(1000000)] +[{1:2,3:4,5:6},{ "simon":"wittbe r"}]
import time, marshal
t1=time.time(); s=marshal.dumps (value);t2=time .time()
t2-t1 0.2247400283813 4766 len(s) 5000061 t1=time.time(); new_value=marsh al.loads(s);t2= time.time()
t2-t1 0.3606879711151 123 new_value == value True

I can't reproduce your large times for marshal.dumps. Could you
post your test code?

Andrew
da***@dalkescie ntific.com

[simonwittber]

> I can't reproduce your large times for marshal.dumps. Could you

post your test code?

Certainly:

import sencode
import marshal
import time

value = [r for r in xrange(1000000)] +
[{1:2,3:4,5:6},{ "simon":"wittbe r"}]

t = time.clock()
x = marshal.dumps(v alue)
print "marshal enc T:", time.clock() - t

t = time.clock()
x = marshal.loads(x )
print "marshal dec T:", time.clock() - t

t = time.clock()
x = sencode.dumps(v alue)
print "sencode enc T:", time.clock() - t
t = time.clock()
x = sencode.loads(x )
print "sencode dec T:", time.clock() - t

[Andrew Dalke]

simonwittber posted his test code.

I tooks the code from the cookbook, called it "sencode" and
added these two lines

dumps = encode
loads = decode
I then ran your test code (unchanged except that my newsreader
folded the "value = ..." line) and got

marshal enc T: 0.21
marshal dec T: 0.4
sencode enc T: 7.76
sencode dec T: 11.56

This is with Python 2.3; the stock one provided by Apple
for my Mac.

I expected the numbers to be like this because the marshal
code is used to make and read the .pyc files and is supposed
to be pretty fast.

BTW, I tried the performance approach I outlined earlier.
The numbers aren't much better

marshal enc T: 0.2
marshal dec T: 0.38
sencode2 enc T: 7.16
sencode2 dec T: 9.49
I changed the format a little bit; dicts are treated a bit
differently.
from struct import pack, unpack
from cStringIO import StringIO

class EncodeError(Exc eption):
pass
class DecodeError(Exc eption):
pass

def encode(data):
f = StringIO()
_encode(data, f.write)
return f.getvalue()

def _encode(data, write, pack = pack):
# The original code use the equivalent of "type(data) is list"
# I preserve that behavior

T = type(data)

if T is int:
write("I")
write(pack("!i" , data))
elif T is list:
write("L")
write(pack("!L" , len(data)))
# Assumes len and 'for ... in' aren't lying
for item in data:
_encode(item, write)
elif T is tuple:
write("T")
write(pack("!L" , len(data)))
# Assumes len and 'for ... in' aren't lying
for item in data:
_encode(item, write)
elif T is str:
write("S")
write(pack("!L" , len(data)))
write(data)
elif T is long:
s = hex(data)[2:-1]
write("B")
write(pack("!i" , len(s)))
write(s)
elif T is type(None):
write("N")
elif T is float:
write("F")
write(pack("!f" , data))
elif T is dict:
write("D")
write(pack("!L" , len(data)))
for k, v in data.items():
_encode(k, write)
_encode(v, write)
else:
raise EncodeError((da ta, T))
def decode(s):
"""
Decode a binary string into the original Python types.
"""
buffer = StringIO(s)
return _decode(buffer. read)

def _decode(read, unpack = unpack):
code = read(1)
if code == "I":
return unpack("!i", read(4))[0]
if code == "D":
size = unpack("!L", read(4))[0]
x = [_decode(read) for i in range(size*2)]
return dict(zip(x[0::2], x[1::2]))
if code == "T":
size = unpack("!L", read(4))[0]
return tuple([_decode(read) for i in range(size)])
if code == "L":
size = unpack("!L", read(4))[0]
return [_decode(read) for i in range(size)]
if code == "N":
return None
if code == "S":
size = unpack("!L", read(4))[0]
return read(size)
if code == "F":
return unpack("!f", read(4))[0]
if code == "B":
size = unpack("!L", read(4))[0]
return long(read(size) , 16)
raise DecodeError(cod e)

dumps = encode
loads = decode
I wonder if this could be improved by a "struct2" module
which could compile a pack/unpack format once. Eg,

float_struct = struct2.struct( "!f")

float_struct.pa ck(f)
return float_struct.un pack('?\x80\x00 \x00')[0]
which might the same as
return float_struct.un pack1('?\x80\x0 0\x00')

Andrew
da***@dalkescie ntific.com

[simonwittber]

Andrew Dalke wrote:

This is with Python 2.3; the stock one provided by Apple
for my Mac.
Ahh that is the difference. I'm running Python 2.4. I've checked my
benchmarks on a friends machine, also in Python 2.4, and received the
same results as my machine.
I expected the numbers to be like this because the marshal
code is used to make and read the .pyc files and is supposed
to be pretty fast.

It would appear that the new version 1 format introduced in Python 2.4
is much slower than version 0, when using the dumps function.

Thanks for your feedback Andrew!

Sw.

[Andrew Dalke]

simonwittber wrote:

It would appear that the new version 1 format introduced in Python 2.4
is much slower than version 0, when using the dumps function.

Interesting. Hadn't noticed that change. Is dump(StringIO() ) as
slow?

Andrew
da***@dalkescie ntific.com