Marshal Obj is String or Binary?

[Mike]

Hi,

The example below shows that result of a marshaled data structure is
nothing but a string

data = {2:'two', 3:'three'}
import marshal
bytes = marshal.dumps(d ata)
type(bytes) <type 'str'> bytes

'{i\x02\x00\x00 \x00t\x03\x00\x 00\x00twoi\x03\ x00\x00\x00t\x0 5\x00\x00\x00th ree0'

Now, I need to store this data safely in my database as CLEAR TEXT, not
BLOB. It seems to me that it should work just fine since it is string
anyways. So, why does O'reilly's Python Cookbook is insisting in saving
it as a binary file and BLOB type?

Am I missing out something?

Thanks,
Mike

[Marc 'BlackJack' Rintsch]

In <11************ **********@g47g 2000cwa.googleg roups.com>, Mike wrote:

The example below shows that result of a marshaled data structure is
nothing but a string

data = {2:'two', 3:'three'}
import marshal
bytes = marshal.dumps(d ata)
type(bytes) <type 'str'> bytes

'{i\x02\x00\x00 \x00t\x03\x00\x 00\x00twoi\x03\ x00\x00\x00t\x0 5\x00\x00\x00th ree0'

Now, I need to store this data safely in my database as CLEAR TEXT, not
BLOB. It seems to me that it should work just fine since it is string
anyways. So, why does O'reilly's Python Cookbook is insisting in saving
it as a binary file and BLOB type?

Am I missing out something?

Yes, that a string is *binary* data. But only a subset of strings is safe
to use as `TEXT` in databases. Do you see all those '\x??' escapes?
'\x00' is *one* byte! A byte with the value zero. Something your DB
doesn't allow in a `TEXT` type.

Ciao,
Marc 'BlackJack' Rintsch

[Mike]

Wait a sec. \x00 may represent a byte when unmarshaled, but as long as
marshal likes it as \x00, I think my db is capable of storing \ x 0 0
characters. What is the problem? Is it that \? I could escape that...
actually I think my django framework already does that for me.

Thanks,
Mike

[Mike]

Wait a sec. \x00 may represent a byte when unmarshaled, but as long as
marshal likes it as \x00, I think my db is capable of storing \ x 0 0
characters. What is the problem? Is it that \? I could escape that...
actually I think my django framework already does that for me.

Thanks,
Mike

[casevh]

Try...

for i in bytes: print ord(i)
or
len(bytes)

What you see isn't always what you have. Your database is capable of
storing \ x 0 0 characters, but your string contains a single byte of
value zero. When Python displays the string representation to you, it
escapes the values so they can be displayed.

casevh

[Giovanni Bajo]

ca****@comcast. net wrote:

Try...

for i in bytes: print ord(i)
or
len(bytes)
What you see isn't always what you have. Your database is capable of
storing \ x 0 0 characters, but your string contains a single byte of
value zero. When Python displays the string representation to you, it
escapes the values so they can be displayed.

He can still store the repr of the string into the database, and then
reconstruct it with eval:

bytes = "\x00\x01\x 02"
bytes '\x00\x01\x02' len(bytes) 3 ord(bytes[0]) 0 rb = repr(bytes)
rb "'\\x00\\x01\\x 02'" len(rb) 14 rb[0] "'" rb[1] '\\' rb[2] 'x' rb[3] '0' rb[4] '0' bytes2 = eval(rb)
bytes == bytes2

True

--
Giovanni Bajo

[Mike]

Thanks everyone. It seems broken storing complex structures as escaped
strings, but I think I'll take my changes.

Thanks,
Mike

[Steven D'Aprano]

On Fri, 13 Jan 2006 22:20:27 -0800, Mike wrote:

Thanks everyone. It seems broken storing complex structures as escaped
strings, but I think I'll take my changes.

Have you read the marshal reference?

http://docs.python.org/lib/module-marshal.html

marshal doesn't store data as escaped strings, it stores them as binary
strings. When you print the binary string to the console, unprintable
characters are shown escaped.

I'm guessing you probably want to use pickle instead of marshal. marshal
is intended only for dealing with .pyc files, and has some important
limitations. pickle is intended to be a general purpose serializer.
--
Steve.

[Max]

Giovanni Bajo wrote:

What you see isn't always what you have. Your database is capable of
storing \ x 0 0 characters, but your string contains a single byte of
value zero. When Python displays the string representation to you, it
escapes the values so they can be displayed.

He can still store the repr of the string into the database, and then
reconstruct it with eval:

Yes, but len(repr('\x00' )) is 4, while len('\x00') is 1. So if he uses
BLOB his data will take almost a quarter of the space, compared to your
method (stored as TEXT).

--Max

[Steven D'Aprano]

On Sat, 14 Jan 2006 12:36:59 +0200, Max wrote:

He can still store the repr of the string into the database, and then
reconstruct it with eval:

Yes, but len(repr('\x00' )) is 4, while len('\x00') is 1.

Incorrect:

len(repr('\x00' )) 6 repr('\x00') "'\\x00'"
So if he uses
BLOB his data will take almost a quarter of the space, compared to your
method (stored as TEXT).

Also incorrect. That depends utterly on which particular characters end up
in the serialised data. You may or may not be able to predict what that
mix may be.

# nothing but printable data
s = ''.join(['a' for i in range(256)])
len(s) 256 len(repr(s)) 258
# nothing but unprintable data
s = ''.join(['\0' for i in range(256)])
len(s) 256 len(repr(s)) 1026
# one particular mix of both printable and unprintable data
s = ''.join([chr(i) for i in range(256)])
len(s) 256 len(repr(s)) 737
# a different mix of both printable and unprintable data
s = '+'.join([chr(i) for i in range(128)])
len(s) 255 len(repr(s))

352

--
Steven.