storing pickles in sql data base

[David Bear]

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

--
David Bear
-- let me buy your intellectual property, I want to own your thoughts --

[Gabriel Genellina]

En Tue, 10 Jul 2007 20:32:01 -0300, David Bear <da********@asu.edu>
escribió:

I need to store pickled objects in postgresql. I reading through the
pickle
docs it says to always open a file in binary mode because you can't be
sure
if the pickled data is binary or text. So I have 2 question. Can I set
the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

I'd use a binary datatype (raw, blob, binary, whatever postgres calls it).
Text columns might be converted or reencoded in some way, binary data
should never be modified in any way.

--
Gabriel Genellina

[Nick Craig-Wood]

David Bear <da********@asu.eduwrote:

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg

>>from cPickle import dumps, loads
a = range(10)
a

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>b = dumps(a).encode("zip").encode("base64").strip()
b

'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'

>>loads(b.decode("base64").decode("zip"))

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>>

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[Chris Mellon]

On 7/11/07, Nick Craig-Wood <ni**@craig-wood.comwrote:

David Bear <da********@asu.eduwrote:

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg

>>from cPickle import dumps, loads
>>a = range(10)
>>a

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>b = dumps(a).encode("zip").encode("base64").strip()
>>b

'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'

>>loads(b.decode("base64").decode("zip"))

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>>

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file. base64 encoding will work on
protocol 1 and 2 (which are binary protocols and not text-safe), but
doing that removes the main benefit of the higher protocols, which is
smaller pickle data.

[Marc 'BlackJack' Rintsch]

On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

Ciao,
Marc 'BlackJack' Rintsch

[Chris Mellon]

On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.netwrote:

On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.

[Gabriel Genellina]

En Thu, 12 Jul 2007 14:18:11 -0300, Chris Mellon <ar*****@gmail.com>
escribió:

On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.net>
wrote:

>On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:

[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses
just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.

Neither. Won't work for Unicode objects then. See bug#1724366
<http://python.org/sf/1724366>

--
Gabriel Genellina