471,339 Members | 1,250 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,339 software developers and data experts.

storing pickles in sql data base

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

--
David Bear
-- let me buy your intellectual property, I want to own your thoughts --
Jul 10 '07 #1
6 7407
En Tue, 10 Jul 2007 20:32:01 -0300, David Bear <da********@asu.edu>
escribió:
I need to store pickled objects in postgresql. I reading through the
pickle
docs it says to always open a file in binary mode because you can't be
sure
if the pickled data is binary or text. So I have 2 question. Can I set
the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?
I'd use a binary datatype (raw, blob, binary, whatever postgres calls it).
Text columns might be converted or reencoded in some way, binary data
should never be modified in any way.

--
Gabriel Genellina

Jul 11 '07 #2
David Bear <da********@asu.eduwrote:
I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?
You could always encode it into text form, eg
>>from cPickle import dumps, loads
a = range(10)
a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>b = dumps(a).encode("zip").encode("base64").strip()
b
'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'
>>loads(b.decode("base64").decode("zip"))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jul 11 '07 #3
On 7/11/07, Nick Craig-Wood <ni**@craig-wood.comwrote:
David Bear <da********@asu.eduwrote:
I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg
>>from cPickle import dumps, loads
>>a = range(10)
>>a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>b = dumps(a).encode("zip").encode("base64").strip()
>>b
'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'
>>loads(b.decode("base64").decode("zip"))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file. base64 encoding will work on
protocol 1 and 2 (which are binary protocols and not text-safe), but
doing that removes the main benefit of the higher protocols, which is
smaller pickle data.
Jul 11 '07 #4
On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.
It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

Ciao,
Marc 'BlackJack' Rintsch
Jul 12 '07 #5
On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.netwrote:
On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.
It's fine as long as you use universal line endings mode.
Jul 12 '07 #6
En Thu, 12 Jul 2007 14:18:11 -0300, Chris Mellon <ar*****@gmail.com>
escribió:
On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.net>
wrote:
>On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses
just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.
Neither. Won't work for Unicode objects then. See bug#1724366
<http://python.org/sf/1724366>

--
Gabriel Genellina

Jul 13 '07 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Erik Max Francis | last post: by
17 posts views Thread by tuvok | last post: by
3 posts views Thread by Ann Huxtable | last post: by
2 posts views Thread by Jacek Dziedzic | last post: by
10 posts views Thread by tshad | last post: by
13 posts views Thread by Josip | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.