473,242 Members | 1,319 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,242 software developers and data experts.

storing pickles in sql data base

I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

--
David Bear
-- let me buy your intellectual property, I want to own your thoughts --
Jul 10 '07 #1
6 8075
En Tue, 10 Jul 2007 20:32:01 -0300, David Bear <da********@asu.edu>
escribió:
I need to store pickled objects in postgresql. I reading through the
pickle
docs it says to always open a file in binary mode because you can't be
sure
if the pickled data is binary or text. So I have 2 question. Can I set
the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?
I'd use a binary datatype (raw, blob, binary, whatever postgres calls it).
Text columns might be converted or reencoded in some way, binary data
should never be modified in any way.

--
Gabriel Genellina

Jul 11 '07 #2
David Bear <da********@asu.eduwrote:
I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?
You could always encode it into text form, eg
>>from cPickle import dumps, loads
a = range(10)
a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>b = dumps(a).encode("zip").encode("base64").strip()
b
'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'
>>loads(b.decode("base64").decode("zip"))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jul 11 '07 #3
On 7/11/07, Nick Craig-Wood <ni**@craig-wood.comwrote:
David Bear <da********@asu.eduwrote:
I need to store pickled objects in postgresql. I reading through the pickle
docs it says to always open a file in binary mode because you can't be sure
if the pickled data is binary or text. So I have 2 question. Can I set the
pickle to be text -- and then store it in a 'text' type field in my sql
table, or should what sql type should I save the pickle as?

You could always encode it into text form, eg
>>from cPickle import dumps, loads
>>a = range(10)
>>a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>b = dumps(a).encode("zip").encode("base64").strip()
>>b
'eJzTyCkw5PI04Er0NARiIyA2BmITIDYFYjMgNgdiCyC25ErUA wD5DQqD'
>>loads(b.decode("base64").decode("zip"))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>
Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file. base64 encoding will work on
protocol 1 and 2 (which are binary protocols and not text-safe), but
doing that removes the main benefit of the higher protocols, which is
smaller pickle data.
Jul 11 '07 #4
On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.
It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

Ciao,
Marc 'BlackJack' Rintsch
Jul 12 '07 #5
On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.netwrote:
On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.
It's fine as long as you use universal line endings mode.
Jul 12 '07 #6
En Thu, 12 Jul 2007 14:18:11 -0300, Chris Mellon <ar*****@gmail.com>
escribió:
On 12 Jul 2007 06:00:59 GMT, Marc 'BlackJack' Rintsch <bj****@gmx.net>
wrote:
>On Wed, 11 Jul 2007 17:14:43 -0500, Chris Mellon wrote:
[pickle]

Protocol 0 (the default) is a text protocol, it's safe to store in a
text field or write to a text file.

It's not really a text protocol it's more a binary protocol that uses
just
the ASCII range of byte values. You have to write and read the "text"
files in binary mode or they break if taken across platform boundaries
because of the different line endings in Linux and Windows for instance.

It's fine as long as you use universal line endings mode.
Neither. Won't work for Unicode objects then. See bug#1724366
<http://python.org/sf/1724366>

--
Gabriel Genellina

Jul 13 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Erik Max Francis | last post by:
Is there any prohibition against mixing different protocols within the same pickle? I don't see anything about this in the Python Library Reference and, after all, the pickle.dump function takes a...
17
by: tuvok | last post by:
How can objects of different types be stored in a collection? For example: struct S1 { /*...*/ }; struct S2 { /*...*/ }; struct Sn { /*...*/ }; template<typename T> class X { public:
3
by: Ann Huxtable | last post by:
Hi, I am writing a nested table structure to mimic data returned from a database (actually, a heirarchical recordset). I am representing the cells where the actual data is stored, by a union:...
2
by: Jacek Dziedzic | last post by:
Hello! Suppose I have a class that contains only public members of builtin types and a default constructor. Because of the default constructor it is no longer an aggregate and therefore no...
3
by: jkugler | last post by:
Hello, I am trying to store huge amounts of data from xml files and put them into a MySQL database. The xml files all are in this format: <?xml version="1.0" encoding="UTF-8"?>...
10
by: tshad | last post by:
I am trying to find the best procedure for storing keys used for encryption. This would also be a question for the connection string to the database. At the moment, this is kept in the web.info...
1
by: icemani | last post by:
Is there any advantages that storing data in hexadecimal octal formate in data base? By Manikandan.
3
by: ashokkuncha | last post by:
hi all, i am new bee and hope some one help me out. in my web application i am using access as data base, one table of the dabase has candidates details as records ( emp no, name , skill set...
13
by: Josip | last post by:
I'm trying to limit a value stored by object (either int or float): class Limited(object): def __init__(self, value, min, max): self.min, self.max = min, max self.n = value def...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.