473,327 Members | 1,896 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

pickle problem

I'm wondering if anyone can help with a workaround for a problem I
currently have. I'm trying to set up a prefork tcp server.
Specifically, I'm setting up a server that forks children and has them
listen on pipes created with os.pipe(). The parent process for the
group starts an inet:tcp server on a given port. In the parent, after
a "socket.accept()", I'm trying to pickle the connection object to
send over an IPC pipe (as stated previously), but I get the following
error:

File "/usr/lib/python2.4/copy_reg.py", line 76, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining
__getstate__ cannot be pickled

Does anyone know of a workaround for this? Maybe my approach to this
is wrong? Any help would be appreciated.

Jay
Jun 27 '08 #1
10 3459
On May 7, 11:02*pm, krustymon...@gmail.com wrote:
I'm wondering if anyone can help with a workaround for a problem I
currently have. *I'm trying to set up a prefork tcp server.
Specifically, I'm setting up a server that forks children and has them
listen on pipes created with os.pipe(). *The parent process for the
group starts an inet:tcp server on a given port. *In the parent, after
a "socket.accept()", I'm trying to pickle the connection object to
send over an IPC pipe (as stated previously), but I get the following
error:

File "/usr/lib/python2.4/copy_reg.py", line 76, in _reduce_ex
* * raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining
__getstate__ cannot be pickled

Does anyone know of a workaround for this? *Maybe my approach to this
is wrong? *Any help would be appreciated.

Jay
The group currently holds that pickling socket objects is non-optimal
to the ends of making money. We can tell you good ways to accomplish
what you set out to do all the same. I find it educational; that's
fine in my book. Part of the resistence comes from a resistence to
monarchy. The bottleneck is the starting of a second Python process.

There are a number of objects which can't be pickled, though. If you
can objectively prioritize the next step in pickling sockets, you may
find out sooner when it will come in to Python. But that's the hard
part; speaker recommends against.

We accept your idea is to get a process around the socket barrier.
Can you close the server in Part N, hear the connection in Part N+1,
spawn a new server in N+2, and handle the first one in N+3? Is it
suitable to your minds?
Jun 27 '08 #2
kr**********@gmail.com wrote:
I'm wondering if anyone can help with a workaround for a problem I
currently have. I'm trying to set up a prefork tcp server.
Specifically, I'm setting up a server that forks children and has them
listen on pipes created with os.pipe(). The parent process for the
group starts an inet:tcp server on a given port. In the parent, after
a "socket.accept()", I'm trying to pickle the connection object to
send over an IPC pipe (as stated previously), but I get the following
error:

File "/usr/lib/python2.4/copy_reg.py", line 76, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining
__getstate__ cannot be pickled

Does anyone know of a workaround for this? Maybe my approach to this
is wrong? Any help would be appreciated.
The error-message is pretty clear I'd say. You use slots - so you are
responsible yourself for implementing the pickling-protocol using
__getstate__ and __setstate__. Looking at the pickle-docs should give you
an idea.

But the really easy solution is: do not use slots. They are intended as
memory-consumption optimization technique, *not* as "I want to declare my
attributes explicitly"-mechanism. So - get rid of them and be a happy
pickler.

Diez
Jun 27 '08 #3
On May 8, 5:48 am, "Diez B. Roggisch" <de...@nospam.web.dewrote:
krustymon...@gmail.com wrote:
I'm wondering if anyone can help with a workaround for a problem I
currently have. I'm trying to set up a prefork tcp server.
Specifically, I'm setting up a server that forks children and has them
listen on pipes created with os.pipe(). The parent process for the
group starts an inet:tcp server on a given port. In the parent, after
a "socket.accept()", I'm trying to pickle the connection object to
send over an IPC pipe (as stated previously), but I get the following
error:
File "/usr/lib/python2.4/copy_reg.py", line 76, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining
__getstate__ cannot be pickled
Does anyone know of a workaround for this? Maybe my approach to this
is wrong? Any help would be appreciated.

The error-message is pretty clear I'd say. You use slots - so you are
responsible yourself for implementing the pickling-protocol using
__getstate__ and __setstate__. Looking at the pickle-docs should give you
an idea.

But the really easy solution is: do not use slots. They are intended as
memory-consumption optimization technique, *not* as "I want to declare my
attributes explicitly"-mechanism. So - get rid of them and be a happy
pickler.

Diez
The thing is, I'm not using slots by choice. I'm using the standard
lib "socket" class, which apparently uses slots.
Jun 27 '08 #4
On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
The thing is, I'm not using slots by choice. I'm using the standard
lib "socket" class, which apparently uses slots.
`socket` objects can't be pickled. Not just because of the `__slot__`\s
but because a substantial part of their state lives in the operating
system's space.

Ciao,
Marc 'BlackJack' Rintsch
Jun 27 '08 #5
Marc 'BlackJack' Rintsch <bj****@gmx.netwrites:
On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
>The thing is, I'm not using slots by choice. I'm using the standard
lib "socket" class, which apparently uses slots.

`socket` objects can't be pickled. Not just because of the
`__slot__`\s but because a substantial part of their state lives in
the operating system's space.
Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:

class Connection(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.init_sock()

def init_sock(self):
self.sock = socket.socket()
self.sock.connect((host, port))
... init communication ...

def __getstate__(self):
# pickle self as a (host, port) pair
return self.host, self.port

def __setstate__(self, state):
# reinstate self by setting host and port and
# recreating the socket
self.host, self.port = state
self.init_sock()
Jun 27 '08 #6
On May 8, 4:35*pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Marc 'BlackJack' Rintsch <bj_...@gmx.netwrites:
On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
The thing is, I'm not using slots by choice. *I'm using the standard
lib "socket" class, which apparently uses slots.
`socket` objects can't be pickled. *Not just because of the
`__slot__`\s but because a substantial part of their state lives in
the operating system's space.

Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:

class Connection(object):
* * def __init__(self, host, port):
* * * * self.host = host
* * * * self.port = port
* * * * self.init_sock()

* * def init_sock(self):
* * * * self.sock = socket.socket()
* * * * self.sock.connect((host, port))
* * * * ... init communication ...

* * def __getstate__(self):
* * * * # pickle self as a (host, port) pair
* * * * return self.host, self.port

* * def __setstate__(self, state):
* * * * # reinstate self by setting host and port and
* * * * # recreating the socket
* * * * self.host, self.port = state
* * * * self.init_sock()
I, local, am mystified that you'd want to pickle a socket. It's a
live connection, which flies on a pocket dollar. You don't want it on
disk, do you?

If you're running a net buoy through a cluster somewhere, do you want
to drop a server and reconnect? Is Amazon's EC2 up and running?

Certainly no one was talking on the internet. Were you?

I don't think you hit anything but banks surfing the web, and the
American dollar is notoriously stuffy. Pump a wi-fi to a diner,
though, and you're just talking more. Where's Usenet?
Jun 27 '08 #7
On Thu, 08 May 2008 23:35:04 +0200, Hrvoje Niksic wrote:
Marc 'BlackJack' Rintsch <bj****@gmx.netwrites:
>On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
>>The thing is, I'm not using slots by choice. I'm using the standard
lib "socket" class, which apparently uses slots.

`socket` objects can't be pickled. Not just because of the
`__slot__`\s but because a substantial part of their state lives in
the operating system's space.

Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:
When does it make sense!?
class Connection(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.init_sock()

def init_sock(self):
self.sock = socket.socket()
self.sock.connect((host, port))
... init communication ...
But if you unpickle it while the original connection is still open it
can't connect. If it's not open anymore, there's no one answering at
host/port anymore or some program that has no idea what this connection
is all about.

Ciao,
Marc 'BlackJack' Rintsch
Jun 27 '08 #8
Marc 'BlackJack' Rintsch <bj****@gmx.netwrites:
>Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:

When does it make sense!?
When recreating the object from on-disk state requires reestablishing
the communication, as the example shows. I'm not saying that doing
that is always a good idea, only that it can be done if/when needed.
But if you unpickle it while the original connection is still open it
can't connect.
Why not? I was thinking "client socket", not "server socket".
Jun 27 '08 #9
On May 8, 7:29 pm, castiro...@gmail.com wrote:
On May 8, 4:35 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Marc 'BlackJack' Rintsch <bj_...@gmx.netwrites:
On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
>The thing is, I'm not using slots by choice. I'm using the standard
>lib "socket" class, which apparently uses slots.
`socket` objects can't be pickled. Not just because of the
`__slot__`\s but because a substantial part of their state lives in
the operating system's space.
Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:
class Connection(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.init_sock()
def init_sock(self):
self.sock = socket.socket()
self.sock.connect((host, port))
... init communication ...
def __getstate__(self):
# pickle self as a (host, port) pair
return self.host, self.port
def __setstate__(self, state):
# reinstate self by setting host and port and
# recreating the socket
self.host, self.port = state
self.init_sock()

I, local, am mystified that you'd want to pickle a socket. It's a
live connection, which flies on a pocket dollar. You don't want it on
disk, do you?

If you're running a net buoy through a cluster somewhere, do you want
to drop a server and reconnect? Is Amazon's EC2 up and running?

Certainly no one was talking on the internet. Were you?

I don't think you hit anything but banks surfing the web, and the
American dollar is notoriously stuffy. Pump a wi-fi to a diner,
though, and you're just talking more. Where's Usenet?
The idea is a pre-fork socket server. What I want to do is fork off
some child processes from the parent and use IPC to send the
connection object (via socket.accept()) over a pipe to one of the
preexisting child processes and have said child handle the
transaction. Think Apache, if you want an example of what I'm trying
to do here. This alleviates the process startup cost that occurs when
you fork. If the socket object can't be serialized, is there another
way to go about handling this in python?
Jun 27 '08 #10
On May 11, 6:21*pm, krustymon...@gmail.com wrote:
On May 8, 7:29 pm, castiro...@gmail.com wrote:


On May 8, 4:35 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Marc 'BlackJack' Rintsch <bj_...@gmx.netwrites:
On Thu, 08 May 2008 08:55:35 -0700, krustymonkey wrote:
The thing is, I'm not using slots by choice. *I'm using the standard
lib "socket" class, which apparently uses slots.
`socket` objects can't be pickled. *Not just because of the
`__slot__`\s but because a substantial part of their state lives in
the operating system's space.
Of course, if it makes sense to pickle sockets in the application, one
is can do so by defining __getstate__ and __setstate__:
class Connection(object):
* * def __init__(self, host, port):
* * * * self.host = host
* * * * self.port = port
* * * * self.init_sock()
* * def init_sock(self):
* * * * self.sock = socket.socket()
* * * * self.sock.connect((host, port))
* * * * ... init communication ...
* * def __getstate__(self):
* * * * # pickle self as a (host, port) pair
* * * * return self.host, self.port
* * def __setstate__(self, state):
* * * * # reinstate self by setting host and port and
* * * * # recreating the socket
* * * * self.host, self.port = state
* * * * self.init_sock()
I, local, am mystified that you'd want to pickle a socket. *It's a
live connection, which flies on a pocket dollar. *You don't want it on
disk, do you?
If you're running a net buoy through a cluster somewhere, do you want
to drop a server and reconnect? *Is Amazon's EC2 up and running?
Certainly no one was talking on the internet. *Were you?
I don't think you hit anything but banks surfing the web, and the
American dollar is notoriously stuffy. *Pump a wi-fi to a diner,
though, and you're just talking more. *Where's Usenet?

The idea is a pre-fork socket server. *What I want to do is fork off
some child processes from the parent and use IPC to send the
connection object (via socket.accept()) over a pipe to one of the
preexisting child processes and have said child handle the
transaction. *Think Apache, if you want an example of what I'm trying
to do here. *This alleviates the process startup cost that occurs when
you fork. *If the socket object can't be serialized, is there another
way to go about handling this in python?- Hide quoted text -

- Show quoted text -
That's out of my expertise. You would have to either hack it on a
router you are running (masquerade), or choose a different family of
sockets. For HTTP, you could issue a 'reroute' return. Just multi-
thread your server process.
Jun 27 '08 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Simon Burton | last post by:
Hi, I am pickling big graphs of data and running into this problem: File "/usr/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/lib/python2.2/pickle.py", line 414, in...
3
by: Michael Hohn | last post by:
Hi, under python 2.2, the pickle/unpickle sequence incorrectly restores a larger data structure I have. Under Python 2.3, these structures now give an explicit exception from...
0
by: Mike P. | last post by:
Hi all, I'm working on a simulation (can be considered a game) in Python where I want to be able to dump the simulation state to a file and be able to load it up later. I have used the standard...
10
by: crystalattice | last post by:
I'm creating an RPG for experience and practice. I've finished a character creation module and I'm trying to figure out how to get the file I/O to work. I've read through the python newsgroup...
5
by: Chris | last post by:
Why can pickle serialize references to functions, but not methods? Pickling a function serializes the function name, but pickling a staticmethod, classmethod, or instancemethod generates an...
2
by: Victor Lin | last post by:
Hi, I encounter a problem with pickle. I download a html from: ...
3
by: fizilla | last post by:
Hello all! I have the following weird problem and since I am new to Python I somehow cannot figure out an elegant solution. The problem reduces to the following question: How to pickle a...
2
by: Nagu | last post by:
I am trying to save a dictionary of size 65000X50 to a local file and I get the memory error problem. How do I go about resolving this? Is there way to partition the pickle object and combine...
0
by: Nagu | last post by:
I am trying to save a dictionary of size 65000X50 to a local file and I get the memory error problem. How do I go about resolving this? Is there way to partition the pickle object and combine...
1
by: IceMan85 | last post by:
Hi to all, I have spent the whole morning trying, with no success to pickle an object that I have created. The error that I get is : Can't pickle 'SRE_Match' object: <_sre.SRE_Match object at...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.