473,288 Members | 1,771 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

persistent deque

I'd like a persistent deque, such that the instance operations all
commit atomicly to file system.
Jun 27 '08 #1
8 1488
"castironpi" <ca********@gmail.comwrote in message
news:29**********************************@c65g2000 hsa.googlegroups.com...
I'd like a persistent deque, such that the instance operations all
commit atomicly to file system.
ok, i made your persistent deque. but there are two important notes
regarding this module: 1) I am a genius. 2) it doesn't actually work for
some reason. i.e., loading an instance from file won't work. i don't know
why. the pickle module is doing something i didn't expect.
from collections import deque
import os, pickle, random

class pdeque(deque):
def __init__(self, filename, initial):
deque.__init__(self, initial)
self.filename = filename
self.tempfilename = ''.join((random.choice("abcdefghijklmnopqrstuvwxyz ")
for x in xrange(10)))+".tmp"
self.save()
def save(self):
pickle.dump(self, open(self.tempfilename,'wb'))
try: os.remove(self.filename)
except: pass
os.rename(self.tempfilename, self.filename)
@classmethod
def load(cls, filename):
return pickle.load(open(filename,'rb'))

for name in "append appendleft extend extendleft remove rotate".split():
func = getattr(deque, name)
def func2(instance, arg, func=func):
result = func(instance, arg)
instance.save()
return result
setattr(pdeque, name, func2)

for name in ("pop", "popleft", "clear"):
func = getattr(deque, name)
def func2(instance, func=func):
result = func(instance)
instance.save()
return result
setattr(pdeque, name, func2)
---

you use it just like a deque object except that when initializing the first
parameter is a filename for it to store in. to load the deque object again
later, call .load(filename) on the pdeque class or a pdeque object (except
that it doesn't work.)

i'm sure somebody here knows what obscure feature of pickling it is that's
doing this counterintuitively. (disclaimer: i've never used pickle to store
a custom type before)

Jun 27 '08 #2
def __init__(self, filename, initial):

should be

def __init__(self, filename, initial=[]):

(that was the whole reason i put the filename first.)

sorry.

Jun 27 '08 #3
inhahe wrote:
def __init__(self, filename, initial):

should be

def __init__(self, filename, initial=[]):

(that was the whole reason i put the filename first.)

sorry.
Defaulting initial to empty list this way is asking for trouble. You should
default it to None and check for None and reset to empty list.

-Larry
Jun 27 '08 #4
oh yeah! thanks for pointing it out, i should have thought of that, i
*just* read about it today, or was it yesterday.

so what's the problem with pickle? i have a feeling, maybe i read it, that
when you pickle a derived class it pickles it as a member of the base class,
is that it?

i don't know how i would get around the problem, though, because i'd have to
know how to access the deque object that my class stores when i do
deque.__init__ in my constructor, so that i could pickle it and my class
variables separately.


"Larry Bates" <la*********@websafe.com`wrote in message
news:0u******************************@comcast.com. ..
inhahe wrote:
> def __init__(self, filename, initial):

should be

def __init__(self, filename, initial=[]):

(that was the whole reason i put the filename first.)

sorry.
Defaulting initial to empty list this way is asking for trouble. You
should
default it to None and check for None and reset to empty list.

-Larry

Jun 27 '08 #5
>
i don't know how i would get around the problem, though, because i'd have
to know how to access the deque object that my class stores when i do
deque.__init__ in my constructor, so that i could pickle it and my class
variables separately.

i decided i could just pickle deque(self), which should return a regular
deque object with the data from self, and then in the load routine make a
pdeque object from it.

that, of couse, gives me another error. 'collections.deque' object has no
attribute 'write'. from the line 'self.write = file.write', which is in
pickle.py
pickling list(self) doesnt work either.

Jun 27 '08 #6
something randomly made me realize why my second solution didn't work, so i
fixed it. now you have a working persistent deque.

1. if you call somepdequeobject.load(filename) (as opposed to
pdeque.load(filename)), do you want it to return a new object that it loaded
from file, or do you want it to change somepdequeobject to be the object in
the file? i'm not sure which way is canonical.

2. i'm not sure if i actually have to use a temp file and then copy it. i
don't know if pickle.dump as an "atomic commit".

3. cPickle would be faster but i think that means you might end up storing
something in your deque that can't be pickled, like a circular reference,
and dunno what else.

4. can someone tell me if the way i'm using the decorator is sane? i've
never used decorators before. it just seems ridiculous to a) define a lambda
that just throws away the parameter, and b) define a meaningless function to
pass to that lambda.

5. Of course, I am a genius.

from collections import deque
import os, pickle, random

class pdeque(deque):
def __init__(self, filename, initial=None):
if initial is None: initial = []
deque.__init__(self, initial)
self.filename = filename
self.tempfilename = ''.join((random.choice("abcdefghijklmnopqrstuvwxyz ")
for x in xrange(10)))+".tmp"
self.save()
def save(self):
pickle.dump(deque(self), open(self.tempfilename,'wb'))
try: os.remove(self.filename)
except: pass
os.rename(self.tempfilename, self.filename)
@classmethod
def load(cls, filename):
return pdeque(filename, pickle.load(open(filename,'rb')))

#todo: change this so that if it's called from an object it loads
#the data into that object?

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return lambda _: func2

for name, func in deque.__dict__.items():
if (not name.startswith("_")) or name in ('__delitem__', '__setitem__'):
@makefunc(func)
def f(): pass
setattr(pdeque, name, f)



"castironpi" <ca********@gmail.comwrote in message
news:29**********************************@c65g2000 hsa.googlegroups.com...
I'd like a persistent deque, such that the instance operations all
commit atomicly to file system.

Jun 27 '08 #7

"inhahe" <in****@gmail.comwrote in message
news:XF*******************@bignews7.bellsouth.net. ..
>
4. can someone tell me if the way i'm using the decorator is sane? i've
never used decorators before. it just seems ridiculous to a) define a
lambda that just throws away the parameter, and b) define a meaningless
function to pass to that lambda.

I thought about the fact that a decorator is merely syntactic sugar, so it
shouldn't have any closure magic that I can't make myself, and I realized
that I could have done it the following way:

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return func2

for name, func in deque.__dict__.items():
if (not name.startswith("_")) or name in ('__delitem__', '__setitem__'):
setattr(pdeque, name, makefunc(func))

Jun 27 '08 #8
En Thu, 22 May 2008 12:20:56 -0300, inhahe <in****@gmail.comescribió:
I thought about the fact that a decorator is merely syntactic sugar, so it
shouldn't have any closure magic that I can't make myself, and I realized
that I could have done it the following way:

def makefunc(func):
def func2(instance, *args):
result = func(instance, *args)
instance.save()
return result
return func2
Using functools.wraps is better because it helps to preserve the function name and signature:

def makefunc(func):
@wraps(func)
def wrapper(self, *args, **kwds):
result = func(self, *args, **kwds)
self.save()
return result
return wrapper

--
Gabriel Genellina

Jun 27 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

16
by: newsock | last post by:
What differences between queue, deque and priority_queue? And under what situations choose one of them to use? Thanks for your help!
7
by: Jenny | last post by:
Hi, I have a class foo which will construct some objects in my code. some of the objects store int values into the data deque, while others store float values to the deque. template <class...
9
by: R.Z. | last post by:
i was wondering whether it pays off in terms of memory use to maintain lots of empty deques (it would be convenient for my algorithms but memory use is more important). and does the size of a deque...
7
by: Dan Trowbridge | last post by:
He everyone, I am just getting started with .NET and I am having a porting problem. I get and error in code that lookssomething like this (really stripped down but you get the idea)... class...
5
by: Russell Warren | last post by:
Does anyone have an easier/faster/better way of popping from the middle of a deque than this? class mydeque(deque): def popmiddle(self, pos): self.rotate(-pos) ret = self.popleft()...
5
by: yancheng.cheok | last post by:
after reading http://www.codeproject.com/vcpp/stl/vector_vs_deque.asp, i realize that deque has its own speed advantage over vector in all the aspect (except for memory deallocation). does it...
15
by: Juha Nieminen | last post by:
I'm sure this is not a new idea, but I have never heard about it before. I'm wondering if this could work: Assume that you have a common base class and a bunch of classes derived from it, and...
4
by: castironpi | last post by:
Some time ago, I was asking about the feasibility of a persistent deque, a double-ended queue. It runs into the typical space allocation problems. If you're storing a pickle, you have to...
2
by: subramanian100in | last post by:
In ISO/IEC 14882:2003 document, in the section '23.2.1.3 deque modifiers', the following is mentioned: iterator insert(iterator position, const T& x); void insert(iterator position, size_type...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.