473,881 Members | 1,691 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

"RuntimeErr or: dictionary changed size during iteration" ; Good atomiccopy operations?

In very rare cases a program crashes (hard to reproduce) :

* several threads work on an object tree with dict's etc. in it. Items
are added, deleted, iteration over .keys() ... ). The threads are "good"
in such terms, that this core data structure is changed only by atomic
operations, so that the data structure is always consistent regarding
the application. Only the change-operations on the dicts and lists
itself seem to cause problems on a Python level ..

* one thread periodically pickle-dumps the tree to a file:
cPickle.dump(ob j, f)


"RuntimeErr or: dictionary changed size during iteration" is raised by
..dump ( or a similar "..list changed ..." )

What can I do about this to get a stable pickle-dump without risiking
execution error or even worse - errors in the pickled file ?

Is a copy.deepcopy ( -> "cPickle.dump(c opy.deepcopy(ob j),f)" ) an
atomic opertion with a guarantee to not fail?

Or can I only retry several times in case of RuntimeError? (which would
apears to me as odd gambling; retry how often?)

Robert
PS: Zope dumps thread exposed data structes regularly. How does the ZODB
in Zope handle dict/list changes during its pickling operations?
---
Python 2.4.1 (#2, May 5 2005, 11:32:06)
[GCC 3.3.5 (Debian 1:3.3.5-12)] on linux2
Mar 11 '06 #1
28 7438
Is a copy.deepcopy ( -> "cPickle.dump(c opy.deepcopy(ob j),f)" ) an
atomic opertion with a guarantee to not fail?

Or can I only retry several times in case of RuntimeError? (which would
apears to me as odd gambling; retry how often?)
For an intermediate solution, I'm playing roulette:

for i in 1,2,3:
try:
cPickle.dump(ob j, f)
break
except RuntimeError,v:
pass
I hope this works for some million years ...
PS: Zope dumps thread exposed data structes regularly. How does the ZODB
in Zope handle dict/list changes during its pickling operations?

Mar 11 '06 #2
robert wrote:
Is a copy.deepcopy ( -> "cPickle.dump(c opy.deepcopy(ob j),f)" ) an
atomic opertion with a guarantee to not fail?

Or can I only retry several times in case of RuntimeError? (which
would apears to me as odd gambling; retry how often?)

For an intermediate solution, I'm playing roulette:

for i in 1,2,3:
try:
cPickle.dump(ob j, f)
break
except RuntimeError,v:
pass


hmm..

for i in 1,2,3:
try:
cPickle.dump(ob j, f)
break
except RuntimeError,v:
f.seek(0);f.tru ncate(0)
Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
instead of free iteration internally, when pickling elementary dicts.
I'd file a bug if no objection.

Robert

I hope this works for some million years ...
PS: Zope dumps thread exposed data structes regularly. How does the
ZODB in Zope handle dict/list changes during its pickling operations?

Mar 11 '06 #3
Em S√°b, 2006-03-11 √*s 12:49 +0100, robert escreveu:
Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
instead of free iteration internally, when pickling elementary dicts.
I'd file a bug if no objection.


AFAICS, it's a problem with your code. You should lock your object while
using it. That's what Threading.Lock is supposed to work for. If you
want to use threads, you have to know in what parts of your code there
should be locks.

Cya,
Felipe.

--
"Quem excele em empregar a força militar subjulga os exércitos dos
outros povos sem travar batalha, toma cidades fortificadas dos outros
povos sem as atacar e destrói os estados dos outros povos sem lutas
prolongadas. Deve lutar sob o Céu com o propósito primordial da
'preservação' . Desse modo suas armas não se embotarão, e os ganhos
poderão ser preservados. Essa é a estratégia para planejar ofensivas."

-- Sun Tzu, em "A arte da guerra"

Mar 11 '06 #4
robert si Ť profuso/a a scrivere su comp.lang.pytho n tutte queste
elucubrazioni:

[cut]

I don't know what's your code like, but a similar error occurred in some of
my software and it was my fault indeed. I think you should either use a
lock, or implement a deepcopy method of your own.

--
EleSSaR^ <us************ **********@yaho o.it>
--
Togli .xyz dalla mia email per contattarmi.
Mar 11 '06 #5
Felipe Almeida Lessa wrote:
Em S√°b, 2006-03-11 √*s 12:49 +0100, robert escreveu:
Meanwhile I think this is a bug of cPickle.dump: It should use .keys()
instead of free iteration internally, when pickling elementary dicts.
I'd file a bug if no objection.

AFAICS, it's a problem with your code. You should lock your object while
using it. That's what Threading.Lock is supposed to work for. If you
want to use threads, you have to know in what parts of your code there
should be locks.


99.99% no. I would have to use a lock everywhere, where I add or remove
something into a dict or list of the struct. Thats not the purpose of
big thread locks. Such simple operations are already atomic by the
definition of Python - and thanks to the global interpreter lock.
(Otherwise I would leave the Python language, God beware ... :-) )

I'm of course aware, where to use locks for resons of the application.
But this is an issue on Python level. And it can be solved gracly and
simple in Python - I guess:

If cPickle.dump (and maybe also copy/deepcopy?) is corrected to work
atomic on dicts (use .keys()) and list-copies or locks python threads)
the problem is solved gracely and generally.

Robert



Mar 11 '06 #6
EleSSaR^ wrote:
robert si Ť profuso/a a scrivere su comp.lang.pytho n tutte queste
elucubrazioni:

[cut]

I don't know what's your code like, but a similar error occurred in some of
my software and it was my fault indeed. I think you should either use a
lock, or implement a deepcopy method of your own.


100s of locks? no (see other message). It should be

own deepcopy: thus, do you already know if the existing deepcopy has the
same problem as cPickle.dump ? (as the problem araises rarely, it is
difficult for me to test it out)

Robert

PS: how does ZODB work with this kind of problem? I thought is uses cPickle?
Mar 11 '06 #7
robert <no*****@no-spam-no-spam.com> wrote:
...
99.99% no. I would have to use a lock everywhere, where I add or remove
something into a dict or list of the struct. Thats not the purpose of
big thread locks. Such simple operations are already atomic by the
definition of Python - and thanks to the global interpreter lock.
(Otherwise I would leave the Python language, God beware ... :-) )


You have misread the Python Language Reference -- if you can give the
URL on which you have read any such promise of atomicity, I will be glad
to fix the docs to make that unambiguous.

There is no such promise (there may be implementation accidents in some
specific implementation which happen to make some operation atomic, but
NO guarantee even there that the next bugfix won't break that).

Farwell and best of luck in finding other languages which support
threads in a way that is more to your liking than Python -- maybe Ruby
suits you, I don't know for sure though.
Alex
Mar 11 '06 #8
robert si Ť profuso/a a scrivere su comp.lang.pytho n tutte queste
elucubrazioni:
own deepcopy: thus, do you already know if the existing deepcopy has the
same problem as cPickle.dump ? (as the problem araises rarely, it is
difficult for me to test it out)
I don't know the exact specs of your object, and I don't know what
operations are you performing on that object, nor the way they're atomic.

It seems like you're trying to save periodically the state of such object
while it is being modified (a sort of backup?), and Python complains about
that. A self-implemented deepcopy might raise anomalies (i.e. your dumped
object may be partly a 'before' object and partly an 'after' object ) as
well.

By the way, you could try employing locks from other threads to dump the
object as well... this would prevent additional locking.
PS: how does ZODB work with this kind of problem? I thought is uses cPickle?


I have no idea about this.
--
EleSSaR^ <us************ **********@yaho o.it>
--
Togli .xyz dalla mia email per contattarmi.
Mar 11 '06 #9
robert si Ť profuso/a a scrivere su comp.lang.pytho n tutte queste
elucubrazioni:

[cut]

P.S.
I'm very bad at threaded programming. Please verify any of my suggestions
^_^
--
EleSSaR^ <us************ **********@yaho o.it>
--
Togli .xyz dalla mia email per contattarmi.
Mar 11 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

16
3933
by: Dave Opstad | last post by:
In this snippet: d = {'x': 1} value = d.get('x', bigscaryfunction()) the bigscaryfunction is always called, even though 'x' is a valid key. Is there a "short-circuit" version of get that doesn't evaluate the second argument if the first is a valid key? For now I'll code around it, but this behavior surprised me a bit...
2
7875
by: microsoft | last post by:
When I create a instance of web services proxy in winform, I get the exception. Stack Trace: System.ArgumentException: Item has already been added. Key in dictionary:"winbootdir" Key being added: "winbootdir" at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add) at System.Collections.Hashtable.Add(Object key, Object value)
11
3133
by: Roy Lawson | last post by:
I have no idea what is going on here. I wrote a simple application in VB.NET to generate a Crystal Report, and I am now trying to move it to ASP.NET with Crstal Enterprise. I wish I could tell you how the report works...but I can't even get past connecting to the database :-) I use the typical: Dim oConnection As New SqlConnection(strCon) Dim cmdText As String = "SELECT * FROM viewLogin WHERE " & _
0
4222
by: Ralf Gedrat | last post by:
Hello! I have a Application, this throws after some time following exception: Item has already been added. Key in dictionary: "- 1" key being added: "- 1" I use Application.Run with ApplicationContext. This error message comes from deeper levels must be thrown (mscorlib.dll?!) ?.
1
3142
by: nobrainer | last post by:
Hi, I'm a linguist and have a large self-built bilingual dictionary. I want to turn it into a electronic dictionary, can anyone tell me how to start with it? I've some knowledge in Java programming, but don't know how to write a programme that allows you to search in the large e-file just like a GUI interface e-dictionary. any directions?
0
1935
by: Erich93063 | last post by:
I am using Cold Fusion to access an ODBC connection for an accounting program called MYOB. They provide the ODBC driver that you can install. I have it installed and I used there utility to check the connection and its successful. I then try to run a very basic query in Cold Fusion and I get the folowing error: Error Executing Database Query. Option value changed - SQLSetStmtOption:RowSet Size 1149 requested, changed to 1
7
16375
by: PW | last post by:
Hi, I have a form with unbound fields on it. The user selects a record from a recordset and I populate the unbound fields. When I try to change the unbound quantity text box, Access 2003 tells me "The data has been changed. Another user edited this record and saved the changes before you attempted to save your changes. Re-edit the record." This does not always happen. And yes, I can re-edit the quantity text
2
2118
by: joe.kimbler | last post by:
What is the best way to handle updates in databases with each release of a software package? I used to work on an accounting package in FoxPro that had a "Data Dictionary" and as your code required changes to the database per release, you'd update the Data Dictionary. When you wanted to create a new database, it would create it completely from the data dictionary. If you already had a functional database and the new release required...
2
1455
by: slais-www | last post by:
Before I set out to reinvent the wheel. I want a dictionary-like structure for which most lookups will be on a key that is not in the "dictionary"; in which case I want a key/value returned which is the closest key that is less than the search term. The batch solution of matching the values in two sorted sequences is efficient but a random lookup would be more convenient. A solution based on AVL trees seems possible.
0
9928
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
11100
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10718
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10816
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9554
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectóplanning, coding, testing, and deploymentówithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7110
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5781
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
2
4196
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3225
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.