473,549 Members | 2,813 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Deterministic destruction and RAII idioms in Python

I have been dabbling in Python for a while now. One of the things that
really appeals to me is that I can seem to be able to use C++-style
RAII idioms to deal with resource management issues.

For those that have no idea what I am talking about (I learn a lot
reading posts on subjects in which I am clueless), consider the
following code snippet:

for line in file(name):
...print line,

This is nice and clean because I don't have to worry about cleaning
up after myself. If I can't rely on the destructor for the file
object to close the file, I must write the code like this:

file_obj = file(name)
for line in file_obj:
...print line,
file_obj.close( )

not nearly as nice. Depending on the type of work you do, this can be
no problem or a major headache when exceptions are thrown into the mix.

Python objects have destructors so it seems that the original intent
was to support deterministic destruction (destructors are not very
useful in most GC'ed languages since you never know when or even if
they will be called). The problem is that other implementations of
Python (Jython and IronPython) do not support deterministic
destruction. So we are left with a problem: is deterministic
destruction an implementation detail of CPython that can go away at
anytime, or is it an official property of the language.

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream " Python or am
I out of luck here?

Jan 30 '06 #1
9 2991

<pl****@alumni. caltech.edu> wrote in message
news:11******** *************@z 14g2000cwz.goog legroups.com...
The problem is that other implementations of
Python (Jython and IronPython) do not support deterministic
destruction. So we are left with a problem: is deterministic
destruction an implementation detail of CPython that can go away at
anytime, or is it an official property of the language.
It is a reliable implementation detail of CPython that will not go away
anytime soon. For the language def, see the Reference Manual section of
the del command.

Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream " Python or am
I out of luck here?


Depends on what you mean by 'mainstream'.

Terry Jan Reedy

Jan 30 '06 #2
pl****@alumni.c altech.edu wrote:
Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream " Python or am
I out of luck here?

Most people rely on that. I do that *all* the time, and I would really dislike
if CPython lose this feature. Anyway, if you want to make your code portable to
other Python implementations , you'll have to implement your RAII idioms in a
slightly different way. Specifically, Python supports a try/finally construct
that it is useful to properly release resources. Eg. the following code is
"correct" even with IronPython:

f = file(name):
try:
for L in f:
print L
finally:
f.close()

Anyway, there is some work being done to add some RAII constructs to the
language that are compatibile across Python implementations . See the new "with"
statement. http://www.python.org/peps/pep-0343.html. This would allow to write
generic "functors" for RAII using generators, such as:

@contextmanager
def opened(filename ):
f = open(filename)
try:
yield f
finally:
f.close()

These building blocks can then be used in the new 'with' statement such as:

with opened("foo.txt ") as f:
for L in f:
print L

Of course, there's more to this (eg, 'with' statement can be used also with
normal objects as long as they support new __enter__ and __exit__ special
method), so go to the link above for more details and more juicy examples that
I'm sure you'll like.
--
Giovanni Bajo
Jan 30 '06 #3
pl****@alumni.c altech.edu writes:
Giving up deterministic destruction in Python would be a real blow for
me, since it is one of its unique features among GC'ed languages.

So what's the deal, can I rely on it in "mainstream " Python or am
I out of luck here?


IMO you shouldn't rely on it. I believe it already doesn't work in
Jython, which uses the regular Java GC system. See PEP 343 for a new
(forthcoming) Python mechanism to do what you're looking for:

http://www.python.org/peps/pep-0343.html

You shouldn't think of reference counting as being like C++
destructors. A destructor definitely runs when the C++ object goes
out of scope. Ref counting frees only the object if there are no
other references when the scope exits. So if you want to rely on the
object getting gc'd when you expect, you have to carefully manage the
references, which defeats the point of the language doing it for you
automatically.
Jan 30 '06 #4
Thanks to all who replied. This has definitely given me something to
chew on.

I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction) but it is a lot more than most GC'ed
languages give me (Java, I am looking at you... :-) ).

As far as my comment about "mainstream " Python, I have always taken
CPython as "Python". I guess this will have to change as Jython and
IronPython bring interesting things to the table, even if they do take
some things away in the process.

Cheers!

Jan 31 '06 #5
pl****@alumni.c altech.edu writes:
I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction)
I think it's better.
As far as my comment about "mainstream " Python, I have always taken
CPython as "Python". I guess this will have to change as Jython and
IronPython bring interesting things to the table, even if they do take
some things away in the process.


Hopefully PyPy will become "mainstream ". How it handles GC, I'm not sure.
Jan 31 '06 #6
Hi Paul,
I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction)


I think it's better.


Is there something specific you have in mind that makes you say that?

I am not a python expert, so I probably do not understand all the
implications of the proposal. What I got out of it is summarized by:

with EXPR as VAR:
BLOCK

The translation of the above statement is:

abc = (EXPR).__contex t__()
exc = (None, None, None)
VAR = abc.__enter__()
try:
try:
BLOCK
except:
exc = sys.exc_info()
raise
finally:
abc.__exit__(*e xc)

Which, looks like you have a constructor (__enter__) and a destructor
(__exit__), which you can always count on being called. I am not sure
how that is better than C++-style RAII. Now, in the interest of full
disclosure, there is this __context__ method which I don't really
follow... maybe there is something important there I should know about?

At first glance, it seems to me that the syntax gets slightly ackward
as you encapsulate more resources. From the proposal:

with opened(filename , "w") as f:
with stdout_redirect ed(f):
print "Hello world"

which is not as nice as simply creating two objects and having them
destroyed in the reverse order that they were created in when they go
out of scope. Is it big deal... no. I am definitely going to have to
play with this stuff a while before I really know its strengths and
weakness relative to C++-style RAII.

If you have some examples, however, to support your feeling that it is
better than C++-style RAII, I would love to see them. I am always
eager to learn new stuff!

Again, I am not criticizing Python or the proposal. I like Python and
I like the capabilities that the proposal implies (to the extent that I
understand them :-) ).

As far as PyPy goes, thanks for reminding me of that. I had forgotten
about that one.

Cheers!

Jan 31 '06 #7
pl****@alumni.c altech.edu writes:
I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction)

I think it's better.

Is there something specific you have in mind that makes you say that?...
Which, looks like you have a constructor (__enter__) and a destructor
(__exit__), which you can always count on being called. I am not sure
how that is better than C++-style RAII.


I didn't understand then. I thought by "determinis tic destruction"
you meant relying on what CPython does now, which is reference
counting with destruction when the last reference is released.

PEP 343 is closer to C++-style RAII. I haven't studied it enough to
say what the differences really are. I do remember thinking C++ RAII
was a nuisance because you had to create classes with destructors to
do what other languages did with try/finally or the equivalent.

So to be clear, I think PEP 343 is better than simply expecting the
CPython reference counting system take care of releasing things when
the scope exits. I'm not much of a C++ user and I don't claim
anything about PEP 343 vs C++.
Jan 31 '06 #8
Hi Paul,
I didn't understand then. I thought by "determinis tic destruction"
you meant relying on what CPython does now, which is reference
counting with destruction when the last reference is released.
I effectively do mean that. In C++ it is guaranteed that a destructor
will be called on an object when it goes out of scope, no matter how
that happens (normal exit, exception, etc...). C++-style RAII just
refers to wrapping up resources in classes so that you don't need to
worry about explicitly freeing them.
I do remember thinking C++ RAII
was a nuisance because you had to create classes with destructors to
do what other languages did with try/finally or the equivalent.


It really depends how often you have to deal with getting/releasing
resouces. You can define a class once and then just deal the resource
in your code with a single line (creating the object). It can also
simplify things when you are dealing with the simultaneous management
of multiple resources. In any case, this is now getting off topic.

Thanks for your input! It was fun.

Jan 31 '06 #9
Paul Rubin wrote:
pl****@alumni.c altech.edu writes:
I looked at pep-0343, it looks interesting. It is not what I really
want (deterministic destruction)

I think it's better.

As far as my comment about "mainstream " Python, I have always taken
CPython as "Python". I guess this will have to change as Jython and
IronPython bring interesting things to the table, even if they do take
some things away in the process.

Hopefully PyPy will become "mainstream ". How it handles GC, I'm not sure.


Right now PyPy gives (at compile-time) the choice between two GCs: naive
reference counting and the conservative Boehm garbage collector
(http://research.hp.com/personal/Hans_Boehm/gc/). Currently we are
working on supporting other garbage collection strategies as well, like
mark-and-sweep, a copying collector or some sort of sophisticated
generational collector.

Cheers,

Carl Friedrich Bolz

Feb 5 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
4749
by: Pierre Rouleau | last post by:
As much as I love Python, I sometimes find myself wishing Python supported the RAII idiom (resource acquisition is initialization) that is available in C++, the emerging D language, and others. In some situations (like controlling non-memory resources) it is nice to be able to create an object that will execute some code on its destruction....
18
1796
by: Tad Marko | last post by:
Howdy! I'm trying to get my head around Python and I've come to realize that there are a lot of idioms in Python that are a bit different than in other languages. I have a class similar to what I've included below. The only difference is that in the real class, I have even more mandatory attributes. Too many to specify at once on a...
7
1566
by: Raymond Hettinger | last post by:
Copying ------- To copy lists and dictionaries, the traditional idioms were l=mylist and d=mydict.copy(). When some of the builtin functions became types, a more consistent approach became possible, l=list(mylist) and d=dict(mydict). In general, mutable types can be copied using their type constructor. This will also work with the new...
4
1241
by: MikeB | last post by:
Hi, Recently I was asked to look at some code where RAII is used to ensure automatic cleanup of a resource. Unfortunately, cleaning up the resource requires that the destructor make a call to an API which can (albeit under dire circumstances) fail. As it stands, in the presence of a failed call to the API, the destructor does nothing...
2
1266
by: Slawomir Lisznianski | last post by:
Do ref classes indeed support instantiation on the stack? Similarly, can native classes be instantiated on gc heap? When you try to have a ref class instance on the stack, you'll get: R : cannot use this type here without a top-level '^'. When you try to instantiate a native class using gcnew using Visual Studio 2005 Beta 1, you'll get:
65
4154
by: Steven Watanabe | last post by:
I know that the standard idioms for clearing a list are: (1) mylist = (2) del mylist I guess I'm not in the "slicing frame of mind", as someone put it, but can someone explain what the difference is between these and: (3) mylist =
6
1329
by: conkerll | last post by:
After several years developing in Java, I've begun to switch to Python for several of my new projects as I have found the language quite interesting. I've read several tutorials and implemented a few sample programs and I've found that Python enables one to program in a variety of different styles (I'm not sure if this was the original...
48
4403
by: Paul Melis | last post by:
Hello, I've always been using the has_key() method to test if a dictionary contains a certain key. Recently I tried the same using 'in', e.g. d = { ... } if k in d: ... and found that it seems to perform a lot better when lots of key-tests
5
1364
by: John Fisher | last post by:
I am working on a framework for data acquisition in Python 2.5, am trying to get a structure going more like this: mark start time start event event finishes count time until next interval start second event…
0
7554
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7480
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
7751
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
1
7511
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6088
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5396
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3525
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3509
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1087
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.