473,699 Members | 2,834 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Why are functions atomic?

Why are functions atomic? (I.e. they are not copied.)

For example, I would like to make a copy of a function so I can change
the default values:
>>from copy import copy
f = lambda x: x
f.func_defaul ts = (1,)
g = copy(f)
g.func_defaul ts = (2,)
f(),g()
(2, 2)

I would like the following behaviour:
>>f(),g()
(1,2)

I know I could use a 'functor' defining __call__ and using member
variables, but this is more complicated and quite a bit slower. (I
also know that I can use new.function to create a new copy, but I
would like to know the rational behind the decision to make functions
atomic before I shoot myself in the foot;-)

Thanks,
Michael.

May 1 '07
31 1776
al***@mac.com (Alex Martelli) wrote:
>Is there a reason for using the closure here? Using function
defaults seems to give better performance:

What measurements show you that...?
....
>
brain:~ alex$ python -mtimeit -s'import powi; p=powi.powerfac tory1(3)'
'p(27)'
1000000 loops, best of 3: 0.485 usec per loop

brain:~ alex$ python -mtimeit -s'import powi; p=powi.powerfac tory2(3)'
'p(27)'
1000000 loops, best of 3: 0.482 usec per loop
Your own benchmark seems to support Michael's assertion although the
difference in performance is so slight that it is unlikely ever to
outweigh the loss in readability.

Modifying powi.py to reduce the weight of the function call overhead and
the exponent operation indicates that using default arguments is faster,
but you have to push it to quite an extreme case before it becomes
significant:

def powerfactory1(e xponent, plus):
def inner(x):
for i in range(1000):
res = x+exponent+plus
return res
return inner

def powerfactory2(e xponent, plus):
def inner(x, exponent=expone nt, plus=plus):
for i in range(1000):
res = x+exponent+plus
return res
return inner

C:\Temp>\python 25\python -mtimeit -s "import powi; p=powi.powerfac tory1
(3,999)" "p(27)"
10000 loops, best of 3: 159 usec per loop

C:\Temp>\python 25\python -mtimeit -s "import powi; p=powi.powerfac tory2
(3,999)" "p(27)"
10000 loops, best of 3: 129 usec per loop
May 3 '07 #21
On May 2, 6:08 am, Carsten Haese <cars...@uniqsy s.comwrote:
On Tue, 2007-05-01 at 22:21 -0700, Michael wrote:
Is there a reason for using the closure here? Using function defaults
seems to give better performance:[...]

It does? Not as far as I can measure it to any significant degree on my
computer.
I agree the performance gains are minimal. Using function defaults
rather than closures, however, seemed much cleaner an more explicit to
me. For example, I have been bitten by the following before:
>>def f(x):
.... def g():
.... x = x + 1
.... return x
.... return g
>>g = f(3)
g()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in g
UnboundLocalErr or: local variable 'x' referenced before assignment

If you use default arguments, this works as expected:
>>def f(x):
.... def g(x=x):
.... x = x + 1
.... return x
.... return g
>>g = f(3)
g()
4

The fact that there also seems to be a performance gain (granted, it
is extremely slight here) led me to ask if there was any advantage to
using closures. It seems not.
An overriding theme in this thread is that you are greatly concerned
with the speed of your solution rather than the structure and
readability of your code.
Yes, it probably does seem that way, because I am burying this code
deeply and do not want to revisit it when profiling later, but my
overriding concern is reliability and ease of use. Using function
attributes seemed the best way to achieve both goals until I found out
that the pythonic way of copying functions failed. Here was how I
wanted my code to work:

@define_options (first_option=' abs_tol')
def step(f,x,J,abs_ tol=1e-12,rel_tol=1e-8,**kwargs):
"""Take a step to minimize f(x) using the jacobian J.
Return (new_x,converge d) where converged is true if the tolerance
has been met.
"""
<compute dx and check convergence>
return (x + dx, converged)

@define_options (first_option=' min_h')
def jacobian(f,x,mi n_h=1e-6,max_h=0.1):
"""Compute jacobian using a step min_h < h < max_h."""
<compute J>
return J

class Minimizer(objec t):
"""Object to minimize a function."""
def __init__(self,s tep,jacobian,** kwargs):
self.options = step.options + jacobian.option s
self.step = step
self.jacobian = jacobian

def minimize(self,f ,x0,**kwargs):
"""Minimize the function f(x) starting at x0."""
step = self.step
jacobian = self.jacobian

step.set_option s(**kwargs)
jacobian.set_op tions(**kwargs)

converged = False
while not converged:
J = jacobian(f,x)
(x,converged) = step(f,x,J)

return x

@property
def options(self):
"""List of supported options."""
return self.options

The idea is that one can define different functions for computing the
jacobian, step etc. that take various parameters, and then make a
custom minimizer class that can provide the user with information
about the supported options etc.

The question is how to define the decorator define_options?

1) I thought the cleanest solution was to add a method f.set_options()
which would set f.func_defaults , and a list f.options for
documentation purposes. The docstring remains unmodified without any
special "wrapping", step and jacobian are still "functions" and
performance is optimal.

2) One could return an instance f of a class with f.__call__,
f.options and f.set_options defined. This would probably be the most
appropriate OO solution, but it makes the decorator much more messy,
or requires the user to define classes rather than simply define the
functions as above. In addition, this is at least a factor of 2.5
timese slower on my machine than option 1) because of the class
instance overhead. (This is my only real performance concern because
this is quite a large factor. Otherwise I would just use this
method.)

3) I could pass generators to Minimize and construct the functions
dynamically. This would have the same performance, but would require
the user to define generators, or require the decorator to return a
generator when the user appears to be defining a function. This just
seems much less elegant.

....
@define_options _generator(firs t_option='min_h ')
def jacobian_gen(f, x,min_h=1e-6,max_h=0.1):
"""Compute jacobian using a step min_h < h < max_h."""
<compute J>
return J

class Minimizer(objec t):
"""Object to minimize a function."""
def __init__(self,s tep_gen,jacobia n_gen,**kwargs) :
self.options = step_gen.option s + jacobian_gen.op tions
self.step_gen = step_gen
self.jacobian_g en = jacobian_gen

def minimize(self,f ,x0,**kwargs):
"""Minimize the function f(x) starting at x0."""
step = self.step_gen(* *kwargs)
jacobian = self.jacobian_g en(**kwargs)

converged = False
while not converged:
J = jacobian(f,x)
(x,converged) = step(f,x,J)

return x
...

4) Maybe there is a better, cleaner way to do this, but I thought that
my option 1) was the most clear, readable and fast. I would
appreciate any suggestions. The only problem is that it does use
mutable functions, and so the user might be tempted to try:

new_step = copy(step)

which would fail (because modifying new_step would also modify step).
I guess that this is a pretty big problem (I could provide a custom
copy function so that

new_step = step.copy()

would work) and I wondered if there was a better solution (or if maybe
copy.py should be fixed. Checking for a defined __copy__ method
*before* checking for pre-defined mutable types does not seem to break
anything.)

Thanks again everyone for your suggestions, it is really helping me
learn about python idioms.

Michael.

May 4 '07 #22
On May 4, 1:36 am, Michael <michael.for... @gmail.comwrote :
On May 2, 6:08 am, Carsten Haese <cars...@uniqsy s.comwrote:
On Tue, 2007-05-01 at 22:21 -0700, Michael wrote:
Is there a reason for using the closure here? Using function defaults
seems to give better performance:[...]
It does? Not as far as I can measure it to any significant degree on my
computer.

I agree the performance gains are minimal. Using function defaults
rather than closures, however, seemed much cleaner an more explicit to
me. For example, I have been bitten by the following before:
>def f(x):

... def g():
... x = x + 1
... return x
... return g>>g = f(3)
>g()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in g
UnboundLocalErr or: local variable 'x' referenced before assignment

If you use default arguments, this works as expected:>>def f(x):

... def g(x=x):
... x = x + 1
... return x
... return g
>g = f(3)
g()

4
>>g()
4
>>g()
4
>>g() # what is going on here????
4
The fact that there also seems to be a performance gain (granted, it
is extremely slight here) led me to ask if there was any advantage to
using closures. It seems not.
An overriding theme in this thread is that you are greatly concerned
with the speed of your solution rather than the structure and
readability of your code.

Yes, it probably does seem that way, because I am burying this code
deeply and do not want to revisit it when profiling later, but my
overriding concern is reliability and ease of use. Using function
attributes seemed the best way to achieve both goals until I found out
that the pythonic way of copying functions failed. Here was how I
wanted my code to work:

@define_options (first_option=' abs_tol')
def step(f,x,J,abs_ tol=1e-12,rel_tol=1e-8,**kwargs):
"""Take a step to minimize f(x) using the jacobian J.
Return (new_x,converge d) where converged is true if the tolerance
has been met.
"""
<compute dx and check convergence>
return (x + dx, converged)

@define_options (first_option=' min_h')
def jacobian(f,x,mi n_h=1e-6,max_h=0.1):
"""Compute jacobian using a step min_h < h < max_h."""
<compute J>
return J

class Minimizer(objec t):
"""Object to minimize a function."""
def __init__(self,s tep,jacobian,** kwargs):
self.options = step.options + jacobian.option s
self.step = step
self.jacobian = jacobian

def minimize(self,f ,x0,**kwargs):
"""Minimize the function f(x) starting at x0."""
step = self.step
jacobian = self.jacobian

step.set_option s(**kwargs)
jacobian.set_op tions(**kwargs)

converged = False
while not converged:
J = jacobian(f,x)
(x,converged) = step(f,x,J)

return x

@property
def options(self):
"""List of supported options."""
return self.options

The idea is that one can define different functions for computing the
jacobian, step etc. that take various parameters, and then make a
custom minimizer class that can provide the user with information
about the supported options etc.

The question is how to define the decorator define_options?

1) I thought the cleanest solution was to add a method f.set_options()
which would set f.func_defaults , and a list f.options for
documentation purposes. The docstring remains unmodified without any
special "wrapping", step and jacobian are still "functions" and
performance is optimal.

2) One could return an instance f of a class with f.__call__,
f.options and f.set_options defined. This would probably be the most
appropriate OO solution, but it makes the decorator much more messy,
or requires the user to define classes rather than simply define the
functions as above. In addition, this is at least a factor of 2.5
timese slower on my machine than option 1) because of the class
instance overhead. (This is my only real performance concern because
this is quite a large factor. Otherwise I would just use this
method.)

3) I could pass generators to Minimize and construct the functions
dynamically. This would have the same performance, but would require
the user to define generators, or require the decorator to return a
generator when the user appears to be defining a function. This just
seems much less elegant.

...
@define_options _generator(firs t_option='min_h ')
def jacobian_gen(f, x,min_h=1e-6,max_h=0.1):
"""Compute jacobian using a step min_h < h < max_h."""
<compute J>
return J

class Minimizer(objec t):
"""Object to minimize a function."""
def __init__(self,s tep_gen,jacobia n_gen,**kwargs) :
self.options = step_gen.option s + jacobian_gen.op tions
self.step_gen = step_gen
self.jacobian_g en = jacobian_gen

def minimize(self,f ,x0,**kwargs):
"""Minimize the function f(x) starting at x0."""
step = self.step_gen(* *kwargs)
jacobian = self.jacobian_g en(**kwargs)

converged = False
while not converged:
J = jacobian(f,x)
(x,converged) = step(f,x,J)

return x
...

4) Maybe there is a better, cleaner way to do this, but I thought that
my option 1) was the most clear, readable and fast. I would
appreciate any suggestions. The only problem is that it does use
mutable functions, and so the user might be tempted to try:

new_step = copy(step)

which would fail (because modifying new_step would also modify step).
I guess that this is a pretty big problem (I could provide a custom
copy function so that

new_step = step.copy()

would work) and I wondered if there was a better solution (or if maybe
copy.py should be fixed. Checking for a defined __copy__ method
*before* checking for pre-defined mutable types does not seem to break
anything.)

Thanks again everyone for your suggestions, it is really helping me
learn about python idioms.

Michael.

May 4 '07 #23
On 3 May 2007 23:36:11 -0700, Michael <mi************ @gmail.comwrote :
On May 2, 6:08 am, Carsten Haese <cars...@uniqsy s.comwrote:
On Tue, 2007-05-01 at 22:21 -0700, Michael wrote:
Is there a reason for using the closure here? Using function defaults
seems to give better performance:[...]
It does? Not as far as I can measure it to any significant degree on my
computer.

I agree the performance gains are minimal. Using function defaults
rather than closures, however, seemed much cleaner an more explicit to
me. For example, I have been bitten by the following before:
>def f(x):
... def g():
... x = x + 1
... return x
... return g
>g = f(3)
g()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in g
UnboundLocalErr or: local variable 'x' referenced before assignment

If you use default arguments, this works as expected:
>def f(x):
... def g(x=x):
... x = x + 1
... return x
... return g
>g = f(3)
g()
4
You aren't getting "bit" by any problem with closures - this is a
syntax problem.
Assigning to x within the scope of g() makes x a local variable. x =
x+1 doesn't work, then, because "x+1" can't be evaluated.

If you just used "return x+1" or if you named the local variable in g
something other than "x", the closure approach would also work as
expected.
May 4 '07 #24
Michael wrote:
On May 2, 6:08 am, Carsten Haese <cars...@uniqsy s.comwrote:
>>On Tue, 2007-05-01 at 22:21 -0700, Michael wrote:
I agree the performance gains are minimal. Using function defaults
rather than closures, however, seemed much cleaner an more explicit to
me. For example, I have been bitten by the following before:
>>>>def f(x):

... def g():
... x = x + 1
Too cute. Don't nest functions in Python; the scoping model
isn't really designed for it.
>>An overriding theme in this thread is that you are greatly concerned
with the speed of your solution rather than the structure and
readability of your code.
....
>
@define_options (first_option=' abs_tol')
def step(f,x,J,abs_ tol=1e-12,rel_tol=1e-8,**kwargs):
"""Take a step to minimize f(x) using the jacobian J.
Return (new_x,converge d) where converged is true if the tolerance
has been met.
"""
Python probably isn't the right language for N-dimensional optimization
if performance is a major concern. That's a very compute-intensive operation.
I've done it in C++, with heavy use of inlines, and had to work hard to
get the performance up. (I was one of the first to do physics engines for
games and animation, which is a rather compute-intensive problem.)

If you're doing number-crunching in Python, it's essential to use
NumPy or some other C library for matrix operations, or it's going to
take way too long.

John Nagle
May 4 '07 #25
On May 4, 5:49 am, "Chris Mellon" <arka...@gmail. comwrote:
You aren't getting "bit" by any problem with closures - this is a
syntax problem.
I understand that it is not closures that are specifically biting me.
However, I got bit, it was unplesant and I don't want to be bit
again;-)

Thus, whenever I need to pass information to a function, I use default
arguments now. Is there any reason not to do this other than the fact
that it is a bit more typing?

Michael

May 4 '07 #26
On May 4, 9:19 am, John Nagle <n...@animats.c omwrote:
... def g():
... x = x + 1

Too cute. Don't nest functions in Python; the scoping model
isn't really designed for it.
How can you make generators then if you don't nest?
Python probably isn't the right language for N-dimensional optimization
if performance is a major concern. That's a very compute-intensive operation.
I've done it in C++, with heavy use of inlines, and had to work hard to
get the performance up. (I was one of the first to do physics engines for
games and animation, which is a rather compute-intensive problem.)

If you're doing number-crunching in Python, it's essential to use
NumPy or some other C library for matrix operations, or it's going to
take way too long.
I know. I am trying to flesh out a modular optimization proposal for
SciPy. Using C++ would defeat the purpose of making it easy to extend
the optimizers. I just want to make things as clean and efficient as
possible when I stumbled on this python copy problem.

Michael.

May 4 '07 #27
On 4 May 2007 12:55:03 -0700, Michael <mi************ @gmail.comwrote :
On May 4, 5:49 am, "Chris Mellon" <arka...@gmail. comwrote:
You aren't getting "bit" by any problem with closures - this is a
syntax problem.

I understand that it is not closures that are specifically biting me.
However, I got bit, it was unplesant and I don't want to be bit
again;-)

Thus, whenever I need to pass information to a function, I use default
arguments now. Is there any reason not to do this other than the fact
that it is a bit more typing?
There are different semantics when the thing you're passing is
mutable. There's also different semantics when it's rebound within the
calling scope, but then the default argument technique is probably
what you want.
Michael

--
http://mail.python.org/mailman/listinfo/python-list
May 4 '07 #28
On 4 May 2007 12:59:39 -0700, Michael <mi************ @gmail.comwrote :
On May 4, 9:19 am, John Nagle <n...@animats.c omwrote:
... def g():
... x = x + 1
Too cute. Don't nest functions in Python; the scoping model
isn't really designed for it.

How can you make generators then if you don't nest?
There's all kinds of good reasons to nest functions, and the "scoping
model isn't really designed for it" somewhat overstates the case -
it's not relevant to many of the reasons you might nest functions, and
it's not (much) of a problem for the rest of them. What you can't do
is rebind values in the enclosing scope, unless the enclosing scope is
global. That's a real, but fairly minor, limitation and you'll be able
to explicitly address your enclosing scope in 3k (or perhaps sooner).
May 4 '07 #29
On May 4, 4:13 am, Dustan <DustanGro...@g mail.comwrote:
On May 4, 1:36 am, Michael <michael.for... @gmail.comwrote :
... def g(x=x):
... x = x + 1
... return x
... return g
>>g = f(3)
>>g()>
4
>g()
4
>g()
4
>g() # what is going on here????
4
Okay, so it is a bad design, but it illustrates the point. What is
happening is that in the body of the function f, a new function is
defined using the value of x passed as an argument to f. Thus, after
the call g = f(3), the body of f is equivalent to

def g(x=3):
x = x + 1
return x

This function is returned, so the call g() uses the default argument
x=3, then computes x = x+1 = 3+1 = 4 and returns 4. Every call is
equivalent to g() == g(3) = 4. Inside g, x is a local variable: it
does not maintain state between function calls. (You might think that
the first example would allow you to mutate the x in the closure, but
this is dangerous and exactly what python is trying to prevent by
making x a local variable when you make assignments in g. This is why
the interpreter complains.)

If you actually want to maintain state, you have to use a mutable
object like a list. The following would do what you seem to expect.
>>def f(x0):
.... def g(x=[x0]):
.... x[0] = x[0] + 1
.... return x[0]
.... return g
....
>>g = f(3)
g()
4
>>g()
5
>>h = f(0)
h()
1
>>h()
2
>>g()
6

May 4 '07 #30

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

99
5892
by: David MacQuigg | last post by:
I'm not getting any feedback on the most important benefit in my proposed "Ideas for Python 3" thread - the unification of methods and functions. Perhaps it was buried among too many other less important changes, so in this thread I would like to focus on that issue alone. I have edited the Proposed Syntax example below to take out the changes unecessary to this discussion. I left in the change of "instance variable" syntax (...
8
3572
by: Glenn Kasten | last post by:
I am wondering which operations in Python are guaranteed to be atomic in the presence of multi-threading. In particular, are assignment and reading of a dictionary entry atomic? For example, initially: dictionary = {} dictionary = old_value Then thread 1 does: v = dictionary And thread 2 does:
42
3348
by: Shayan | last post by:
Is there a boolean flag that can be set atomically without needing to wrap it in a mutex? This flag will be checked constantly by multiple threads so I don't really want to deal with the overhead of mutexes or semaphores. Thanks. Shayan
5
23679
by: Pegboy | last post by:
What does it mean to make a function atomic? Something I read on it wasn't very clear, but made me think that I needed to disable interrupts for that function. Whether that's the case or not, how would I make the following function atomic? int Test; int GetTestValue( void ) {
28
7386
by: robert | last post by:
In very rare cases a program crashes (hard to reproduce) : * several threads work on an object tree with dict's etc. in it. Items are added, deleted, iteration over .keys() ... ). The threads are "good" in such terms, that this core data structure is changed only by atomic operations, so that the data structure is always consistent regarding the application. Only the change-operations on the dicts and lists itself seem to cause problems...
6
6247
by: blackstreetcat | last post by:
consider this code : int i; //gobal var Thread1: i=some value; Thread2: if (i==2) dosomething(); else dosomethingelse();
11
2065
by: japhy | last post by:
Is there a way to read a line (a series of characters ending in a newline) from a file (either by descriptor or stream) atomically, without buffering additional contents of the file?
0
1929
by: yogeeswar | last post by:
Hi All I wrote SP with number of delete commands in an atomic block.And there is a possibility of deleting records from parent table before child table so I wrote a handler to handle the exception.in that handler I am using the ROLLBACK to rollback all the records which were deleted if any exception occurs. THE PROBLEM IS: It is working fine, if the deleting records size is small.but if i tried to delete thousands of records(lorge size)...
2
2933
by: Freedom fighter | last post by:
Hello, Is a singleton class the same as an atomic class? I know that a singleton class can only be instantiated once, but does that concept apply to an atomic class? Thank you.
0
8685
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
1
8905
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8880
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7743
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6532
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5869
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4373
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3053
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2342
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.