473,572 Members | 3,179 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Profiling, sum-comprehension vs reduce

cnb
This must be because of implementation right? Shouldn't reduce be
faster since it iterates once over the list?
doesnt sum first construct the list then sum it?

-----------------------
>>============= =============== ==== RESTART =============== =============== ==
reduce with named function: 37.9864357062
reduce with nested, named function: 39.4710288598
reduce with lambda: 39.2463927678
sum comprehension: 25.9530121845
>>============= =============== ==== RESTART =============== =============== ==
reduce with named function: 36.4529584067
reduce with nested, named function: 37.6278529813
reduce with lambda: 38.2629448715
sum comprehension: 26.0197561422
>>>


from timeit import Timer

def add(x,y):
return x+y

def rednamed(lst):
return reduce(add, lst)

def rednn(lst):
def add2(x,y):
return x+y
return reduce(add2, lst)

def redlambda(lst):
return reduce(lambda x,y:x+y, lst)

def com(lst):
return sum(x for x in lst)

s = xrange(101)

t1 = Timer('rednamed (s)', 'from __main__ import rednamed, s')
t2 = Timer('rednn(s) ', 'from __main__ import rednn, s')
t3 = Timer('redlambd a(s)', 'from __main__ import redlambda, s')
t4 = Timer('com(s)', 'from __main__ import com, s')

print "reduce with named function: ", t1.timeit()
print "reduce with nested, named function: ", t2.timeit()
print "reduce with lambda: ", t3.timeit()
print "sum comprehension: ", t4.timeit()
---------------------------------------
also, using range instead of xrange doesnt seem to generate a
performance-penalty:
>>>
reduce with named function: 36.7560729087
reduce with nested, named function: 38.5393266463
reduce with lambda: 38.3852953378
sum comprehension: 27.9001007111
>>>
Sep 13 '08 #1
7 3490
On Sat, 13 Sep 2008 01:06:22 -0700, cnb wrote:
This must be because of implementation right? Shouldn't reduce be faster
since it iterates once over the list? doesnt sum first construct the
list then sum it?
No it doesn't. Why should it?
also, using range instead of xrange doesnt seem to generate a
performance-penalty:
(De)Allocating a list of length 100 isn't very slow. Try some million
elements. And watch the memory consumption too.

Ciao,
Marc 'BlackJack' Rintsch
Sep 13 '08 #2
doesnt sum first construct the list then sum it?
def com(lst):
return sum(x for x in lst)
You construct a generator over an existing list in your code.
Try sum([x for x in lst]) to see the effect of additional list
construction. And while you're at it, try the simple sum(lst).

Cheers,

- harold -
Sep 13 '08 #3
Bas
On Sep 13, 10:06*am, cnb <circularf...@y ahoo.sewrote:
This must be because of implementation right? Shouldn't reduce be
faster since it iterates once over the list?
doesnt sum first construct the list then sum it?
No, sum also iterates the sequence just once and doesn't create a
list. It is probably implemented similar to

def sum(sequence, start=0):
it = iter(sequence)
total = start
for i in it:
total += i
return i

but then implemented in C for speed. Reduce is probably implemented
pretty similar, but then with a random function instead of addition.
Make sure that you understand the difference between generator
expression and list comprehension, and that [f(x) for x in something]
is (almost) equal to list(f(x) for x in something), so you can emulate
a LC by using the list constructor on the equivalent GE.

HTH,
Bas
Sep 13 '08 #4
On Sat, 13 Sep 2008 01:06:22 -0700, cnb wrote:
This must be because of implementation right? Shouldn't reduce be faster
since it iterates once over the list? doesnt sum first construct the
list then sum it?
What makes you think that?

Given the speed of sum(), it sure doesn't look like it's generating a
full list before summing. Why would it?

reduce with named function: 37.9864357062
reduce with nested, named function: 39.4710288598
reduce with lambda: 39.2463927678
sum comprehension: 25.9530121845
If you want to see reduce really shine, time it with a C-based function
rather than one written in pure Python:
>>Timer('reduce (add, xrange(10000))' ,
.... 'from operator import add').repeat(nu mber=10000)
[19.724750995635 986, 19.410486936569 214, 19.614511013031 006]
>>>
Timer('reduce (add, xrange(10000))' ,
.... 'def add(x, y): return x+y').repeat(nu mber=10000)
[45.210143089294 434, 44.814558982849 121, 46.906874895095 825]
You probably won't see much (if any) benefit for small lists, so make
sure your test is on a significantly-sized input list.

Of course, sum() is even faster than reduce:
>>Timer('sum(xr ange(10000))'). repeat(number=1 0000)
[9.8149249553680 42, 8.7169640064239 502, 9.5062401294708 252]

--
Steven
Sep 13 '08 #5


Steven D'Aprano wrote:
If you want to see reduce really shine, time it with a C-based function
rather than one written in pure Python:
>>>Timer('reduc e(add, xrange(10000))' ,
... 'from operator import add').repeat(nu mber=10000)
[19.724750995635 986, 19.410486936569 214, 19.614511013031 006]
>>>Timer('reduc e(add, xrange(10000))' ,
... 'def add(x, y): return x+y').repeat(nu mber=10000)
[45.210143089294 434, 44.814558982849 121, 46.906874895095 825]
....
Of course, sum() is even faster than reduce:
>>>Timer('sum(x range(10000))') .repeat(number= 10000)
[9.8149249553680 42, 8.7169640064239 502, 9.5062401294708 252]
'Of course', because the irreducible difference between reduce(add.seq)
and sum(seq) is that reduce has to call an add function while sum has
the operation built-in in place of a call.

Sep 13 '08 #6
Terry Reedy <tj*****@udel.e duwrites:
>Of course, sum() is even faster than reduce:
>>>>Timer('sum( xrange(10000))' ).repeat(number =10000)
[9.8149249553680 42, 8.7169640064239 502, 9.5062401294708 252]

'Of course', because the irreducible difference between
reduce(add.seq) and sum(seq) is that reduce has to call an add
function while sum has the operation built-in in place of a call.
Note that, despite appearances, it's not as built-in as one might
wish. sum(seq) is still completely generic and works on all
number-like objects (in fact on all objects that define an __add__
operation except strings, which are explicitly forbidden). This means
that it can't just go ahead and execute a C addition, it must properly
call PyNumber_Add (the C API call equivalent to Python's "+"
operator), which will then inspect the objects and invoke the
appropriate implementation of addition. On the other hand,
operator.add is defined to do exactly that -- call PyNumber_Add on its
arguments and return the result.

It's not entirely obvious why summing numbers using the same method,
repeatedly calling PyNumber_Add, would be significantly slower for
reduce than for sum. Admittedly reduce has to execute a Python-level
function call which requires allocating and filling out a tuple, only
to reach PyNumber_Add, which sum calls immediately. But
builtin_reduce( ) is actually careful to optimize those repeated
function calls, for example by reusing the same tuple across the loop.
Sep 13 '08 #7
On Sep 13, 9:27*pm, Hrvoje Niksic <hnik...@xemacs .orgwrote:
Note that, despite appearances, it's not as built-in as one might
wish. *sum(seq) is still completely generic and works on all
number-like objects (in fact on all objects that define an __add__
operation except strings, which are explicitly forbidden). *This means
that it can't just go ahead and execute a C addition, it must properly
call PyNumber_Add (the C API call equivalent to Python's "+"
operator), which will then inspect the objects and invoke the
appropriate implementation of addition.
The time machine strikes again! Try using Py2.6.
The built-in sum() function is much smarter and faster.
It does in fact use C addition.
Raymond
Sep 16 '08 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
3698
by: James Sleeman | last post by:
Hi all, I'm trying to find a performance problem I've got in a reasonably large system I'm writing. Anyway, I've had a look around at the offerings for profiling php requests... xdebug : seems to make my requests just hit the 30 sec timeout all the time. apd : segfaults every time I make a request dbg : no front-end for linux (I looked...
0
464
by: Irmen de Jong | last post by:
Okay I tried some profiling, but am uncertain about the results I'm getting. They confuse the hell out of me. I have a test program (see below) that essentially has two loops that get called repeatedly. One that is an idle wait loop, and one that is a busy-wait CPU hogger. I wanted to see what profiling results that would give. The total...
3
10975
by: Richard Wallace | last post by:
Hello all, I'm looking for some input on the best tools to use for profiling multithreaded C++ code developed on GNU/Linux and compiled using gcc-3.1. More specifically, the distro in use is RH 7.2 running kernel 2.4.7 with SMP. The box has dual processors. Some of the options I've found are gprof - standard GNU profiler that comes...
6
3738
by: cournape | last post by:
Hi there, I have some scientific application written in python. There is a good deal of list processing, but also some "simple" computation such as basic linear algebra involved. I would like to speed things up implementing some of the functions in C. So I need profiling. I first tried to use the default python profiler, but profiling my...
13
2255
by: Jens Theisen | last post by:
Hello, I want to apologise in advance for this being off topic. It's not neither A C nor a C++ question, but to profiling in general, though I my chances are best to find the answer in the C/C++ community. I have a C++ program to profile and went about it by producing large history files of calling dependencies with associated times. It...
0
2556
by: EdgarSanchez | last post by:
Hello Group, This mail is intended to announce the release of the newest version of SpeedTrace .NET Tracing & Profiling tool, SpeedTrace Pro. This tool is an improvement of the previous version that was launched in this group last year. SpeedTrace Pro has come up with the following new features: · Support .NET Framework 2.0 (Windows...
1
2026
by: =?UTF-8?B?TWFydGluIFDDtnBwaW5n?= | last post by:
Hello, I am searching for a good profiling tool for C# which can be easily embedded into Visual Studio. My objective is to measure the times and counts of every method in my program. If used the Coverage Tool of TestDriven.Net (http://www.testdriven.net/) which worked fine if I have defined any Unit test-cases.
0
2570
by: L'eau Prosper Research | last post by:
Press Release: L'eau Prosper Research (Website: http://www.leauprosper.com) releases new TradeStation 8 Add-on - L'eau Prosper Market Manipulation Profiling Tools Set. L'eau Prosper Market Manipulation Profiling Tools Set is a set of advanced tools that help Serious Traders analyze the market direction, market manipulative behavior and...
0
2339
by: L'eau Prosper Research | last post by:
NEW TradeStation 8 Add-on - L'eau Prosper Market Manipulation Profiling Tools Set By L'eau Prosper Research Press Release: L'eau Prosper Research (Website: http://www.leauprosper.com) releases new TradeStation 8 Add-on - L'eau Prosper Market Manipulation Profiling Tools Set. L'eau Prosper Market Manipulation Profiling Tools Set is a...
9
1795
by: Peter Duniho | last post by:
I'm especially hoping Ben Voigt and/or Bob Powell see this (I saw their names on the m.p.d.f.performance newsgroup :) ) I would have posted to the performance newsgroup, but I see very little on there that actually seems to relate to the _tools_ per se while this newsgroup is actually somewhat related to the tools, and most of the useful...
0
7957
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8155
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7705
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
6337
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5524
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5248
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3685
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3673
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2138
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.