Method much slower than function?

[idoerg]

Hi all,

I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.

############### ## START SOURCE #############
# The function

def readgenome(file handle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
return s

# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
self.s += line.strip()

############### ## END SOURCE ##############
When running the function and the method on a 20,000 line text file, I
get the following:

>>cProfile.run( "bar.readgenome (open('cb_foo') )")

20004 function calls in 10.214 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 10.214 10.214 <string>:1(<mod ule>)
1 10.205 10.205 10.214 10.214 reader.py:11(re adgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.009 0.000 0.009 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}

>>cProfile.run( "z=r.readgenome (open('cb_foo') )")

20004 function calls in 0.041 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 0.041 0.041 <string>:1(<mod ule>)
1 0.035 0.035 0.041 0.041 reader.py:2(rea dgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.007 0.000 0.007 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}
The method takes 10 seconds, the function call 0.041 seconds!

Yes, I know that I wrote the underlying code rather inefficiently, and
I can streamline it with a single file.read() call instead if an
xreadlines() + strip loop. Still, the differences in performance are
rather staggering! Any comments?

Thanks,

Iddo

[Neil Cerutti]

On 2007-06-14, id****@gmail.co m <id****@gmail.c omwrote:

Hi all,

I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.

############### ## START SOURCE #############
# The function

def readgenome(file handle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
return s

# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
self.s += line.strip()

############### ## END SOURCE ##############
When running the function and the method on a 20,000 line text file, I
get the following:

>>>cProfile.run ("bar.readgenom e(open('cb_foo' ))")

20004 function calls in 10.214 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 10.214 10.214 <string>:1(<mod ule>)
1 10.205 10.205 10.214 10.214 reader.py:11(re adgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.009 0.000 0.009 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}

>>>cProfile.run ("z=r.readgenom e(open('cb_foo' ))")

20004 function calls in 0.041 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 0.041 0.041 <string>:1(<mod ule>)
1 0.035 0.035 0.041 0.041 reader.py:2(rea dgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.007 0.000 0.007 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}
The method takes 10 seconds, the function call 0.041 seconds!

Yes, I know that I wrote the underlying code rather
inefficiently, and I can streamline it with a single
file.read() call instead if an xreadlines() + strip loop.
Still, the differences in performance are rather staggering!
Any comments?

It is likely the repeated attribute lookup, self.s, that's
slowing it down in comparison to the non-method version.

Try the following simple optimization, using a local variable
instead of an attribute to build up the result.

# The method in a class
class bar:
def readgenome(self , filehandle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
self.s = s

To further speed things up, think about using the str.join idiom
instead of str.+=, and using a generator expression instead of an
explicit loop.

# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''.join(line.st rip() for line in filehandle)

--
Neil Cerutti

[Gabriel Genellina]

En Wed, 13 Jun 2007 21:40:12 -0300, <id****@gmail.c omescribió:

Hi all,

I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.

############### ## START SOURCE #############
# The function

def readgenome(file handle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
return s

# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
self.s += line.strip()

In the function above, s is a local variable, and accessing local
variables is very efficient (using an array of local variables, the
compiler assigns statically an index for each one).
Using self.s, on the other hand, requires a name lookup for each access.
The most obvious change would be to use a local variable s, and assign
self.s = s only at the end. This should make both methods almost identical
in performance.
In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)
And since you have Python 2.5, you can use the file as its own iterator;
combining all this:

return ''.join(line.st rip() for line in filehandle if '>' not in line)

--
Gabriel Genellina

[Grant Edwards]

On 2007-06-14, id****@gmail.co m <id****@gmail.c omwrote:

The method takes 10 seconds, the function call 0.041 seconds!

What happens when you run them in the other order?

The first time you read the file, it has to read it from disk.
The second time, it's probably just reading from the buffer
cache in RAM.

--
Grant Edwards grante Yow! Catsup and Mustard
at all over the place! It's
visi.com the Human Hamburger!

[sjdevnull]

Gabriel Genellina wrote:

In the function above, s is a local variable, and accessing local
variables is very efficient (using an array of local variables, the
compiler assigns statically an index for each one).
Using self.s, on the other hand, requires a name lookup for each access.
The most obvious change would be to use a local variable s, and assign
self.s = s only at the end. This should make both methods almost identical
in performance.

Yeah, that's a big deal and makes a significant difference on my
machine.

In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)

Ehh. Python 2.5 (and probably some earlier versions) optimize += on
strings pretty well.

a=""
for i in xrange(100000):
a+="a"

and:

a=[]
for i in xrange(100000):
a.append("a")
a="".join(a)

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue
here.

And since you have Python 2.5, you can use the file as its own iterator;
combining all this:

return ''.join(line.st rip() for line in filehandle if '>' not in line)

That's probably pretty good.

[Paul Rubin]

"sj*******@yaho o.com" <sj*******@yaho o.comwrites:

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue here.

You should not rely on using 2.5 or even on that optimization staying in
CPython. Best is to use StringIO or something comparable.

[Gabriel Genellina]

En Thu, 14 Jun 2007 01:39:29 -0300, sj*******@yahoo .com
<sj*******@yaho o.comescribió:

Gabriel Genellina wrote:

>In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)

Ehh. Python 2.5 (and probably some earlier versions) optimize += on
strings pretty well.

a=""
for i in xrange(100000):
a+="a"

and:

a=[]
for i in xrange(100000):
a.append("a")
a="".join(a)

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue
here.

Yes, for concatenating a lot of a's, sure... Try again using strings
around the size of your expected lines - and make sure they are all
different too.

pyimport timeit
py>
pydef f1():
.... a=""
.... for i in xrange(100000):
.... a+=str(i)*20
....
pydef f2():
.... a=[]
.... for i in xrange(100000):
.... a.append(str(i) *20)
.... a="".join(a)
....
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)

....after a few minutes I aborted the process...

--
Gabriel Genellina

[Leo Kislov]

On Jun 13, 5:40 pm, [e-mail] m wrote:

Hi all,

I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.

>cProfile.run(" bar.readgenome( open('cb_foo')) ")

20004 function calls in 10.214 CPU seconds

>cProfile.run(" z=r.readgenome( open('cb_foo')) ")

20004 function calls in 0.041 CPU seconds

I suspect open files are cached so the second reader
picks up where the first one left: at the of the file.
The second call doesn't do any text processing at all.

-- Leo

[Bruno Desthuilliers]

Neil Cerutti a écrit :
(snip)

class bar:
def readgenome(self , filehandle):
self.s = ''.join(line.st rip() for line in filehandle)

=>
self.s = ''.join(line.st rip() for line in filehandle if not
'>' in line)

[Peter Otten]

Gabriel Genellina wrote:

En Thu, 14 Jun 2007 01:39:29 -0300, sj*******@yahoo .com
<sj*******@yaho o.comescribió:

>Gabriel Genellina wrote:

>>In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)

Ehh. Python 2.5 (and probably some earlier versions) optimize += on
strings pretty well.

a=""
for i in xrange(100000):
a+="a"

and:

a=[]
for i in xrange(100000):
a.append("a")
a="".join(a)

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue
here.

Yes, for concatenating a lot of a's, sure... Try again using strings
around the size of your expected lines - and make sure they are all
different too.

pyimport timeit
py>
pydef f1():
... a=""
... for i in xrange(100000):
... a+=str(i)*20
...
pydef f2():
... a=[]
... for i in xrange(100000):
... a.append(str(i) *20)
... a="".join(a)
...
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)

...after a few minutes I aborted the process...

I can't confirm this.

$ cat join.py
def f1():
a = ""
for i in xrange(100000):
a += str(i)*20

def f2():
a = []
for i in xrange(100000):
a.append(str(i) *20)
a = "".join(a)

def f3():
a = []
append = a.append
for i in xrange(100000):
append(str(i)*2 0)
a = "".join(a)

$ python2.5 -m timeit -s 'from join import f1' 'f1()'
10 loops, best of 3: 212 msec per loop
$ python2.5 -m timeit -s 'from join import f2' 'f2()'
10 loops, best of 3: 259 msec per loop
$ python2.5 -m timeit -s 'from join import f3' 'f3()'
10 loops, best of 3: 236 msec per loop

Peter