timeit.timeit and timeit.repeat give different answers

[Dan Christensen]

The test below is done with Python 2.4a1 compiled from source, but
the same thing happens with Debian's python2.3_2.3.4-2.

Python 2.4a1 (#1, Jul 11 2004, 12:20:32)
[GCC 3.3.4 (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import timeit
import time
t=timeit.Timer("pass","pass",time.time)
print t.timeit() 0.0560228824615 print t.repeat(1)

[0.042751073837280273]

The results are consistent if repeated, i.e. timeit always produces
about 0.056 and repeat always produces about 0.043.

It doesn't help to do:

for i in range(10):
print t.timeit()

vs.

print t.repeat(10)

The problem happens on both machines I've tested it on. One is a
desktop. No other jobs running.

With Python 2.3, the results are affected by the setting of the
PYTHONPATH environment variable. With it unset or set to an empty
directory /tmp/foo I get results like above. With it set to an empty
directory /tmp/python, I get results that agree with each other.

But with Python 2.4, all three settings give results that disagree.

If I use the ipython shell, things are even worse: the results
disagree by a factor of almost 2!

I've read timeit.py and can't see how this might happen.

Any thoughts?

Dan

[Dan Christensen]

Can anyone confirm whether this discrepancy happens with other
installations of python on other hardware/OS's? It's a bit
disconcerting.

Thanks,

Dan

Dan Christensen <jd*@uwo.ca> writes:

The test below is done with Python 2.4a1 compiled from source, but
the same thing happens with Debian's python2.3_2.3.4-2.

Python 2.4a1 (#1, Jul 11 2004, 12:20:32)
[GCC 3.3.4 (Debian)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import timeit
import time
t=timeit.Timer("pass","pass",time.time)
print t.timeit() 0.0560228824615 print t.repeat(1)

[0.042751073837280273]

The results are consistent if repeated, i.e. timeit always produces
about 0.056 and repeat always produces about 0.043.

It doesn't help to do:

for i in range(10):
print t.timeit()

vs.

print t.repeat(10)

The problem happens on both machines I've tested it on. One is a
desktop. No other jobs running.

With Python 2.3, the results are affected by the setting of the
PYTHONPATH environment variable. With it unset or set to an empty
directory /tmp/foo I get results like above. With it set to an empty
directory /tmp/python, I get results that agree with each other.

But with Python 2.4, all three settings give results that disagree.

If I use the ipython shell, things are even worse: the results
disagree by a factor of almost 2!

I've read timeit.py and can't see how this might happen.

Any thoughts?

Dan

[Peter Otten]

Dan Christensen wrote:

Can anyone confirm whether this discrepancy happens with other
installations of python on other hardware/OS's? It's a bit
disconcerting.

Measured on an aging Suse 8.1:

Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
[GCC 3.2] on linux2
Type "help", "copyright", "credits" or "license" for more informa
tion.

import timeit, time
t = timeit.Timer()
t.timeit() 0.054198980331420898 t.repeat(1) [0.053848981857299805] t.timeit() 0.061758041381835938 t.repeat(1) [0.053132057189941406] t.timeit() 0.055379867553710938 t.repeat(1) [0.074712038040161133] tt = t.repeat(1000)
tt.sort()
tt[0] 0.053070068359375 tt[-1] 0.056523799896240234

And now what?

I'd just stick with the commandline's "best of N" strategy. Also, I would
expect the "pass" statement to be the fastest to execute and therefore the
least accurate to measure.

Peter

[Peter Otten]

Dan Christensen wrote:

Hmm, if I type this at the python prompt:

r = []
for i in range(1): # Note: only one loop!
ti = t.timeit()
r.append(ti)
print r

I get data that agrees with t.repeat but disagrees with:
....
t.timeit()

You could try "intermediate simplifications" to spot the culprit:

r = [None]
for i in range(1):
r[i] = t.timeit()

and

for i in range(1):
ti = t.timeit()

I've no idea what's happening.

Peter