Learning to think differently...

[true911m]

Here's one to chuckle at later...

I was playing around with the random number generator, and wondering how random it is, so I decided to test it.

Made a list of ten zeros. Seeded random, created a loop, and incremented list[random.randint(0,9)] each time.

Results were a little rough at 1000 and 10000, so I decided to try some bigger numbers (the more you flip a coin, the closer your results should approach 50%...).

Let's try 10**8.

Here's the good part. In typical BASIC-esque mentality, and remembering one of my Python books saying that 'for' is a little quicker than 'while', I used this:

for i in range(10**8):
 

I waited a couple minutes, as I thought I'd need to, when Windows tells me it needs to expand my page file (hmm, I just installed that memory optimizer...), and the program failed with a MemoryError.

I was chasing my tail, watching the memory graph eat 250+ free MB each time I started the program, and eliminating sections of code to find the culprit. I finally tried 'while' with a counter, which ran and finished without a blip on the graph.

Of course, while you chuckle, I now realize it wasn't the 'for', but the list with 10**8 items in it that caused my problem. :)

Anyway, so this isn't completely wasted bandwidth, I think I recall reading about an alternate version of range() that delivers each item just-in-time, rather than altogether at the beginning, but I couldn't find it in the Python docs. Does anyone know what that function might be?

Thanks.

[bartonc]

Yeah, range() is notorious. The docs say:

2.3.6.3 XRange Type
The xrange type is an immutable sequence which is commonly used for looping. The advantage of the xrange type is that an xrange object will always take the same amount of memory, no matter the size of the range it represents. There are no consistent performance advantages.
XRange objects have very little behavior: they only support indexing, iteration, and the len() function.

[true911m]

xrange() ... that's it! I'll see how that stacks up.

I referred earlier to an assertion that 'for' is more efficient than 'while', and that makes sense perhaps for its intended purpose, which is iterating through a list.

But what are your thoughts on an application like this, where the actual need is a counter? It seems the glaring inefficiency in this case is not the for/while difference, but the creation of the list in the first place with 'for', rather than incrementing a counter as it is typically used in the BASICs.

Is there a more pythonic loop/count technique I might use besides either of these?

[bvdet]

To repeat a number of times:

for i in xrange(num):

For a counter for use as an index to a list, I like to use:

for i in range(len(lst)):

. To step through a list, I like:

for i in list:

Sometimes you might need a counter as well as step through a list:

for i, str1 in enumerate(lst):
    print i, str1

For what is 'pythonic' and what is not, I'll leave to the experts to decide!

[true911m]

I thought that using xrange might be a stubborn means of furthering the BASIC view of the problem, by sticking with the use of 'for' -- I thought that generating that list, even partially, would be a waste of (CPU) time.

However, xrange performed very well, with only slight overhead compared to the 'while' solution.

Averages for seven runs of 1 million iterations each:

for - range():
9.59 secs

for - xrange():
6.76 secs

while - i+=1
6.72 secs

Again, this is a situation where the counting is merely academic, and therefore range() must create the list for the sole purpose of performing the iteration. In situations where pre-existing lists of data are processed, 'for' should outperform 'while'.

Oh, and regarding that randint() test earlier, it seems quite random - variance was less than 0.005% from the "ideal" for a sample of 100 million generated numbers.

[true911m]

Oh, and thanks, bv.

Barton has responded to so many of my posts that I didn't realize you were someone else. :)

[bvdet]

Oh, and thanks, bv.

Barton has responded to so many of my posts that I didn't realize you were someone else. :)

You are welcome! This is an interesting thread. Keep posting.

[bartonc]

You are welcome! This is an interesting thread. Keep posting.

Our community is growing and maturing! Keep at it guys!