question on list comprehensions

[Darren Dale]

Hi,

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

In practice, res is a complex 2D numarray. For this reason, the regular
output of a list comprehension will not work: constructing a list of every
intermediate result will result in huge hits in speed and memory.

I saw this article at ASPN:
http://aspn.activestate.com/ASPN/Coo.../Recipe/204297

def thislist():
"""Return a reference to the list object being constructed by the
list comprehension from which this function is called. Raises an
exception if called from anywhere else.
"""
import sys
d = sys._getframe(1 ).f_locals
nestlevel = 1
while '_[%d]' % nestlevel in d:
nestlevel += 1
return d['_[%d]' % (nestlevel - 1)].__self__

Could the list comprehension include something like thislist().pop( 0), to be
called when len(thislist)>1 ? (I think this could work, but am having
trouble with the syntax.) Or is there a better way to approach the problem?

Thank you,

Darren

[Fredrik Lundh]

Darren Dale wrote:

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

why?

</F>

[Roberto Antonio Ferreira De Almeida]

Darren Dale wrote:

Hi,

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i
res[0] = (10000 * (10000-1))/2.0 ;-)
In practice, res is a complex 2D numarray. For this reason, the regular
output of a list comprehension will not work: constructing a list of every
intermediate result will result in huge hits in speed and memory.

Why do you *need* to replace the for loop with a listcomp? Could you
give more details about what you're doing with the complex array?

Roberto

[Darren Dale]

Fredrik Lundh wrote:

Darren Dale wrote:

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

why?

</F>

I explained why in the original post...

[Darren Dale]

Darren Dale wrote:

Fredrik Lundh wrote:

Darren Dale wrote:

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

why?

</F>

I explained why in the original post...

or I guess I just implied it. The speedup is critical.

[Darren Dale]

Roberto Antonio Ferreira De Almeida wrote:

Darren Dale wrote:

Hi,

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

res[0] = (10000 * (10000-1))/2.0 ;-)

In practice, res is a complex 2D numarray. For this reason, the regular
output of a list comprehension will not work: constructing a list of
every intermediate result will result in huge hits in speed and memory.

Why do you *need* to replace the for loop with a listcomp? Could you
give more details about what you're doing with the complex array?

Roberto

OK. As usual, I am having trouble clearly expressing myself. Sorry about
that. Prepare for some physics:

I am simulating diffraction from an array of particles. I have to calculate
a complex array for each particle, add up all these arrays, and square the
magnitude of the result. If I do a for loop, it takes about 6-8 seconds for
a 2000 element array added up over 250 particles. In reality, I will have
2500 particles, or even 2500x2500 particles.

The list comprehension takes only 1.5 seconds for 250 particles. Already,
that means the time has decreased from 40 hours to 10, and that time can be
reduced further if python is not constantly requesting additionaly memory
to grow the resulting list.

I guess that means that I would like to avoid growing the list and popping
the previous result if possible, and just over-write the previous result.

[Carlos Ribeiro]

On Thu, 14 Oct 2004 11:16:12 -0400, Darren Dale <dd**@cornell.e du> wrote:

I am simulating diffraction from an array of particles. I have to calculate
a complex array for each particle, add up all these arrays, and square the
magnitude of the result. If I do a for loop, it takes about 6-8 seconds for
a 2000 element array added up over 250 particles. In reality, I will have
2500 particles, or even 2500x2500 particles.

Can't you just sum them inplace as you calculate every new array, for
each particle? Something like this (pseudo code):

result = make_empty_arra y()
for particle in particles:
result += make_complex_ar ray(particle)
return square(result)

It does not grow the result array indefinitely... . so it solves your
problem, if that's what you need/want to avoid.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: ca********@gmai l.com
mail: ca********@yaho o.com

[Darren Dale]

Carlos Ribeiro wrote:

On Thu, 14 Oct 2004 11:16:12 -0400, Darren Dale <dd**@cornell.e du> wrote:

I am simulating diffraction from an array of particles. I have to
calculate a complex array for each particle, add up all these arrays, and
square the magnitude of the result. If I do a for loop, it takes about
6-8 seconds for a 2000 element array added up over 250 particles. In
reality, I will have 2500 particles, or even 2500x2500 particles.

Can't you just sum them inplace as you calculate every new array, for
each particle? Something like this (pseudo code):

result = make_empty_arra y()
for particle in particles:
result += make_complex_ar ray(particle)
return square(result)

It does not grow the result array indefinitely... . so it solves your
problem, if that's what you need/want to avoid.

You're example is what I have now, it's what I want to replace.

I am trying to make the operations general, so I can do 1x1, 1xN, and NxN
result arrays. In the case of NxN, the overhead in the for loop can be
small compared to the time required for the array operations. But for a 1x1
result array, the for loop is a huge time sink. I thought it would be nice
to get the speedup of the list comprehension, but see no way to change the
array in place. The list comprehension wants to create a new list instead.

At any rate, I found another way to frame my problem to get the speedup.
It's entirely based on linear algebra operations, no ingenious coding
involved, so I wont share the details. They are pretty mundane.

[Josiah Carlson]

> OK. As usual, I am having trouble clearly expressing myself. Sorry about

that. Prepare for some physics:

I am simulating diffraction from an array of particles. I have to calculate
a complex array for each particle, add up all these arrays, and square the
magnitude of the result. If I do a for loop, it takes about 6-8 seconds for
a 2000 element array added up over 250 particles. In reality, I will have
2500 particles, or even 2500x2500 particles.

The list comprehension takes only 1.5 seconds for 250 particles. Already,
that means the time has decreased from 40 hours to 10, and that time can be
reduced further if python is not constantly requesting additionaly memory
to grow the resulting list.

I guess that means that I would like to avoid growing the list and popping
the previous result if possible, and just over-write the previous result.

Download scientific python: http://www.scipy.org/
And get your linear algebra on. Using Numeric arrays, the size of the
matrices (because they are matrices) are much smaller than the size of
an equivalent Python list of lists, so memory may stop being a concern.

- Josiah

[Mustafa Demirhan]

Why not just use while loops instead of for loops? You dont have to
create a new array each time you want a loop - you can simply use an
index integer.

i = 0
while i < 5000000:
res [0] = res [0] + i
i = i + 1

Takes less than 2 seconds on my laptop.
Darren Dale <dd**@cornell.e du> wrote in message news:<ck******* ***@news01.cit. cornell.edu>...

Hi,

I need to replace the following loop with a list comprehension:

res=[0]
for i in arange(10000):
res[0]=res[0]+i

In practice, res is a complex 2D numarray. For this reason, the regular
output of a list comprehension will not work: constructing a list of every
intermediate result will result in huge hits in speed and memory.

I saw this article at ASPN:
http://aspn.activestate.com/ASPN/Coo.../Recipe/204297

def thislist():
"""Return a reference to the list object being constructed by the
list comprehension from which this function is called. Raises an
exception if called from anywhere else.
"""
import sys
d = sys._getframe(1 ).f_locals
nestlevel = 1
while '_[%d]' % nestlevel in d:
nestlevel += 1
return d['_[%d]' % (nestlevel - 1)].__self__

Could the list comprehension include something like thislist().pop( 0), to be
called when len(thislist)>1 ? (I think this could work, but am having
trouble with the syntax.) Or is there a better way to approach the problem?

Thank you,

Darren