question regarding list comprehensions

[Pat]

I have written chunks of Python code that look this:

new_array = []
for a in array:
if not len( a ):
continue
new_array.append( a )

and...

string = ""
for r in results:
if not r.startswith( '#' ):
string =+ r

It seems that a list comprehension could clean up the code, but I seem
to have a mental block on list comprehensions. I've read up a lot on
this subject in my books and on the Internet and for whatever reason,
I'm having problems with this idiom (if that's the correct expression).

I've made a number of attempts to solve this on my own but I keep
getting errors.

Could someone please tell me how I could convert the above code to
something more elegant but readily understandable?

Finally, if someone could point me to a good tutorial or explain list
compressions I would be forever in your debt.

[Diez B. Roggisch]

Pat wrote:

I have written chunks of Python code that look this:

new_array = []
for a in array:
if not len( a ):
continue
new_array.append( a )

new_array = [a for a in array if len(a)]

and...

string = ""
for r in results:
if not r.startswith( '#' ):
string =+ r

"".join(r for r in results if not r.startswith("#"))

Diez

[Steven D'Aprano]

On Mon, 20 Oct 2008 10:20:03 -0400, Pat wrote:

Finally, if someone could point me to a good tutorial or explain list
compressions I would be forever in your debt.

Think of a for-loop:

for x in (1, 2, 3):
x

Creates x=1, then x=2, then x=3. It doesn't do anything with the x's, but
just creates them. Let's turn it into a list comp, and collect the x's:

>>[x for x in (1, 2, 3)]

[1, 2, 3]

for x in (1, 2, 3):
2*x+1

Creates x=1, then evaluates 2*x+1 = 3. Then it repeats for x=2, then x=3.
Here it is as a list comp:

>>[2*x+1 for x in (1, 2, 3)]

[3, 5, 7]

for x in (1, 2, 3):
if x != 2:
2*x+1

Here it is as a list comp:

>>[2*x+1 for x in (1, 2, 3) if x != 2]

[3, 7]
You can use any sort of sequence inside a list comp, not just a tuple.

>>[c.upper() for c in "abcd"]

['A', 'B', 'C', 'D']
You can nest list comps:

>>[y+1 for y in [2*x+1 for x in (1, 2, 3)]]

[4, 6, 8]
Advanced: you can use tuple-unpacking:

>>[(y,x) for (x,y) in [(1,2), (3, 4)]]

[(2, 1), (4, 3)]

and also multiple for-loops:

>>[(x,c) for x in (1, 2) for c in "abc"]

[(1, 'a'), (1, 'b'), (1, 'c'), (2, 'a'), (2, 'b'), (2, 'c')]

That last one is close to:

for x in (1, 2):
for c in "abc":
(x, c)
--
Steven

[Bruno Desthuilliers]

Pat a écrit :

I have written chunks of Python code that look this:

new_array = []
for a in array:
if not len( a ):
continue
new_array.append( a )

# à la lisp
new_array = filter(None, array)

# à la haskell
new_array = [a for a in array if a]

NB : all builtin empty builtin sequence type have a false boolen value,
so 'not len(a)' and 'not a' are usually practicaly equivalent.

<ot>
While we're at it : I assume you mean 'list', not 'array'. That's at
least how the builtin type is named...
</ot>

and...

string = ""
for r in results:
if not r.startswith( '#' ):
string =+ r

# à la lisp
string = "".join(filter(lambda s: not s.startwith('#'), results)

# à la haskell
string = "".join(r for r in results if not s.startswith('#'))

It seems that a list comprehension could clean up the code, but I seem
to have a mental block on list comprehensions. I've read up a lot on
this subject in my books and on the Internet and for whatever reason,
I'm having problems with this idiom (if that's the correct expression).

I've made a number of attempts to solve this on my own but I keep
getting errors.

Could someone please tell me how I could convert the above code to
something more elegant but readily understandable?

Basically, a list expression is written as (pseudo BNF):

'['<expressionfor item in <iterable_expression[if
<conditional_expression>] ']'
where
- <expressionis any Python expression
- <iterable_expressionis any Python expression that evals to an iterable
- the 'if <conditional_expression>' part is optional

The resulting list will be obtained by evaluating <expressionfor each
item of <iterable_expressionfor which <conditional_expressionis
verified.

Since the main goal of list expressions is to build, well, lists, one
usually uses an <expressionthat evals to something we possibly want to
be part of the new list !-)
Now for something more practical:

First let's rewrite your first snippet to get rid of the continue statement:

new_list = []
for item in source_list:
if len(item):
new_list.append(item)
And let's rewrite the second one to replace string concatenation with
the more idiomatic (and much more flexible) list / join idiom:

new_list = []
for item in results:
if not item.startswith('#'):
new_list.append(item)

string = "".join(new_list)
You may start to see a kind of pattern here:

0. new_list = []
1. for item in sequence:
2. if condition(item):
3. new_list.append(item)

The equivalent list comprehension is:

new_list = [item for item in sequence if condition(item)]
Since you didn't apply any transformation to 'item' in your for loops,
the <expressionpart is as simple as it can be : we want the item, period.

The <iterable_expressionis here again quite simple : it's your source
sequence. Ditto for the the <conditional_expression>.

IOW, your for-loop pattern is really:

new_list = []
for item in <iterable_expression>:
if <conditional_expression>:
new_list.append(<expression>)
Once you've identified the different parts, translating this for-loop to
a list comprehension is mostly a matter of reording.

Finally, if someone could point me to a good tutorial or explain list
compressions I would be forever in your debt.

http://en.wikipedia.org/wiki/List_comprehension
http://en.wikipedia.org/wiki/Set-builder_notation
HTH

[Pat]

Steven D'Aprano wrote:

On Mon, 20 Oct 2008 10:20:03 -0400, Pat wrote:

>Finally, if someone could point me to a good tutorial or explain list
compressions I would be forever in your debt.

Think of a for-loop:

for x in (1, 2, 3):
x

Creates x=1, then x=2, then x=3. It doesn't do anything with the x's, but
just creates them. Let's turn it into a list comp, and collect the x's:

>>>[x for x in (1, 2, 3)]

[1, 2, 3]

for x in (1, 2, 3):
2*x+1

Creates x=1, then evaluates 2*x+1 = 3. Then it repeats for x=2, then x=3.
Here it is as a list comp:

>>>[2*x+1 for x in (1, 2, 3)]

[3, 5, 7]

for x in (1, 2, 3):
if x != 2:
2*x+1

Here it is as a list comp:

>>>[2*x+1 for x in (1, 2, 3) if x != 2]

[3, 7]
You can use any sort of sequence inside a list comp, not just a tuple.

>>>[c.upper() for c in "abcd"]

['A', 'B', 'C', 'D']
You can nest list comps:

>>>[y+1 for y in [2*x+1 for x in (1, 2, 3)]]

[4, 6, 8]
Advanced: you can use tuple-unpacking:

>>>[(y,x) for (x,y) in [(1,2), (3, 4)]]

[(2, 1), (4, 3)]

and also multiple for-loops:

>>>[(x,c) for x in (1, 2) for c in "abc"]

[(1, 'a'), (1, 'b'), (1, 'c'), (2, 'a'), (2, 'b'), (2, 'c')]

That last one is close to:

for x in (1, 2):
for c in "abc":
(x, c)

Thank you. I think that clears up the mystery a bit. I added your note
to my snippets file. When I have a situation that I can't resolve, I'll
ask with the specifics.

I really do try to learn this as much as possible on my own without just
flinging a question onto the forum.

[Pat]

Diez B. Roggisch wrote:

Pat wrote:

>I have written chunks of Python code that look this:

new_array = []
for a in array:
if not len( a ):
continue
new_array.append( a )

new_array = [a for a in array if len(a)]

>and...

string = ""
for r in results:
if not r.startswith( '#' ):
string =+ r

"".join(r for r in results if not r.startswith("#"))

Diez

Thank you very much! That's exactly what I was looking for. That's
much cleaner than my code.

This is a great forum and I really appreciate everyone's help.