List Comprehension Syntax

[Moosebumps]

Does anyone here find the list comprehension syntax awkward?

I like it because it is an expression rather than a series of statements,
but it is a little harder to maintain it seems.

e.g. you could do:

result = []
for element in list:
if element[0:4] == 'blah':
result.append( element.replace ( 'blah', 'moof' ) )

or just:

result = [ element.replace ( 'blah', 'moof' ) for element in list if
element[0:4] == 'blah' ]

The second looks cleaner in some cases, but it is less maintainable. It
tends to promote long lines. It all seems to run together and such. And
often you would need to add another condition or modify it, it seems better
to use the first way even though it has the ugly extra init (and which would
cause more allocs than the list comprehension? because it wouldn't know the
size off the bat?)

I am in favor of short lines like I think Guido said in the style guide. I
like each line to be so simple as to not require any thinking reading it,
e.g.

I prefer:

x = c( d, e )
y = f( g, h )
z = b( x, y )
w = a( z )

to stuff like this:

w = a( b( c( d, e ), f( g, h ) ) )

It is more maintainable, when you need to make a change, just insert a line,
rather than having to decode an expression.

Along the same lines, it seems more maintainable to split things up.

You could do:

result = [
element.replace ( 'blah', 'moof' )
for element in list
if element[0:4] == 'blah' ]

I guess, but that seems awkward to me. Looks too much like a for loop and
an if, and then the value is at the top, which reads funny to me.
(Strangely putting it on one line doesn't read as funny, but it is less
readable.) Maybe I just have to get used to it. Which do you prefer?
Comments?

MB

[Ville Vainio]

>>>>> "Moosebumps " == Moosebumps <cr**@crap.crap > writes:

Moosebumps> Does anyone here find the list comprehension syntax awkward?

....

Moosebumps> I am in favor of short lines like I think Guido said
Moosebumps> in the style guide. I like each line to be so simple
Moosebumps> as to not require any thinking reading it,

You need to think of the total complexity involved with having several
lines. When you see a list comprehension, you know what to expect -
transformation and/or filtering applied to a list. Therefore, you can
easily read and write out the beast.

LC's also encourage the list transformation/filtering approach to
problems, which I find absolutely splendid. Appending elements to a
list manually is tedious, especially if the problem really is a
stereotypical problem solved by a LC (and interestingly, most problems
are ;-).

Moosebumps> You could do:

Moosebumps> result = [
Moosebumps> element.replace ( 'blah', 'moof' )
Moosebumps> for element in list
Moosebumps> if element[0:4] == 'blah' ]

Moosebumps> I guess, but that seems awkward to me. Looks too much
Moosebumps> like a for loop and an if, and then the value is at
Moosebumps> the top, which reads funny to me. (Strangely putting
Moosebumps> it on one line doesn't read as funny, but it is less
Moosebumps> readable.) Maybe I just have to get used to it.
Moosebumps> Which do you prefer?

It's just a matter of getting used to it. Admittedly LC's are
sometimes confusing for newbies, but they are an example of such a
feature where the tradeoff between newbie and non-newbie friendliness
has really paid off.

Now that genexps are coming around, you'll be facing even bigger
payoffs. So just keep using them, even if they might not feel as
maintanable at the moment. LC's (and genexps even to a bigger extent)
are pretty much what defines the "pythonic" way of doing things for me
these days.

--
Ville Vainio http://tinyurl.com/2prnb

[Eric S. Johansson]

Ville Vainio wrote:

>>"Moosebum ps" == Moosebumps <cr**@crap.crap > writes:

Moosebumps> Does anyone here find the list comprehension syntax awkward?

...

Moosebumps> I am in favor of short lines like I think Guido said
Moosebumps> in the style guide. I like each line to be so simple
Moosebumps> as to not require any thinking reading it,

You need to think of the total complexity involved with having several
lines. When you see a list comprehension, you know what to expect -
transformation and/or filtering applied to a list. Therefore, you can
easily read and write out the beast.

LC's also encourage the list transformation/filtering approach to
problems, which I find absolutely splendid. Appending elements to a
list manually is tedious, especially if the problem really is a
stereotypical problem solved by a LC (and interestingly, most problems
are ;-).

Moosebumps> You could do:

Moosebumps> result = [
Moosebumps> element.replace ( 'blah', 'moof' )
Moosebumps> for element in list
Moosebumps> if element[0:4] == 'blah' ]

Moosebumps> I guess, but that seems awkward to me. Looks too much
Moosebumps> like a for loop and an if, and then the value is at
Moosebumps> the top, which reads funny to me. (Strangely putting
Moosebumps> it on one line doesn't read as funny, but it is less
Moosebumps> readable.) Maybe I just have to get used to it.
Moosebumps> Which do you prefer?

It's just a matter of getting used to it. Admittedly LC's are
sometimes confusing for newbies, but they are an example of such a
feature where the tradeoff between newbie and non-newbie friendliness
has really paid off.

as someone approaching 25 years experience with programming languages
and their implications, I find list comprehension's incomprehensibl e.
Personally I think my learning is inhibited by the documentation which
seems driven by the syntax of list comprehensions rather than the
execution model and how to recognize when it's appropriate to apply that
model.

I find multiple short lines actually easier to comprehend because it
translates into a common mental model of mine. The current syntax for
those comprehension's reminds me of APL which was a wonderful language
if you thought in nothing but vectors.

I will admit this discussion has goaded me into trying to learn list
comprehensions again as I am trying to solve a problem which is
filtering a dictionary of data elements based on predicates selecting
individual elements and/or predicates on the offset from the start of
the previous predicate list.

By the way, similarly hampered-by-the-documentation are generators,
iterators, and profiling time bases. I'm having a lot of difficulty
pulling out model abstractions, and as I said above, understanding where
to apply them in problem spaces.

Now that genexps are coming around, you'll be facing even bigger
payoffs. So just keep using them, even if they might not feel as
maintanable at the moment. LC's (and genexps even to a bigger extent)
are pretty much what defines the "pythonic" way of doing things for me
these days.

this is not meant to be picking on you in any way shape or form but my
experience has been that any time you find yourself having to "thinking
in the language", you are not really solving the right problem and are
more likely using a collection of magic tricks to confound and amaze
others and possibly insure job security.

if you have models that can be implemented independent of the language
and you can express a problem in terms that are natural to the problem,
you invariably have a better solution for the people following you as
well as the machine.

generalize, don't pythonize.

---eric

[Ville Vainio]

>>>>> "Eric" == Eric S Johansson <es*@harvee.org > writes:

Eric> as someone approaching 25 years experience with programming
Eric> languages and their implications, I find list
Eric> comprehension's incomprehensibl e. Personally I think my
Eric> learning is inhibited by the

I found LC's a bit odd too at first. Previous programming experience
probably doesn't matter too much with them, because they are quite
different.

Eric> documentation which seems driven by the syntax of list
Eric> comprehensions rather than the execution model and how to
Eric> recognize when it's appropriate to apply that model.

Probably. It's all too easy to dismiss if the documentation doesn't
sell it well (this is fixable, luckily). I kinda ignored LC's too, but
persistent ramblings on c.l.py (by Alex Martelli and others) changed
that, for which I'm grateful and feel honor-bound to continue the
pseudo-oral tradition :-).

The main thing to realize about list comprehensions is that
they simply provide a more elegant way to do 'map' and 'filter' when a
function to be applied is not something trivial like str or
int. Having LC's handy urges one to go ahead with map/filter like
approaches to problems where implementing new functions (or calling
old ones via lambda) seems like an unnecessary hassle.

Eric> I find multiple short lines actually easier to comprehend
Eric> because it translates into a common mental model of mine.

It helps if your mental model involves manipulating lots of
lists. I've found that the list manipulation model works great for me,
allowing me to solve most problems quickly and (I think) elegantly. I
guess it depends a lot on what you are doing - my python use is mostly
just scripting these days (for reasons not in my control, of course
;-).

Eric> I will admit this discussion has goaded me into trying to
Eric> learn list comprehensions again as I am trying to solve a
Eric> problem which is filtering a dictionary of data elements
Eric> based on predicates selecting individual elements and/or
Eric> predicates on the offset from the start of the previous
Eric> predicate list.

Good for you. Do it with map and filter (and in-scope funcs using
closures) and go LC only afterwards if that feels easier. I believe
people still feel more comfortable with LCs than nested scopes,
because the LC is "visually" more in the same scope. The part after
"and/or" seemed too mysterious to give the solution now, but it seems
you'll need to implement a function in addition to the LC to keep the
solution clean.

Eric> By the way, similarly hampered-by-the-documentation are
Eric> generators, iterators, and profiling time bases. I'm having
Eric> a lot of difficulty pulling out model abstractions, and as I
Eric> said above, understanding where to apply them in problem
Eric> spaces.

Do you feel it's the offical docs that are lacking, or have you tried
reading some Python books? I've probably been in the "enthusiast "
crowd that gets a kick from reading those "what's new" docs, PEPs and
such?

Eric> experience has been that any time you find yourself having
Eric> to "thinking in the language", you are not really solving
Eric> the right problem and are more likely using a collection of
Eric> magic tricks to confound and amaze others and possibly
Eric> insure job security.

Eric> if you have models that can be implemented independent of
Eric> the language and you can express a problem in terms that are
Eric> natural to the problem, you invariably have a better
Eric> solution for the people following you as well as the
Eric> machine.

Luckily, the underlying model of LCs is very well understood (map &
filter), and solving things the list-processing way is a time-honed
practice. In the "amaze your friends" front, LCs are more in the "look
how elegant and concise this can be" genre, not in the "check this
out, all recursion and no variables, I bet you can't get it even after
staring it for 5 minutes" genre loved by some academics.

I think you'll find that you don't need to sacrifice any of your old
models or even aesthetic preferences to appreciate LCs.

--
Ville Vainio http://tinyurl.com/2prnb

[Dave Brueck]

Eric S. Johansson wrote:

Ville Vainio wrote:

It's just a matter of getting used to it. Admittedly LC's are
sometimes confusing for newbies, but they are an example of such a
feature where the tradeoff between newbie and non-newbie friendliness
has really paid off.
as someone approaching 25 years experience with programming languages
and their implications, I find list comprehension's incomprehensibl e.
Personally I think my learning is inhibited by the documentation which
seems driven by the syntax of list comprehensions rather than the
execution model and how to recognize when it's appropriate to apply that
model.

What I like about LCs is that they made sense to me before reading the
documentation - they just came across as very expressive of what was
happening. I use the simplest forms, i.e.

[op(x) for x in y]
[x for x in y if z]
(as well as some slight variations)

quite a bit, but rarely do I use the more complex forms (e.g. multiple
for's) because (1) the meaning doesn't jump right out as quickly (to me)
and (2) the more complex they are, the more they seem like a fancy trick
rather than the right thing to do.

But most any time you're using map or filter or basically doing:

L = []
for item in L2:
L.append(op(ite m))

the LC form is often more desirable because it screams "I'm taking a
sequence and using it to create a list" - the intent of the code is very
clear. And by extension, if you're using an LC to do something obtuse,
you deserve a slap on the wrist (e.g. you write [f() for f in x] and
throw away the resulting list).

Now that genexps are coming around, you'll be facing even bigger
payoffs. So just keep using them, even if they might not feel as
maintanable at the moment. LC's (and genexps even to a bigger extent)
are pretty much what defines the "pythonic" way of doing things for me
these days.

this is not meant to be picking on you in any way shape or form but my
experience has been that any time you find yourself having to "thinking
in the language", you are not really solving the right problem and are
more likely using a collection of magic tricks to confound and amaze
others and possibly insure job security.

That's not what I understood by Ville's comment. I think he just meant
that LC's (and genexps) are powerful tools in the Python toolbox, and
useful enough that the OP should continue working to become familiar
with them. They're not obscure magic tricks but "first class" features
of the language.

I've seen many comments on c.l.py to the effect of "LCs seem bad because
they can abused", citing bizarre made-up examples with 4 loops and as
many if statements. Those *are* magic tricks and should be avoided, but
then again any feature can be abused.
if you have models that can be implemented independent of the language
and you can express a problem in terms that are natural to the problem,
you invariably have a better solution for the people following you as
well as the machine.

generalize, don't pythonize.

I'm not so sure. Why program to the lowest common language denominator?
I don't suggest going to the extreme to use obscure language quirks just
because you can, but it doesn't make sense to avoid using a feature at
your disposal because it's unique to a language (or, in this case, a
small set of languages).

Half the reason you use one language over another is because of the
toolset it gives you. In the case of list comprehensions, they are
usually chosen for the very reason that they *do* allow you to express a
problem in natural terms.

-Dave

[Eric S. Johansson]

Ville Vainio wrote:

Eric> documentation which seems driven by the syntax of list
Eric> comprehensions rather than the execution model and how to
Eric> recognize when it's appropriate to apply that model.

Probably. It's all too easy to dismiss if the documentation doesn't
sell it well (this is fixable, luckily). I kinda ignored LC's too, but
persistent ramblings on c.l.py (by Alex Martelli and others) changed
that, for which I'm grateful and feel honor-bound to continue the
pseudo-oral tradition :-).
I understand, having been the creator of more than a couple oral
traditions myself.
The main thing to realize about list comprehensions is that
they simply provide a more elegant way to do 'map' and 'filter' when a
function to be applied is not something trivial like str or
int. Having LC's handy urges one to go ahead with map/filter like
approaches to problems where implementing new functions (or calling
old ones via lambda) seems like an unnecessary hassle.
maybe that's the problem. I've never seen anyplace where it's easier to
use map and filter.
It helps if your mental model involves manipulating lots of
lists. I've found that the list manipulation model works great for me,
allowing me to solve most problems quickly and (I think) elegantly. I
guess it depends a lot on what you are doing - my python use is mostly
just scripting these days (for reasons not in my control, of course
;-).
well, I do manipulate a few lists but more often, I manipulate
dictionaries. I tend to think more in terms of sets, bags and queues.

if you want to take on a bigger (open source) project, I have one you
can work on. ;-)
Good for you. Do it with map and filter (and in-scope funcs using
closures) and go LC only afterwards if that feels easier. I believe
people still feel more comfortable with LCs than nested scopes,
because the LC is "visually" more in the same scope. The part after
"and/or" seemed too mysterious to give the solution now, but it seems
you'll need to implement a function in addition to the LC to keep the
solution clean.
I will show you what I create the easiest which I can guarantee you will
be a iterative solution. then maybe we can work through the process of
converting it to a list comprehension.
Eric> By the way, similarly hampered-by-the-documentation are
Eric> generators, iterators, and profiling time bases. I'm having
Eric> a lot of difficulty pulling out model abstractions, and as I
Eric> said above, understanding where to apply them in problem
Eric> spaces.

Do you feel it's the offical docs that are lacking, or have you tried
reading some Python books? I've probably been in the "enthusiast "
crowd that gets a kick from reading those "what's new" docs, PEPs and
such?
I've tried reading a few sources. and I'm not trying to throw bricks
because I do know how hard it is to write user understandable
documentation especially if you have been totally immersed in a project
for a while.
Luckily, the underlying model of LCs is very well understood (map &
filter), and solving things the list-processing way is a time-honed
practice. In the "amaze your friends" front, LCs are more in the "look
how elegant and concise this can be" genre, not in the "check this
out, all recursion and no variables, I bet you can't get it even after
staring it for 5 minutes" genre loved by some academics.

but even good tools can be used for the purposes of evil... ;-)

---eric

[Eric S. Johansson]

Dave Brueck wrote:

What I like about LCs is that they made sense to me before reading the
documentation - they just came across as very expressive of what was
happening. I use the simplest forms, i.e. .... throw away the resulting list).
good advice. Thank you for making this clear.
I'm not so sure. Why program to the lowest common language denominator?
I don't suggest going to the extreme to use obscure language quirks just
because you can, but it doesn't make sense to avoid using a feature at
your disposal because it's unique to a language (or, in this case, a
small set of languages).

Half the reason you use one language over another is because of the
toolset it gives you. In the case of list comprehensions, they are
usually chosen for the very reason that they *do* allow you to express a
problem in natural terms.

it all depends on your definition of natural terms. ;-)

For me, the abstractions I use tend to be higher level than what most
languages support and is always a loss of clarity in the translation to
implementation. Python minimizes the translation distance for me.

Sometimes I find myself avoiding language quirks and features because
I'm trying to get a job done and I don't want to go through the mental
puzzle of mapping the abstract form into the implementation form using a
particular feature. So for example, I use for loops in preference to
list comprehensions just because it's faster to implement and get the
job done.

Most customers don't pay you for pretty code, they pay you to accomplish
something quickly and make it understandable to the less skilled people
following you. which probably explains why so much software is crap but
that's a whole different discussion.

your observation about reasons for choosing languages are certainly
accurate for most people but for me, it's quite different. If I can't
write code using speech recognition without doing a job on my throat, I
won't use the language. List comprehension syntax is approaching dammed
ugly for speech recognition users.

but thank you again for your clear example of how you use list
comprehensions.

---eric

[Ville Vainio]

>>>>> "Eric" == Eric S Johansson <es*@harvee.org > writes:

It helps if your mental model involves manipulating lots of
lists. I've found that the list manipulation model works great
for me,

Eric> well, I do manipulate a few lists but more often, I
Eric> manipulate dictionaries. I tend to think more in terms of
Eric> sets, bags and queues.

These work as sequences (by lists, I meant sequences) as
well. Especially as we get genexps, LCish operations can be performed
for free without space penalty.

Eric> if you want to take on a bigger (open source) project, I
Eric> have one you can work on. ;-)

No doubt ;-).

Eric> I will show you what I create the easiest which I can guarantee you
Eric> will be a iterative solution. then maybe we can work through the
Eric> process of converting it to a list comprehension.

Sounds like a plan.

--
Ville Vainio http://tinyurl.com/2prnb

[Ville Vainio]

>>>>> "Eric" == Eric S Johansson <es*@harvee.org > writes:

Eric> it all depends on your definition of natural terms. ;-)

Eric> For me, the abstractions I use tend to be higher level than
Eric> what most languages support and is always a loss of clarity
Eric> in the translation to implementation. Python minimizes the
Eric> translation distance for me.

That's good news, because LCs are higher level than explicit loops.

Eric> a particular feature. So for example, I use for loops in
Eric> preference to list comprehensions just because it's faster
Eric> to implement and get the job done.

That's mostly a result of you not a comfortable grasp of LCs yet. As
with most things in Python, learning them is going to be worth the
time used for learning.
Eric> your observation about reasons for choosing languages are
Eric> certainly accurate for most people but for me, it's quite
Eric> different. If I can't write code using speech recognition
Eric> without doing a job on my throat, I won't use the language.
Eric> List comprehension syntax is approaching dammed ugly for
Eric> speech recognition users.

How? The only special thing about LCs are the surrounding ['s. It's
also a lot less speaking, which is probably a good thing.

Admittedly I have never used speech recognition software. How do you
speak out the following equivalent snippets:

----

files = [f.lower() for f in allfiles if f.endswith(".tx t")]

----
files = []
for f in allfiles:
if f.endswith(".tx t"):
files.append(f. lower())

----
If it has something to do with line breaking, the following is
obviously ok too (and in no way inferior to the one-line approach):

files = [f.lower()
for f in allfiles
if f.endswith(".tx t")]

--
Ville Vainio http://tinyurl.com/2prnb

[Eric S. Johansson]

Ville Vainio wrote:

Admittedly I have never used speech recognition software. How do you
speak out the following equivalent snippets:

----

files = [f.lower() for f in allfiles if f.endswith(".tx t")]
files equal sign between brackets foxtrot dot lower matched parens for
foxtrot in all no space files if foxtrot dot ends no space with matched
parens between quotes dot tango x-ray tango

is a reasonably close approximation not counting misrecognitions .
personally, I would never ever use single character variables unless I'm
typing them and I would never use merged words. I would always join
them with _ because it's easier and more accurate than saying no-space.
and mixed case is right out because you have to state every time you
shift case unless NaturallySpeaki ng just happens to know the word in the
right case.

and the matched brackets etc. macros are of my own creation
----
files = []
files equal sign matched brackets
for f in allfiles:
for foxtrot in all no space files :
if f.endswith(".tx t"):
if foxtrot dot ends no space with between parens between quotes dot
tango x-ray tango end colon
files.append(f. lower())
files dot append between parens foxtrot dot lower matched parens
and it's not fun. take a look at camram if you want to see a fair
amount of Python written probably 80 or 90 percent by voice. Its
somewhere over 5000 lines of code if I'm not counting improperly.

If it has something to do with line breaking, the following is
obviously ok too (and in no way inferior to the one-line approach):

it's easier with speech recognition to say small things and correct
especially when coding. unfortunately, NaturallySpeaki ng does not work
extremely well with applications that do not use a very limited set of
edit controls.

---eric