By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
445,743 Members | 1,072 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 445,743 IT Pros & Developers. It's quick & easy.

What's do list comprehensions do that generator expressions don't?

P: n/a
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?

Thanks,
<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #1
Share this Question
Share on Google+
23 Replies


P: n/a
Mike Meyer wrote:
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?


Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #2

P: n/a
Robert Kern wrote:
Mike Meyer wrote:
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?


Never. If you really need a list

list(x*x for x in xrange(10))


Not quite true. If you discovered the unlikely scenario that the
construction of a list from the generator expression was an efficiency
bottleneck, you might choose a list comprehension -- they're slightly
faster when you really do want a list:

$ python -m timeit "list(x*x for x in xrange(10))"
100000 loops, best of 3: 6.54 usec per loop

$ python -m timeit "[x*x for x in xrange(10)]"
100000 loops, best of 3: 5.08 usec per loop

STeVe
Jul 19 '05 #3

P: n/a
Steven Bethard wrote:
Robert Kern wrote:
Mike Meyer wrote:
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?


Never. If you really need a list

list(x*x for x in xrange(10))


Not quite true. If you discovered the unlikely scenario that the
construction of a list from the generator expression was an efficiency
bottleneck, you might choose a list comprehension -- they're slightly
faster when you really do want a list:

$ python -m timeit "list(x*x for x in xrange(10))"
100000 loops, best of 3: 6.54 usec per loop

$ python -m timeit "[x*x for x in xrange(10)]"
100000 loops, best of 3: 5.08 usec per loop


Okay, okay, *almost* never.

However, I don't expect that speed relationship to hold past Python 2.4.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #4

P: n/a
jfj
Robert Kern wrote:
Mike Meyer wrote:
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?

If you want a list right away you'd use a list comprehension.
X =[i for i in something() if somethingelse()]
random.shuffle(X)
print x[23]

On the other hand it's generator expressions which should be used
only when the code can be written in as a pipe. For example a filter
of a -otherwise- very long list:

make_fractal_with_seed (x for x in range(100000000) if
fibonacci_prime (x))



Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.


Why???
Then we should also remove:
x=[] to x=list()
x=[1,2,3] to x=list(1,2,3)

I think "list" is useful only:
1) to subclass it
2) to convert a list/tuple/string to a list, which is
done extremely fast.

But for iterators I find the list comprehension syntax nicer.
jfj

Jul 19 '05 #5

P: n/a
jfj wrote:
Robert Kern wrote:
Mike Meyer wrote:
Ok, we've added list comprehensions to the language, and seen that
they were good. We've added generator expressions to the language, and
seen that they were good as well.

I'm left a bit confused, though - when would I use a list comp instead
of a generator expression if I'm going to require 2.4 anyway?

If you want a list right away you'd use a list comprehension.
X =[i for i in something() if somethingelse()]
random.shuffle(X)
print x[23]

On the other hand it's generator expressions which should be used
only when the code can be written in as a pipe. For example a filter
of a -otherwise- very long list:

make_fractal_with_seed (x for x in range(100000000) if fibonacci_prime
(x))
Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.


Why???
Then we should also remove:
x=[] to x=list()
x=[1,2,3] to x=list(1,2,3)


Well, that last one doesn't work. Removing the empty list literal would
be inconsistent.
I think "list" is useful only:
1) to subclass it
2) to convert a list/tuple/string to a list, which is
done extremely fast.
Add "any iteratable". Genexps are iterables.
But for iterators I find the list comprehension syntax nicer.


Possibly. I find them too similar with little enough to choose between
them, hence the OP's question.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #6

P: n/a
jfj
I jfj wrote:

make_fractal_with_seed (x for x in range(100000000) if fibonacci_prime
(x))


and this is stupendus.
At least range should be xrange.
jfj

Jul 19 '05 #7

P: n/a
jfj
Robert Kern wrote:
jfj wrote:
2) to convert a list/tuple/string to a list, which is
done extremely fast.

Add "any iteratable". Genexps are iterables.


The thing is that when you want to convert a tuple to a list
you know already the size of it and you can avoid using append()
and expanding the list gradually. For iterables you can't avoid
appending items until StopIteration so using list() doesn't have
any advantage. The OP was about genexps vs list comprehensions
but this is about list() vs. list comprehensions.
Possibly. I find them too similar with little enough to choose between them, hence the OP's question.


One solution is to forget about list(). If you want a list use [].
Unless you want to convert a tuple...

I think a better question would be "What do *generator expressions* do
that list comprehensions don't?". And always use list comprehensions
unless you want the extra bit.
jfj

Jul 19 '05 #8

P: n/a
jfj wrote:
Robert Kern wrote:
jfj wrote:
2) to convert a list/tuple/string to a list, which is
done extremely fast.
Add "any iteratable". Genexps are iterables.


The thing is that when you want to convert a tuple to a list
you know already the size of it and you can avoid using append()
and expanding the list gradually. For iterables you can't avoid
appending items until StopIteration so using list() doesn't have
any advantage.


But no real disadvantage either.
The OP was about genexps vs list comprehensions
but this is about list() vs. list comprehensions.
Yes, and list(genexp) replaces list comprehensions rather handily.
There's very little reason to prefer list comprehensions over the
list(genexp) construction. If anything, it's pretty much a wash. That
was the OP's question:

[mike] when would I use a list comp instead of a generator expression if
I'm going to require 2.4 anyway?

Possibly. I find them too similar with little enough to choose between
them, hence the OP's question.


One solution is to forget about list(). If you want a list use [].
Unless you want to convert a tuple...

I think a better question would be "What do *generator expressions* do
that list comprehensions don't?". And always use list comprehensions
unless you want the extra bit.


<shrug> What we have now are two very similar constructs, one of which
is more general and subsumes nearly all the uses of the other. The
presence of both is confusing, as evidenced by the fact that the OP
asked his question. In the Great Breaking of Backwards Compatibility,
list comprehensions should go away.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #9

P: n/a
jfj <jf*@freemail.gr> writes:
I think a better question would be "What do *generator expressions* do
that list comprehensions don't?". And always use list comprehensions
unless you want the extra bit.


As the OP, I can say why I didn't ask those questions.

Generator expressions don't build the entire list in memory before you
have to deal with it. This makes it possible to deal with expressions
that are to long to fit in memory.

Which means that the real rule should be always use generator
expressions, unless you *know* the expression will always fit in
memory.

Which leads to the obvious question of why the exception.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #10

P: n/a
jfj
Mike Meyer wrote:
jfj <jf*@freemail.gr> writes:

I think a better question would be "What do *generator expressions* do
that list comprehensions don't?". And always use list comprehensions
unless you want the extra bit.

As the OP, I can say why I didn't ask those questions.


Sorry. I was referring to the subject line:)

Generator expressions don't build the entire list in memory before you
have to deal with it. This makes it possible to deal with expressions
that are to long to fit in memory.

Which means that the real rule should be always use generator
expressions, unless you *know* the expression will always fit in
memory.


Consider this code which I also included the first reply:

x = [i for in in something()]
random.shuffle (x)
x.sort ()

Shuffle and sort are two examples where need *the entire list* to
work. Similarily for a dictionary where the values are small lists.
In this example using a generator buys you *nothing* because you
will immediately build a list.

So there are cases where we need the list as the product of an algorithm
and a generator is not good enough. In fact, in my experience with
python so far I'd say that those cases are the most common case.

That is the one question.
The other question is "why not list(generator) instead of[list
comprehension]?"

I guess that lists are *so important* that having a primary language
feature for building them is worth it. On the other hand "list()" is
not a primary operator of the python language. It is merely a builtin
function.
jfj

Jul 19 '05 #11

P: n/a
On Sun, 24 Apr 2005 22:59:12 -0700, Robert Kern wrote:
Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.


Why "remove" them? Instead, we have these things called "comprehensions"
(which, now that I say that, seems a rather odd name), and you can control
whether they result in a list or a generator with () or [].

I don't see why they need to be "removed". Lists are already a special
case of the "only one way to do it" principle ([] vs. list()), and
pragmatically I don't see any reason to remove them here; it doesn't add
comprehensibility, leaving them in doesn't significantly affect the mental
size of the code (the *comprehension* is the hard part, the final form
should be relatively simple), it's not worth breaking that code.
Jul 19 '05 #12

P: n/a
Jeremy Bowers wrote:
On Sun, 24 Apr 2005 22:59:12 -0700, Robert Kern wrote:
Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.

Why "remove" them? Instead, we have these things called "comprehensions"
(which, now that I say that, seems a rather odd name), and you can control
whether they result in a list or a generator with () or [].


They are *not* the same thing. They have completely different
implementations although they have very similar syntax and the former
subsumes just about *every* use of the latter. That's pointless except
for maintaining backwards compatibility. The difference in
implementation is one reason why generator expressions were not called
generator comprehensions.

What's more, the list comprehension implementation is warty. It leaks
variables.
I don't see why they need to be "removed". Lists are already a special
case of the "only one way to do it" principle ([] vs. list()), and
pragmatically I don't see any reason to remove them here; it doesn't add
comprehensibility,
Yes it does. The OP was confused as to which to use now that both exist.
This has come up here before.
leaving them in doesn't significantly affect the mental
size of the code (the *comprehension* is the hard part, the final form
should be relatively simple), it's not worth breaking that code.


Like I said, this would only happen at 3.0 when all of your code will
break anyways.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

Jul 19 '05 #13

P: n/a
>>>>> "Jeremy" == Jeremy Bowers <je**@jerf.org> writes:

Jeremy> On Sun, 24 Apr 2005 22:59:12 -0700, Robert Kern wrote:
Never. If you really need a list

list(x*x for x in xrange(10))

Sadly, we can't remove list comprehensions until 3.0.


Jeremy> Why "remove" them? Instead, we have these things called
Jeremy> "comprehensions" (which, now that I say that, seems a
Jeremy> rather odd name), and you can control whether they result
Jeremy> in a list or a generator with () or [].

Still, list comprehensions should be implemented in terms of genexps
to get rid of the LC variable that is visible outside the scope of the
LC.

Jeremy> should be relatively simple), it's not worth breaking that
Jeremy> code.

Well, the code that relies on the dangling variable deserves to break.

--
Ville Vainio http://tinyurl.com/2prnb
Jul 19 '05 #14

P: n/a
On 25 Apr 2005 23:33:48 +0300, Ville Vainio <vi***@spammers.com> wrote:
>> "Jeremy" == Jeremy Bowers <je**@jerf.org> writes:
Jeremy> On Sun, 24 Apr 2005 22:59:12 -0700, Robert Kern wrote: >> Never. If you really need a list
>>
>> list(x*x for x in xrange(10))
>>
>> Sadly, we can't remove list comprehensions until 3.0.

Jeremy> Why "remove" them? Instead, we have these things called
Jeremy> "comprehensions" (which, now that I say that, seems a
Jeremy> rather odd name), and you can control whether they result
Jeremy> in a list or a generator with () or [].

Still, list comprehensions should be implemented in terms of genexps
to get rid of the LC variable that is visible outside the scope of the
LC.


+1 . I think that we should still have the form [genexp] , but without
the dangling variable, and implemented with generator expressions. It
seems to me that it is inconsistent if I can do list(genexp) but not
[genexp] , since they are so similar. Once that happens, we can tell
people who ask the OP's question that [genexp] is just another way to
spell list(genexp), and he should use it if he wants the entire list
constructed in memory.
Jeremy> should be relatively simple), it's not worth breaking that
Jeremy> code.

Well, the code that relies on the dangling variable deserves to break.


Agreed.

Peace
Bill Mill
bill.mill at gmail.com
Jul 19 '05 #15

P: n/a
On Mon, 25 Apr 2005 16:48:46 -0400, Bill Mill wrote:
On 25 Apr 2005 23:33:48 +0300, Ville Vainio <vi***@spammers.com> wrote:

Still, list comprehensions should be implemented in terms of genexps to
get rid of the LC variable that is visible outside the scope of the LC.

+1 . I think that we should still have the form [genexp] , but without the
dangling variable, and implemented with generator expressions. It seems to
me that it is inconsistent if I can do list(genexp) but not [genexp] ,
since they are so similar. Once that happens, we can tell people who ask
the OP's question that [genexp] is just another way to spell list(genexp),
and he should use it if he wants the entire list constructed in memory.


This is what I meant.

Robert Kern says the implementations really differ, but I submit that
is an accident of the order they were created in, not a fundamental
constraint. list(genexp) works today and does almost exactly the same
thing, minus an optimization or two that people are working on
generalizing anyways, which is the right approach. I doubt that Python 3.0
would have two radically different implementations; they'll just have the
genexp implementation, and an optimization for list creation if the list
creation can know the size in advance, regardless of where it came from.
Jeremy> should be relatively simple), it's not worth breaking that
Jeremy> code.

Well, the code that relies on the dangling variable deserves to break.


Agreed.


This is not what I meant and I agree with it too. Breaking code depending
on variable leaking is one thing; breaking all code that uses list
comprehensions is quite another. Just because 3.0 is going to break a lot
of code anyhow doesn't mean we should be *gratuitous* about it!
Jul 19 '05 #16

P: n/a
Jeremy Bowers <je**@jerf.org> writes:
On Mon, 25 Apr 2005 16:48:46 -0400, Bill Mill wrote:
generalizing anyways, which is the right approach. I doubt that Python 3.0
would have two radically different implementations; they'll just have the
genexp implementation, and an optimization for list creation if the list
creation can know the size in advance, regardless of where it came from.


Why do we have to wait for Python 3.0 for this? Couldn't list
comprehensions and generator expression be unified without breaking
existing code that didn't deserve to be broken?

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #17

P: n/a
On Mon, 25 Apr 2005 23:00:57 -0500, Mike Meyer wrote:
Why do we have to wait for Python 3.0 for this? Couldn't list
comprehensions and generator expression be unified without breaking
existing code that didn't deserve to be broken?


We don't; my mentioning 3.0 was just in reference to a previous comment.
In fact it'll probably happen sooner than then, I just have no direct
knowledge. (I know some people were talking about creating iterators that
know how many things they contain, which is the critical optimization, but
I don't know if that's gotten anywhere; I don't track the dev list as it
is too much traffic for my already strained email client :-) )
Jul 19 '05 #18

P: n/a
On Mon, 25 Apr 2005 16:48:46 -0400, Bill Mill <bi*******@gmail.com> wrote:
On 25 Apr 2005 23:33:48 +0300, Ville Vainio <vi***@spammers.com> wrote:
>>>>> "Jeremy" =3D=3D Jeremy Bowers <je**@jerf.org> writes:=20
Jeremy> On Sun, 24 Apr 2005 22:59:12 -0700, Robert Kern wrote:
>> Never. If you really need a list
>>
>> list(x*x for x in xrange(10))
>>
>> Sadly, we can't remove list comprehensions until 3.0.

=20
Jeremy> Why "remove" them? Instead, we have these things called
Jeremy> "comprehensions" (which, now that I say that, seems a
Jeremy> rather odd name), and you can control whether they result
Jeremy> in a list or a generator with () or [].
=20
Still, list comprehensions should be implemented in terms of genexps
to get rid of the LC variable that is visible outside the scope of the
LC.
=20


+1 . I think that we should still have the form [genexp] , but without
the dangling variable, and implemented with generator expressions. It
seems to me that it is inconsistent if I can do list(genexp) but not
[genexp] , since they are so similar. Once that happens, we can tell
people who ask the OP's question that [genexp] is just another way to
spell list(genexp), and he should use it if he wants the entire list
constructed in memory.

ISTM you have to account for
def foo(g): return g ... foo(123) 123 foo(c for c in 'abc') <generator object at 0x02EF154C> [(c for c in 'abc')] [<generator object at 0x02EF158C>] [c for c in 'abc']

['a', 'b', 'c']
Jeremy> should be relatively simple), it's not worth breaking that
Jeremy> code.
=20
Well, the code that relies on the dangling variable deserves to break.


Agreed.

Probably ;-)

Regards,
Bengt Richter
Jul 19 '05 #19

P: n/a
bo**@oz.net (Bengt Richter) writes:
On Mon, 25 Apr 2005 16:48:46 -0400, Bill Mill <bi*******@gmail.com> wrote:
+1 . I think that we should still have the form [genexp] , but without
the dangling variable, and implemented with generator expressions. It
seems to me that it is inconsistent if I can do list(genexp) but not
[genexp] , since they are so similar. Once that happens, we can tell
people who ask the OP's question that [genexp] is just another way to
spell list(genexp), and he should use it if he wants the entire list
constructed in memory.

ISTM you have to account for
>>> def foo(g): return g ... >>> foo(123) 123 >>> foo(c for c in 'abc') <generator object at 0x02EF154C> >>> [(c for c in 'abc')] [<generator object at 0x02EF158C>] >>> [c for c in 'abc']

['a', 'b', 'c']


Right. But that shouldn't be hard to do. Let genexp stand for a a
generator expression/list comprehension without any brackets on it at
all. Then [genexp] is the syntax to expand the list. [(genexp)] is the
syntax to create a list of one element - a generator
object. foo(genexp) will do the right thing.

The question under these circumstances is then: do you want bare
genexp to mean something? Right now, it's a syntax error. But there's
no reason you couldn't have:

y = x for x in stuff

assign a generator object to y.

<mike
--
Mike Meyer <mw*@mired.org> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
Jul 19 '05 #20

P: n/a

"Mike Meyer" <mw*@mired.org> wrote in message
The question under these circumstances is then: do you want bare
genexp to mean something? Right now, it's a syntax error. But there's
no reason you couldn't have:

y = x for x in stuff

assign a generator object to y.


Since this was considered, there is a reason why we don't have this. As I
remenber, but vaguely, Guido's reasoning was both stylistic and technical,
but you'd have to check the archives for more.

Terry J. Reedy


Jul 19 '05 #21

P: n/a
Mike Meyer wrote:
The question under these circumstances is then: do you want bare
genexp to mean something? Right now, it's a syntax error. But there's
no reason you couldn't have:

y = x for x in stuff


I like

y = (x for x in stuff)

as it is, it's way more readable.

Reinhold
Jul 19 '05 #22

P: n/a
On Tue, 26 Apr 2005 02:12:07 -0500, Mike Meyer wrote:
Right. But that shouldn't be hard to do. Let genexp stand for a a
generator expression/list comprehension without any brackets on it at all.
Then [genexp] is the syntax to expand the list. [(genexp)] is the syntax
to create a list of one element - a generator object. foo(genexp) will do
the right thing.


In other words, everything as it is today, only with different
implementations under the hood. :-)

Rolling back around to the original point, I don't see any reason to ban
[x for x in thing], changing current practice, when it is so easy to keep
it. [(x for x in thing)] is also current practice (perhaps "bad style" in
some sense, but definately possible in 2.4), and I don't think there is a
good reason to change that, either.
Jul 19 '05 #23

P: n/a
On Tue, 26 Apr 2005 13:22:28 -0400, "Terry Reedy" <tj*****@udel.edu> wrote:

"Mike Meyer" <mw*@mired.org> wrote in message
The question under these circumstances is then: do you want bare
genexp to mean something? Right now, it's a syntax error. But there's
no reason you couldn't have:

y = x for x in stuff

assign a generator object to y.


Since this was considered, there is a reason why we don't have this. As I
remenber, but vaguely, Guido's reasoning was both stylistic and technical,
but you'd have to check the archives for more.

Because it is too reminiscent of a lambda expression assignment? ;-)

Regards,
Bengt Richter
Jul 19 '05 #24

This discussion thread is closed

Replies have been disabled for this discussion.