reduce to be removed?

[Dustan]

According to the following page on Wikipedia:
http://en.wikipedia.org/wiki/Python_...re_development
reduce is going to be removed in python 3.0. It talks of an
accumulation loop; I have no idea what that's supposed to mean. So,

===============================

>>x =\

[[1,2,3],
[4,5,6],
[7,8,9]]

>>reduce(lambda a,b:a+b, x, [])

[1, 2, 3, 4, 5, 6, 7, 8, 9]
===============================

What's an accumulation loop, and how would I convert this code so it's
compatible with the future 3.0 (preferably in a short sweet expression
that I can embed in a list comprehension)?

[Fredrik Lundh]

Dustan wrote:

What's an accumulation loop, and how would I convert this code so it's
compatible with the future 3.0

the release of Python 3.0 is far away, and nobody knows how it's going
to look. trying to be future-compatible at this time is a major waste
of time and (not quite as wasteful as reopening yet another old "let's
make some pointless change to the language" thread, but almost).

surely you must have something better to do with your time ?

</F>

[Dustan]

Fredrik Lundh wrote:

Dustan wrote:

What's an accumulation loop, and how would I convert this code so it's
compatible with the future 3.0

the release of Python 3.0 is far away, and nobody knows how it's going
to look. trying to be future-compatible at this time is a major waste
of time and (not quite as wasteful as reopening yet another old "let's
make some pointless change to the language" thread, but almost).

surely you must have something better to do with your time ?

</F>

It's always nice to know there are such good-natured people ready to
help on this group. Anyway, I figured out a way to get the builtin
function 'sum' to work as I need:
sum([[1,2,3],[4,5,6],[7,8,9]], [])

On an unrelated note, can anyone explain this unpredictable behavior on
IDLE? I swear I never hit ctrl-c...
==============================

>>help(sum)

Help on built-in function sum in module __builtin__:

sum(...)
sum(sequence, start=0) -value

Returns the sum of a sequence of numbers (NOT strings) plus the
value
of parameter 'start'. When the sequence is empty, returns start.
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
help(sum)
File "C:\Python25\lib\site.py", line 346, in __call__
return pydoc.help(*args, **kwds)
File "C:\Python25\lib\pydoc.py", line 1642, in __call__
self.help(request)
File "C:\Python25\lib\pydoc.py", line 1687, in help
self.output.write('\n')
File "C:\Python25\lib\idlelib\PyShell.py", line 1246, in write
self.shell.write(s, self.tags)
File "C:\Python25\lib\idlelib\PyShell.py", line 1235, in write
raise KeyboardInterrupt
KeyboardInterrupt
==============================

[Virgil Dupras]

Dustan wrote:

According to the following page on Wikipedia:
http://en.wikipedia.org/wiki/Python_...re_development
reduce is going to be removed in python 3.0. It talks of an
accumulation loop; I have no idea what that's supposed to mean. So,

===============================

>x =\

[[1,2,3],
[4,5,6],
[7,8,9]]

>reduce(lambda a,b:a+b, x, [])

[1, 2, 3, 4, 5, 6, 7, 8, 9]
===============================

What's an accumulation loop, and how would I convert this code so it's
compatible with the future 3.0 (preferably in a short sweet expression
that I can embed in a list comprehension)?

itertools.chain or sum(x,[])

[Fredrik Lundh]

Dustan wrote:

It's always nice to know there are such good-natured people ready to
help on this group.

any special reason why you keep pretending that some random wikipedia
editor knows more about a future Python release than the people that
develops Python ?

Anyway, I figured out a way to get the builtin
function 'sum' to work as I need:
sum([[1,2,3],[4,5,6],[7,8,9]], [])

sum() is designed for adding numbers, not sequences. abusing it
for sequences leads to inefficient code, and extremely bad worst-
case behaviour, since you end up copying the same data over and
over and over again -- the function even checks for strings for
this very reason:

>>sum(["123", "456", "789"], "")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]

(maybe it should check for other well-known containers as well?)

if you care about writing robust code, why not just use a for-loop,
and the list extend method?

</F>

[Dustan]

Fredrik Lundh wrote:

Dustan wrote:

It's always nice to know there are such good-natured people ready to
help on this group.

any special reason why you keep pretending that some random wikipedia
editor knows more about a future Python release than the people that
develops Python ?

Be careful how you word that - this is the first time I've ever
referenced wikipedia in a question on this forum.

Anyway, I figured out a way to get the builtin
function 'sum' to work as I need:
sum([[1,2,3],[4,5,6],[7,8,9]], [])

sum() is designed for adding numbers, not sequences. abusing it
for sequences leads to inefficient code, and extremely bad worst-
case behaviour, since you end up copying the same data over and
over and over again

Thanks for some quality feedback for a change. Why can't you do that
more often?

-- the function even checks for strings for
this very reason:

>>sum(["123", "456", "789"], "")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]

(maybe it should check for other well-known containers as well?)

if you care about writing robust code, why not just use a for-loop,
and the list extend method?

Because I'm embedding this expression in a list comprehension (as I
stated in my original post), and last time I checked, it's not possible
to treat a for-loop as an expression (which is what a list
comprehension requires).

</F>

[Fredrik Lundh]

Dustan wrote:

Because I'm embedding this expression in a list comprehension

because?

</F>

[Dustan]

Fredrik Lundh wrote:

Dustan wrote:

Because I'm embedding this expression in a list comprehension

because?

</F>

Because I thought I would be able to get an answer without revealing
the exact details of what I am doing. I didn't realize that wasn't an
option. I'll try once more to give you an idea of what I'm trying to
accomplish without letting on the details.

>>foo =\

[[[1,2,3],[4,5,6],[7,8,9]],
[[3,2,1],[6,5,4],[9,8,7]]]

Here, foo appears to be a 3-dimensional list - except it's supposed to
be 2-dimensional. The inner-list-of-lists is a result of how I'm
producing the data, and now I want to do a mass-concatenation (or
extending) of the inner-list-of-lists, and come up with this:

>>foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

True

What's the best way to accomplish this?

It's not quite this simple, but let's just see what you can come up
with the information at hand, and I'll see if I can adapt it to my
needs.

[Fredrik Lundh]

Dustan wrote:

> Because I'm embedding this expression in a list comprehension

because?

Because I thought I would be able to get an answer without revealing
the exact details of what I am doing.

alright, let's try again: why do you need a self-contained reduce
replacement that can be embedded inside a list comprehension ?

</F>

[Dustan]

Fredrik Lundh wrote:

Dustan wrote:

Because I'm embedding this expression in a list comprehension

because?

>
Because I thought I would be able to get an answer without revealing
the exact details of what I am doing.

alright, let's try again: why do you need a self-contained reduce
replacement that can be embedded inside a list comprehension ?

</F>

>>foo =\

[[[1,2,3],[4,5,6],[7,8,9]],
[[3,2,1],[6,5,4],[9,8,7]]]

Here, foo appears to be a 3-dimensional list - except it's supposed to
be 2-dimensional. The inner-list-of-lists is a result of how I'm
producing the data, and now I want to do a mass-concatenation (or
extending) of the inner-list-of-lists, and come up with this result:

>>foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

True

What's the best way to accomplish this?

It's not quite this simple, but let's just see what you can come up
with the information at hand, and I'll see if I can adapt it to my
needs.

[Robert Kern]

Dustan wrote:

Fredrik Lundh wrote:

>if you care about writing robust code, why not just use a for-loop,
and the list extend method?

Because I'm embedding this expression in a list comprehension (as I
stated in my original post), and last time I checked, it's not possible
to treat a for-loop as an expression (which is what a list
comprehension requires).

As with all such things, you stick the implementation in a well-named function
and simply call the function everywhere. The implementation never needs to be a
one-liner expression.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Fredrik Lundh]

Dustan wrote:

>>>foo =\

[[[1,2,3],[4,5,6],[7,8,9]],
[[3,2,1],[6,5,4],[9,8,7]]]

Here, foo appears to be a 3-dimensional list - except it's supposed to
be 2-dimensional. The inner-list-of-lists is a result of how I'm
producing the data, and now I want to do a mass-concatenation (or
extending) of the inner-list-of-lists, and come up with this result:

>>>foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

True

that still doesn't explain your "the expression must be used in a list
comprehension" requirement, though. assuming that the sizes are
varying, and at least sometimes a lot larger than 3x3, I'd probably
write the above as

for index, item in enumerate(foo):
this = []
for i in item:
this.extend(i)
foo[index] = this

which should be pretty efficient, since it avoids unnecessary function
calls, and is amortized linear time instead of O(N**2).

or, if I was in a hurry, and didn't really care if the inner sequences
were lists or tuples:

foo = map(Tkinter._flatten, foo)

</F>

[Steven D'Aprano]

On Sat, 11 Nov 2006 17:42:32 -0800, Dustan wrote:

>alright, let's try again: why do you need a self-contained reduce
replacement that can be embedded inside a list comprehension ?

</F>

>>>foo =\

[[[1,2,3],[4,5,6],[7,8,9]],
[[3,2,1],[6,5,4],[9,8,7]]]

Here, foo appears to be a 3-dimensional list - except it's supposed to
be 2-dimensional. The inner-list-of-lists is a result of how I'm
producing the data, and now I want to do a mass-concatenation (or
extending) of the inner-list-of-lists, and come up with this result:

>>>foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

True

What's the best way to accomplish this?

I don't know if this is the best, but it didn't take long to come up with
it:

>>foo = [[[1,2,3],[4,5,6],[7,8,9]],

.... [[3,2,1],[6,5,4],[9,8,7]]]

>>>
def unroll(list3d):

.... newl = []
.... for sublist in list3d:
.... newl.append(sum(sublist, []))
.... return newl
....

>>bar = unroll(foo)
bar

[[1, 2, 3, 4, 5, 6, 7, 8, 9], [3, 2, 1, 6, 5, 4, 9, 8, 7]]
Repeat after me:

"Not everything has to be a one-liner."
If sum() is too slow, because your sub-lists are huge, you can easily
factor that out and replace it with something using extend.
--
Steven.

[Dustan]

Robert Kern wrote:

Dustan wrote:

Fredrik Lundh wrote:

if you care about writing robust code, why not just use a for-loop,
and the list extend method?

Because I'm embedding this expression in a list comprehension (as I
stated in my original post), and last time I checked, it's not possible
to treat a for-loop as an expression (which is what a list
comprehension requires).

As with all such things, you stick the implementation in a well-named function
and simply call the function everywhere. The implementation never needs to be a
one-liner expression.

It's already in a function, but in order to convert the reduce function
into a for-loop that I can use as an expression, I would have to create
another independent function, which would make that code more cluttered
than it already is.

Unless you're saying to perform the list comprehension manually. That
would be two for-loops I use in the list comprehension, plus another
one to take place of the reduce function. Sure, that spreads out each
individual step a little more, but it also makes it more difficult to
understand what the overall goal of the code is (I'm going for
readability as well as easy maintenance here).

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Dustan]

Fredrik Lundh wrote:

Dustan wrote:

>>foo =\

[[[1,2,3],[4,5,6],[7,8,9]],
[[3,2,1],[6,5,4],[9,8,7]]]

Here, foo appears to be a 3-dimensional list - except it's supposed to
be 2-dimensional. The inner-list-of-lists is a result of how I'm
producing the data, and now I want to do a mass-concatenation (or
extending) of the inner-list-of-lists, and come up with this result:

>>foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

True

that still doesn't explain your "the expression must be used in a list
comprehension" requirement, though.

Oh, right; sorry about the confusion. The list isn't quite that simple,
and in order to pull the right pieces together, I use a list
comprehension.

assuming that the sizes are
varying, and at least sometimes a lot larger than 3x3, I'd probably
write the above as

for index, item in enumerate(foo):
this = []
for i in item:
this.extend(i)
foo[index] = this

I'll see if that works.

which should be pretty efficient, since it avoids unnecessary function
calls, and is amortized linear time instead of O(N**2).

or, if I was in a hurry, and didn't really care if the inner sequences
were lists or tuples:

foo = map(Tkinter._flatten, foo)

</F>

Steven D'Aprano wrote:

I don't know if this is the best, but it didn't take long to come up with
it:

>foo = [[[1,2,3],[4,5,6],[7,8,9]],

... [[3,2,1],[6,5,4],[9,8,7]]]

>>
def unroll(list3d):

... newl = []
... for sublist in list3d:
... newl.append(sum(sublist, []))
... return newl

deja vu...

>bar = unroll(foo)
bar

[[1, 2, 3, 4, 5, 6, 7, 8, 9], [3, 2, 1, 6, 5, 4, 9, 8, 7]]
Repeat after me:

"Not everything has to be a one-liner."

Not everything has to be a one-liner. But readability helps.

If sum() is too slow, because your sub-lists are huge, you can easily
factor that out and replace it with something using extend.
--
Steven.

[Robert Kern]

Dustan wrote:

Robert Kern wrote:

>Dustan wrote:

>>Fredrik Lundh wrote:
if you care about writing robust code, why not just use a for-loop,
and the list extend method?
Because I'm embedding this expression in a list comprehension (as I
stated in my original post), and last time I checked, it's not possible
to treat a for-loop as an expression (which is what a list
comprehension requires).

As with all such things, you stick the implementation in a well-named function
and simply call the function everywhere. The implementation never needs to be a
one-liner expression.

It's already in a function, but in order to convert the reduce function
into a for-loop that I can use as an expression, I would have to create
another independent function, which would make that code more cluttered
than it already is.

Unless you're saying to perform the list comprehension manually. That
would be two for-loops I use in the list comprehension, plus another
one to take place of the reduce function. Sure, that spreads out each
individual step a little more, but it also makes it more difficult to
understand what the overall goal of the code is (I'm going for
readability as well as easy maintenance here).

Let's just say that we disagree entirely on what constitutes readability and
maintainability.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Paddy]

Dustan wrote:

Anyway, I figured out a way to get the builtin
function 'sum' to work as I need:
sum([[1,2,3],[4,5,6],[7,8,9]], [])

Hah!
No-one expects sum to be used on anything but numbers.

Except lists as above.

No-one expects sum to be used on anything but numbers, and maybe lists
too.

;-)
(My apologies for my poor grasp of the Spanish Inquisition sketch; and
I was alluding to
this thread:
http://groups.google.com/group/comp....4525384f8028b5 )

[Ben Finney]

"Dustan" <Du**********@gmail.comwrites:

Steven D'Aprano wrote:

Repeat after me:

"Not everything has to be a one-liner."

Not everything has to be a one-liner. But readability helps.

Indeed. Complex one-liners are rarely as readable as a well-named
function call, implemented with several explicit readable lines in
one place, and then re-used as many times as necessary.

--
\ "Anyone who believes exponential growth can go on forever in a |
`\ finite world is either a madman or an economist." -- Kenneth |
_o__) Boulding |
Ben Finney

[Dustan]

Alright, I can see I'm a bit outvoted here. I tried your suggestions
and it worked fine.

I'll also try to consider in the future that part of the problem might
be lack of information conveyed on my part.

[George Sakkis]

Dustan wrote:

Alright, I can see I'm a bit outvoted here. I tried your suggestions
and it worked fine.

I'll also try to consider in the future that part of the problem might
be lack of information conveyed on my part.

If you insist on one-liners, it can be done without sum(), though it
probably doesn't buy you much in readability:

from itertools import chain[list(chain(*row)) for row in foo]

By the way, if this was not a toy example and you're doing serious work
with n-dimensional arrays, make yourself a favor and install NumPy;
it's usually both faster and more elegant for array manipulations than
pure python.

George

[Steve Holden]

Dustan wrote:

Alright, I can see I'm a bit outvoted here. I tried your suggestions
and it worked fine.

I'll also try to consider in the future that part of the problem might
be lack of information conveyed on my part.

Well, since such honest and public self-examination is so rarely seen
(especially in the face of the effbot's occasional vehement bluntness)
allow me to congratulate you on your open-mindedness.

Sometimes it seems like "egoless programming" went out with the 1980's.
I'm sure I'll enjoy reading your future posts.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Steve Holden]

Dustan wrote:

Fredrik Lundh wrote:

>Dustan wrote:

[...]

>Repeat after me:

"Not everything has to be a one-liner."

Not everything has to be a one-liner. But readability helps.

Indeed. And there is absolutely no conflict between those two statements.

Guido resisted introducing tertiary expressions for as long as he did
precisely because he realised that some people would use them to produce
unreadable one-liners instead of readable multi-line expressions of the
same idea.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Steve Holden]

Paddy wrote:

Dustan wrote:

>Anyway, I figured out a way to get the builtin
function 'sum' to work as I need:
sum([[1,2,3],[4,5,6],[7,8,9]], [])

Hah!
No-one expects sum to be used on anything but numbers.

Except lists as above.

No-one expects sum to be used on anything but numbers, and maybe lists
too.

;-)

[voice mumbles:] "What about tuples"

Right, tuples too. But apart from tuples and lists, nobody expects sum
to be used on anything but numbers.

In actual fact when Alex Martelli introduced sum() he intended it to be
polymorphic over all the container types including strings. The check to
exclude the string case was added when it was determined that it was
terribly inefficient to concatenate strings that way. the same may well
apply to other sequences.

I suppose it's only a matter of time before someone wants to define
dict.__add__ ...

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Fredrik Lundh]

Steve Holden wrote:

In actual fact when Alex Martelli introduced sum() he intended it to be
polymorphic over all the container types including strings. The check to
exclude the string case was added when it was determined that it was
terribly inefficient to concatenate strings that way. the same may well
apply to other sequences.

an important difference is that if you repeatedly copy a string object,
you'll actually copy *all* data in the string over and over again. when
you copy a list, you only copy references to the objects in the list.
the size of the data *in* the list doesn't matter.

I suppose it's only a matter of time before someone wants to define
dict.__add__ ...

that's been proposed quite a few times, and always gets stuck when it's
time to define the exact semantics for dealing with collisions. there
are simply too many ways to do it, and all of them are can be trivially
and efficiently implemented in terms of copy/update or for-in (or, in
quite a few cases, by using sets instead of dicts).

</F>

[Steve Holden]

Fredrik Lundh wrote:

Steve Holden wrote:

[...]

>I suppose it's only a matter of time before someone wants to define
dict.__add__ ...

that's been proposed quite a few times, and always gets stuck when it's
time to define the exact semantics for dealing with collisions. there
are simply too many ways to do it, and all of them are can be trivially
and efficiently implemented in terms of copy/update or for-in (or, in
quite a few cases, by using sets instead of dicts).

But, as we both know, the fact that something can be trivially
implemented in Python doesn't stop people asking for it to be added as
native.

oh-no-indeed-ly y'rs - steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Kay Schluehr]

On 12 Nov., 00:14, Fredrik Lundh <[e-mail]:

if you care about writing robust code, why not just use a for-loop,
and the list extend method?

</F>

Because generic solutions using HOFs increase abstraction and reduce()
the amount of code one has to write even when they are outcasted by
Guido?

[Dustan]

George Sakkis wrote:

Dustan wrote:

Alright, I can see I'm a bit outvoted here. I tried your suggestions
and it worked fine.

I'll also try to consider in the future that part of the problem might
be lack of information conveyed on my part.

If you insist on one-liners, it can be done without sum(), though it
probably doesn't buy you much in readability:

from itertools import chain[list(chain(*row)) for row in foo]

By the way, if this was not a toy example and you're doing serious work
with n-dimensional arrays, make yourself a favor and install NumPy;
it's usually both faster and more elegant for array manipulations than
pure python.

1. I've already written pretty much all the code, and a complete
rewrite would be rather difficult.

2. While I haven't taken a good look at NumPy, my intuition tells me it
won't work with complex data types, which wouldn't work for me at all.

Am I correct on that second one?

[Robert Kern]

Dustan wrote:

2. While I haven't taken a good look at NumPy, my intuition tells me it
won't work with complex data types, which wouldn't work for me at all.

Am I correct on that second one?

No. numpy can make arrays of Python objects in addition to arrays of double,
unsigned int, etc.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Antoon Pardon]

On 2006-11-15, Robert Kern <ro*********@gmail.comwrote:

Dustan wrote:

>2. While I haven't taken a good look at NumPy, my intuition tells me it
won't work with complex data types, which wouldn't work for me at all.

Am I correct on that second one?

No. numpy can make arrays of Python objects in addition to arrays of double,
unsigned int, etc.

Does numpy still gain you much if you have arrays of Python objects?

--
Antoon Pardon

[Robert Kern]

Antoon Pardon wrote:

On 2006-11-15, Robert Kern <ro*********@gmail.comwrote:

>Dustan wrote:

>>2. While I haven't taken a good look at NumPy, my intuition tells me it
won't work with complex data types, which wouldn't work for me at all.

Am I correct on that second one?

No. numpy can make arrays of Python objects in addition to arrays of double,
unsigned int, etc.

Does numpy still gain you much if you have arrays of Python objects?

It depends on what you want out of it. You won't get fast math, but you will get
convenient multidimensional indexing, slicing, stacking, reshaping, and
transposition.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco