473,320 Members | 1,832 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

flatten a level one list

Is there some smart/fast way to flatten a level one list using the
latest iterator/generator idioms.

The problem arises in coneverting lists of (x,y) coordinates into a
single list of coordinates eg

f([(x0,y0),(x1,y1),....]) --> [x0,y0,x1,y1,....] or

g([x0,x1,x2,......],[y0,y1,y2,....]) --> [x0,y0,x1,y1,....]

clearly if f is doable then g can be done using zip. I suppose this is a
special case flatten, but can flatten be done fast? The python recipes
seem rather slow compared to the builtin functions.
--
Robin Becker
Jan 12 '06 #1
32 2733

Robin Becker wrote:
Is there some smart/fast way to flatten a level one list using the
latest iterator/generator idioms.

The problem arises in coneverting lists of (x,y) coordinates into a
single list of coordinates eg

f([(x0,y0),(x1,y1),....]) --> [x0,y0,x1,y1,....] or

g([x0,x1,x2,......],[y0,y1,y2,....]) --> [x0,y0,x1,y1,....]

clearly if f is doable then g can be done using zip. I suppose this is a
special case flatten, but can flatten be done fast? The python recipes
seem rather slow compared to the builtin functions.


how fast is fast ?

for this case, is the following good enough ?

def flat(li):
for x,y in li:
yield x
yield y

Jan 12 '06 #2
Robin Becker schrieb:
Is there some smart/fast way to flatten a level one list using the
latest iterator/generator idioms.

The problem arises in coneverting lists of (x,y) coordinates into a
single list of coordinates eg

f([(x0,y0),(x1,y1),....]) --> [x0,y0,x1,y1,....] or

g([x0,x1,x2,......],[y0,y1,y2,....]) --> [x0,y0,x1,y1,....]

clearly if f is doable then g can be done using zip. I suppose this is a
special case flatten, but can flatten be done fast? The python recipes
seem rather slow compared to the builtin functions.


well then:

first of all, i need to say, if speed really matters, do it in C.
that being said, python can be fast, too. for this task psyco is your
friend. i got this output from the script given below:

without psyco:

flatten1: 2.78046748059
flatten2: 2.90226239686
flatten3: 4.91070862996
goopy_flatten1: 8.22951110963
goopy_flatten2: 8.56373180172

with psyco:

flatten1: 1.17390339924
flatten2: 1.7209583052
flatten3: 1.18490295558
goopy_flatten1: 1.34892236194
goopy_flatten2: 1.68568386584

the goopy function is taken from the google-functional package (but is
treated a bit unfair, i must admit, being wrapped in a lambda)

so, what does that show us? izip seems a bit faster than zip with these
input data. you want to do your own timings with more realistic data.
and all these functions are what just came to my mind, i'm sure they
can be improved.

hope this helps,

--
David.

used script:
----------------------------------------------------------------

from itertools import izip

xdata = range(1000)
ydata = range(1000)[::-1]

def flatten1():
return [x for pair in izip(xdata, ydata) for x in pair]

def flatten2():
return [x for pair in zip(xdata, ydata) for x in pair]

def flatten3():
res = []
for pair in izip(xdata, ydata):
for x in pair:
res.append(x)
return res

def goopy_flatten(seq):
lst = []
for x in seq:
if type(x) is list or type(x) is tuple:
for val in x:
lst.append(val)
else:
lst.append(x)
return lst

goopy_flatten1 = lambda: goopy_flatten(izip(xdata, ydata))
goopy_flatten2 = lambda: goopy_flatten(zip(xdata, ydata))

if __name__=='__main__':
from timeit import Timer

functions = ['flatten1', 'flatten2', 'flatten3', 'goopy_flatten1', 'goopy_flatten2']

print 'without psyco:'
print

for fn in functions:
t = Timer(fn+'()', 'from __main__ import '+fn)
print fn+':', t.timeit(5000)

try: import psyco; psyco.full()
except ImportError: pass

print
print 'with psyco:'
print

for fn in functions:
t = Timer(fn+'()', 'from __main__ import '+fn)
print fn+':', t.timeit(5000)
Jan 12 '06 #3
Robin Becker <ro***@SPAMREMOVEjessikat.fsnet.co.uk> writes:
f([(x0,y0),(x1,y1),....]) --> [x0,y0,x1,y1,....]

import operator
a=[(1,2),(3,4),(5,6)]
reduce(operator.add,a)

(1, 2, 3, 4, 5, 6)
Jan 12 '06 #4
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
import operator
a=[(1,2),(3,4),(5,6)]
reduce(operator.add,a)

(1, 2, 3, 4, 5, 6)


(Note that the above is probably terrible if the lists are large and
you're after speed.)
Jan 12 '06 #5
> Robin Becker schrieb:
Is there some smart/fast way to flatten a level one list using the
latest iterator/generator idioms. ....

David Murmann wrote:
Some functions and timings

....

Here are some more timings of David's functions, and a couple of additional
contenders that time faster on my box (I don't have psyco):

# From David Murman
from itertools import izip

xdata = range(1000)
ydata = range(1000)[::-1]

def flatten1(x, y):
return [i for pair in izip(x, y) for i in pair]

def flatten2(x, y):
return [i for pair in zip(x, y) for i in pair]

def flatten3(x, y):
res = []
for pair in izip(x, y):
for i in pair:
res.append(i)
return res
# New attempts:
from itertools import imap
def flatten4(x, y):
l = []
list(imap(l.extend, izip(x, y)))
return l
from Tkinter import _flatten
def flatten5(x, y):
return list(_flatten(zip(x, y)))

flatten_funcs = [flatten1, flatten2, flatten3, flatten4, flatten5]

def testthem():
flatten1res = flatten_funcs[0](xdata, ydata)
for func in flatten_funcs:
assert func(xdata, ydata) == flatten1res

def timethem():
for func in flatten_funcs:
print shell.timefunc(func, xdata, ydata)
testthem()
timethem() flatten1(...) 704 iterations, 0.71msec per call
flatten2(...) 611 iterations, 0.82msec per call
flatten3(...) 344 iterations, 1.46msec per call
flatten4(...) 1286 iterations, 389.08usec per call
flatten5(...) 1219 iterations, 410.24usec per call


Michael

Jan 12 '06 #6
Michael Spencer wrote:
> Robin Becker schrieb:
>> Is there some smart/fast way to flatten a level one list using the
>> latest iterator/generator idioms. ...

David Murmann wrote:
> Some functions and timings ...


Here's one more that's quite fast using Psyco, but only average without it.
def flatten6():
n = min(len(xdata), len(ydata))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]

-tim


Here are some more timings of David's functions, and a couple of additional
contenders that time faster on my box (I don't have psyco):

# From David Murman
from itertools import izip

xdata = range(1000)
ydata = range(1000)[::-1]

def flatten1(x, y):
return [i for pair in izip(x, y) for i in pair]

def flatten2(x, y):
return [i for pair in zip(x, y) for i in pair]

def flatten3(x, y):
res = []
for pair in izip(x, y):
for i in pair:
res.append(i)
return res
# New attempts:
from itertools import imap
def flatten4(x, y):
l = []
list(imap(l.extend, izip(x, y)))
return l
from Tkinter import _flatten
def flatten5(x, y):
return list(_flatten(zip(x, y)))

flatten_funcs = [flatten1, flatten2, flatten3, flatten4, flatten5]

def testthem():
flatten1res = flatten_funcs[0](xdata, ydata)
for func in flatten_funcs:
assert func(xdata, ydata) == flatten1res

def timethem():
for func in flatten_funcs:
print shell.timefunc(func, xdata, ydata)
>>> testthem()
>>> timethem() flatten1(...) 704 iterations, 0.71msec per call
flatten2(...) 611 iterations, 0.82msec per call
flatten3(...) 344 iterations, 1.46msec per call
flatten4(...) 1286 iterations, 389.08usec per call
flatten5(...) 1219 iterations, 410.24usec per call >>>


Michael


Jan 12 '06 #7
Tim Hochberg wrote:
Michael Spencer wrote:
> Robin Becker schrieb:
>> Is there some smart/fast way to flatten a level one list using the
>> latest iterator/generator idioms.

...

David Murmann wrote:
> Some functions and timings

...


Here's one more that's quite fast using Psyco, but only average without it.
def flatten6():
n = min(len(xdata), len(ydata))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]

-tim

Indeed:

I added yours to the list (after adding the appropriate return)
testthem()
timethem() flatten1(...) 702 iterations, 0.71msec per call
flatten2(...) 641 iterations, 0.78msec per call
flatten3(...) 346 iterations, 1.45msec per call
flatten4(...) 1447 iterations, 345.66usec per call
flatten5(...) 1218 iterations, 410.55usec per call
flatten6(...) 531 iterations, 0.94msec per call


(See earlier post for flatten1-5)

Michael

Jan 12 '06 #8
Paul Rubin wrote:
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
>import operator
>a=[(1,2),(3,4),(5,6)]
>reduce(operator.add,a)


(1, 2, 3, 4, 5, 6)

(Note that the above is probably terrible if the lists are large and
you're after speed.)

yes, and it is all in C and so could be a contender for the speed champ.
I guess what you're saying is that it's doing

(1,2)
(1,2)+(3,4)
(1,2,3,4)+(5,6)

ie we do n or n-1 tuple additions each of which requires tuple
allocation etc etc

A fast implementation would probably allocate the output list just once
and then stream the values into place with a simple index.
--
Robin Becker
Jan 12 '06 #9
Another try:

def flatten6(x, y):
return list(chain(*izip(x, y)))

(any case, this is shorter ;-)

Cyril

On 1/12/06, Michael Spencer <ma**@telcopartners.com> wrote:
Tim Hochberg wrote:
Michael Spencer wrote:
> Robin Becker schrieb:
>> Is there some smart/fast way to flatten a level one list using the
>> latest iterator/generator idioms.
...

David Murmann wrote:
> Some functions and timings
...


Here's one more that's quite fast using Psyco, but only average withoutit.
def flatten6():
n = min(len(xdata), len(ydata))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]

-tim

Indeed:

I added yours to the list (after adding the appropriate return)
>>> testthem()
>>> timethem() flatten1(...) 702 iterations, 0.71msec per call
flatten2(...) 641 iterations, 0.78msec per call
flatten3(...) 346 iterations, 1.45msec per call
flatten4(...) 1447 iterations, 345.66usec per call
flatten5(...) 1218 iterations, 410.55usec per call
flatten6(...) 531 iterations, 0.94msec per call >>>


(See earlier post for flatten1-5)

Michael

--
http://mail.python.org/mailman/listinfo/python-list

Jan 12 '06 #10
Tim Hochberg wrote:
Here's one more that's quite fast using Psyco, but only average without
it. def flatten6():
n = min(len(xdata), len(ydata))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]


I you require len(xdata) == len(ydata) there's an easy way to move the loop
into C:

def flatten7():
n = len(xdata)
assert len(ydata) == n
result = [None] * (2*n)
result[::2] = xdata
result[1::2] = ydata
return result

$ python -m timeit 'from flatten import flatten6 as f' 'f()'
1000 loops, best of 3: 847 usec per loop
$ python -m timeit 'from flatten import flatten7 as f' 'f()'
10000 loops, best of 3: 43.9 usec per loop

Peter
Jan 12 '06 #11
Robin Becker <ro***@SPAMREMOVEjessikat.fsnet.co.uk> writes:
>>reduce(operator.add,a)

...
A fast implementation would probably allocate the output list just
once and then stream the values into place with a simple index.


That's what I hoped "sum" would do, but instead it barfs with a type
error. So much for duck typing.
Jan 12 '06 #12
Robin Becker wrote:
Paul Rubin wrote:
Paul Rubin <http://ph****@NOSPAM.invalid> writes:
>>import operator
>>a=[(1,2),(3,4),(5,6)]
>>reduce(operator.add,a)

(1, 2, 3, 4, 5, 6)

(Note that the above is probably terrible if the lists are large and
you're after speed.)

yes, and it is all in C and so could be a contender for the speed champ.
I guess what you're saying is that it's doing

That is what I thought too but seems that [x for pair in li for x in
pair] is the fastest on my machine and what is even stranger is that if
I use psyco.full(), I got a 10x speed up for this solution(list
comprehension) which is head and shoulder above all the other suggested
so far.

Jan 12 '06 #13
Well, maybe it's time to add a n-levels flatten() function to the
language (or to add it to itertools). Python is open source, but I am
not able to modify its C sources yet... Maybe Raymond Hettinger can
find some time to do it for Py 2.5.

Bye,
bearophile

Jan 12 '06 #14
[Robin Becker]
Is there some smart/fast way to flatten a level one list using the
latest iterator/generator idioms.

The problem arises in coneverting lists of (x,y) coordinates into a
single list of coordinates eg

f([(x0,y0),(x1,y1),....]) --> [x0,y0,x1,y1,....]


Here's one way:
d = [('x0','y0'), ('x1','y1'), ('x2','y2'), ('x3', 'y3')]
list(chain(*d))

['x0', 'y0', 'x1', 'y1', 'x2', 'y2', 'x3', 'y3']

FWIW, if you're into working out puzzles, there's no end of interesting
iterator algebra tricks. Here are a few identities for your
entertainment:

# Given s (any sequence) and n (a non-negative integer):
assert zip(*izip(*tee(s,n))) == [tuple(s)]*n
assert list(chain(*tee(s,n))) == list(s)*n
assert map(itemgetter(0),groupby(sorted(s))) == sorted(set(s))
Raymond

Jan 12 '06 #15
In article <7x************@ruckus.brouhaha.com>,
Paul Rubin <http://ph****@NOSPAM.invalid> wrote:
Robin Becker <ro***@SPAMREMOVEjessikat.fsnet.co.uk> writes:
>>>>>reduce(operator.add,a)

...

That's what I hoped "sum" would do, but instead it barfs with a type
error. So much for duck typing.


sum(...)
sum(sequence, start=0) -> value

If you're using sum() as a 1-level flatten you need to give it
start=[].

--
\S -- si***@chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
Jan 12 '06 #16
Sion Arrowsmith <si***@chiark.greenend.org.uk> writes:
sum(sequence, start=0) -> value

If you're using sum() as a 1-level flatten you need to give it
start=[].


Oh, right, I should have remembered that. Thanks. Figuring out
whether it's quadratic or linear would still take an experiment or
code inspection which I'm not up for at the moment.
Jan 12 '06 #17
Peter Otten wrote:
Tim Hochberg wrote:

Here's one more that's quite fast using Psyco, but only average without
it.

def flatten6():
n = min(len(xdata), len(ydata))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]

I you require len(xdata) == len(ydata) there's an easy way to move the loop
into C:

def flatten7():
n = len(xdata)
assert len(ydata) == n
result = [None] * (2*n)
result[::2] = xdata
result[1::2] = ydata
return result

$ python -m timeit 'from flatten import flatten6 as f' 'f()'
1000 loops, best of 3: 847 usec per loop
$ python -m timeit 'from flatten import flatten7 as f' 'f()'
10000 loops, best of 3: 43.9 usec per loop

Peter


That's the winner for my machine and it works in the case I need :)

The numbers are microseconds/call

I used 20 reps and n from 10 up to 1000.

no psyco
Name 10 20 100 200 500 1000
flatten1 111.383 189.298 745.709 1397.300 3499.579 6628.775
flatten2 142.923 209.496 907.182 1521.618 3565.397 7197.228
flatten3 176.224 314.342 1385.958 2733.560 6726.693 12879.067
flatten4 112.696 163.010 518.250 901.288 1979.749 3657.364
flatten5 78.334 110.768 386.949 711.794 1617.664 3255.749
flatten6 142.867 230.420 894.639 1767.012 4499.734 9017.906
flatten6a 163.093 263.330 1071.337 2084.287 5209.433 10383.610
flatten6b 180.582 275.761 1063.794 2074.705 5057.123 10043.567
flatten6c 167.898 253.664 974.202 1948.181 4821.339 9562.780
flatten6d 132.475 201.702 738.194 1406.659 3612.107 7242.038
flatten7 59.030 62.354 90.347 130.771 254.613 438.994
flatten8 88.978 173.737 1667.111 5674.297 28907.501 106330.749
flatten8a 107.388 225.951 2323.563 7088.136 34254.381 114538.384
psyco
Name 10 20 100 200 500 1000
flatten1 84.424 114.596 393.374 714.728 1809.448 3197.837
flatten2 102.387 136.302 507.243 942.494 2276.770 4451.990
flatten3 85.206 111.020 379.713 715.957 1607.104 3191.188
flatten4 102.667 144.599 509.255 856.450 1839.591 3425.128
flatten5 79.898 115.490 383.904 730.484 1739.411 3515.978
flatten6 54.560 61.293 183.012 332.109 837.146 1604.366
flatten6a 79.647 108.114 405.107 752.917 1873.674 3824.620
flatten6b 111.746 132.978 473.189 907.378 2217.600 4257.357
flatten6c 98.756 110.629 376.724 730.037 1772.963 3524.247
flatten6d 59.253 69.199 172.731 295.820 717.577 1402.720
flatten7 51.291 39.754 65.707 104.902 233.214 405.694
flatten8 87.050 166.837 1665.407 5410.576 28459.567 107847.422
flatten8a 122.753 251.457 2766.944 7931.204 36353.503 120773.674

###############################
from itertools import izip
import timeit

_R=100

def flatten1(x, y):
'''D Murman'''
return [i for pair in izip(x, y) for i in pair]

def flatten2(x, y):
'''D Murman'''
return [i for pair in zip(x, y) for i in pair]

def flatten3(x, y):
'''D Murman'''
res = []
for pair in izip(x, y):
for i in pair:
res.append(i)
return res

# New attempts:
from itertools import imap
def flatten4(x, y):
'''D Murman'''
l = []
list(imap(l.extend, izip(x, y)))
return l
from Tkinter import _flatten
def flatten5(x, y):
'''D Murman'''
return list(_flatten(zip(x, y)))

def flatten6(x,y):
'''Tim Hochberg'''
n = min(len(x), len(y))
result = [None] * (2*n)
for i in xrange(n):
result[2*i] = xdata[i]
result[2*i+1] = ydata[i]
return result

def flatten6a(x,y):
'''Robin Becker variant of 6'''
n = min(len(x), len(y))
result = [None] * (2*n)
for i in xrange(n):
result[2*i:2*i+2] = xdata[i],ydata[i]
return result

def flatten6b(x,y):
'''Robin Becker variant of 6'''
n = min(len(x), len(y))
result = [None] * (2*n)
for i,pair in enumerate(zip(xdata,ydata)):
result[2*i:2*i+2] = pair
return result

def flatten6c(x,y):
'''Robin Becker variant of 6'''
n = min(len(x), len(y))
result = [None] * (2*n)
for i,pair in enumerate(izip(xdata,ydata)):
result[2*i:2*i+2] = pair
return result

def flatten6d(x,y):
'''Robin Becker variant of 6'''
n = min(len(x), len(y))
result = [None] * (2*n)
j = 0
for i in xrange(n):
result[j] = xdata[i]
result[j+1] = ydata[i]
j+=2
return result

from operator import add as operator_add
def flatten8(x,y):
'''Paul Rubin'''
return reduce(operator_add,zip(x,y),())

def flatten8a(x,y):
'''Robin Becker variant of 8'''
return reduce(operator_add,(xy for xy in izip(x,y)),())

def flatten7(x,y):
'''Peter Otten special case equal lengths'''
n = len(x)
assert len(y) == n
result = [None] * (2*n)
result[::2] = x
result[1::2] = y
return result

funcs = [(n,v) for n,v in globals().items() if callable(v) and
n.startswith('flatten')]
funcs.sort()

def testthem():
res0 = funcs[0][1](xdata, ydata)
for name,func in funcs:
res = list(func(xdata, ydata))
if res!=res0:
print name,' fails', type(res0), type(res), res0[:5],res[:5],
res0[-5:],res[-5:]

def timethem(D,n):
for name,func in funcs:
t = timeit.Timer(name+"(xdata,ydata)",'from __main__ import
xdata,ydata,'+name)
D.setdefault(name,{})[n] = 1e7*t.timeit(_R)/float(_R)

if __name__=='__main__':
N = [10, 20, 100, 200, 500, 1000]
xdata = range(N[-1])
ydata = xdata[::-1]
testthem()
for p in 'no psyco','psyco':
D={}
if p=='psyco':
import psyco
psyco.full()
for n in N:
xdata = range(n)
ydata = xdata[::-1]
timethem(D,n)
print '\n',p
fmt = '%%%ds' % max(map(len,[x[0] for x in funcs]))
print (fmt + len(N)*' %9d') % (('Name',)+tuple(N))
fmt1 = fmt + len(N)*' %9.3f'
for name,func in funcs:
print fmt1 % tuple([name]+[D[name][n] for n in N])
print
###############################
--
Robin Becker

Jan 12 '06 #18
Robin Becker schrieb:
# New attempts:
from itertools import imap
def flatten4(x, y):
'''D Murman'''
l = []
list(imap(l.extend, izip(x, y)))
return l
from Tkinter import _flatten
def flatten5(x, y):
'''D Murman'''
return list(_flatten(zip(x, y)))


well, i would really like to take credit for these, but they're
not mine ;) (credit goes to Michael Spencer). i especially like
flatten4, even if its not as fast as the phenomenally faster
flatten7.
--
David Murmann (NN!) ;)
Jan 12 '06 #19
Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
sum(...)
sum(sequence, start=0) -> value

If you're using sum() as a 1-level flatten you need to give it
start=[].


Except if you are trying to sum arrays of strings...
sum(["a","b","c"], "") Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]


I've no idea why this limitation is here... perhaps it is because pre
python2.4 calling += on strings was very slow?

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jan 12 '06 #20
Peter Otten wrote:
I you require len(xdata) == len(ydata) there's an easy way to move the loop
into C:

def flatten7():
n = len(xdata)
assert len(ydata) == n
result = [None] * (2*n)
result[::2] = xdata
result[1::2] = ydata
return result

$ python -m timeit 'from flatten import flatten6 as f' 'f()'
1000 loops, best of 3: 847 usec per loop
$ python -m timeit 'from flatten import flatten7 as f' 'f()'
10000 loops, best of 3: 43.9 usec per loop

Peter

Very nice observation, Peter. Easily the "winner". It reminds me of
str.translate beating python loops in filtering applications:
http://groups.google.com/group/comp....3cdc374144a4bd
What's more, you can generalize your approach to any number of sequences and
un-equal lengths, with only modest loss of speed:

def interleave(*args, **kw):
"""Peter Otten flatten7 (generalized by Michael Spencer)
Interleave any number of sequences, padding shorter sequences if kw pad
is supplied"""
dopad = "pad" in kw
pad = dopad and kw["pad"]
count = len(args)
lengths = map(len, args)
maxlen = max(lengths)
result = maxlen*count*[None]
for ix, input in enumerate(args):
try:
result[ix::count] = input
except ValueError:
if dopad:
result[ix::count] = input + [pad]*(maxlen-lengths[ix])
else:
raise
return result
def test_interleave():
assert interleave([1,3,5], [2,4,6]) == [1,2,3,4,5,6]
assert interleave([1,2,3]) == [1,2,3]
assert interleave(*[[1,2,3]]*10) == [1]*10+[2]*10+[3]*10
assert interleave(range(0,1000,2),range(1,1000,2)) == range(1000)
def raises(func,*args, **kw):
exc = kw.get("exc", Exception)
try:
func(*args)
except exc:
return True
assert raises(interleave,[1,2],[3,4,5])
assert interleave([1,3],[2,4,6], pad = None) == [1,2,3,4,None,6]
def timethem():
for n in [1000, 10000, 100000, 1000000]:
x = range(n); y = range(n)
print "List length: ",n
for func in [flatten7, interleave]:
print shell.timefunc(func, x, y)
timethem() List length: 1000
flatten7(...) 6442 iterations, 77.62usec per call
interleave(...) 5475 iterations, 91.33usec per call
List length: 10000
flatten7(...) 318 iterations, 1.57msec per call
interleave(...) 315 iterations, 1.59msec per call
List length: 100000
flatten7(...) 25 iterations, 20.61msec per call
interleave(...) 24 iterations, 21.06msec per call
List length: 1000000
flatten7(...) 3 iterations, 198.51msec per call
interleave(...) 3 iterations, 198.91msec per call

Michael

Jan 12 '06 #21

David Murmann wrote:
Robin Becker schrieb:
# New attempts:
from itertools import imap
def flatten4(x, y):
'''D Murman'''
l = []
list(imap(l.extend, izip(x, y)))
return l
from Tkinter import _flatten
def flatten5(x, y):
'''D Murman'''
return list(_flatten(zip(x, y)))


well, i would really like to take credit for these, but they're
not mine ;) (credit goes to Michael Spencer). i especially like
flatten4, even if its not as fast as the phenomenally faster
flatten7.

Me too. And expand a bit on flatten4, I got this interesting result.

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools; a=zip(xrange(1000),xrange(1000))" "l=len(a);
li=[None]*l*2;li[::2]=range(1000); li[1::2]=range(1000)"
1000 loops, best of 3: 318 usec per loop

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools,psyco; a=zip(xrange(1000),xrange(1000));li=[]"
"filter(li.extend,a)"
1000 loops, best of 3: 474 usec per loop

Still 50% slower but it has the advantage that it works on all kinds of
sequence as they can be efficiently izip() together.

Jan 13 '06 #22
Nick Craig-Wood <ni**@craig-wood.com> wrote:
Sion Arrowsmith <si***@chiark.greenend.org.uk> wrote:
sum(...)
sum(sequence, start=0) -> value

If you're using sum() as a 1-level flatten you need to give it
start=[].


Except if you are trying to sum arrays of strings...
>>> sum(["a","b","c"], "") Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead] >>>


I've no idea why this limitation is here... perhaps it is because pre
python2.4 calling += on strings was very slow?


No: when I implemented sum, I originally specialcased sum on strings to
map down to a ''.join -- Guido decided it was confusing and had no
advantage wrt calling ''.join directly so he made me put in that check.
Alex
Jan 13 '06 #23
bo****@gmail.com wrote:
David Murmann wrote:
> # New attempts:
> from itertools import imap
> def flatten4(x, y):
> '''D Murman'''
> l = []
> list(imap(l.extend, izip(x, y)))
> return l
well, i would really like to take credit for these, but they're
not mine ;) (credit goes to Michael Spencer). i especially like
flatten4, even if its not as fast as the phenomenally faster
flatten7.

Me too. And expand a bit on flatten4, I got this interesting result.

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools; a=zip(xrange(1000),xrange(1000))" "l=len(a);
li=[None]*l*2;li[::2]=range(1000); li[1::2]=range(1000)"
1000 loops, best of 3: 318 usec per loop

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools,psyco; a=zip(xrange(1000),xrange(1000));li=[]"
"filter(li.extend,a)"
1000 loops, best of 3: 474 usec per loop


For a fair comparison you'd have to time the zip operation.
Still 50% slower but it has the advantage that it works on all kinds of
sequence as they can be efficiently izip() together.


Creating a list via list/map/filter just for the side effect is not only bad
taste,

~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];ext=lst.extend'
'for i in a: ext(i)'
1000000 loops, best of 3: 1.23 usec per loop
~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];filter(lst.extend,
a)'
1000000 loops, best of 3: 1.63 usec per loop

it is also slower than an explicit loop. Don't do it.

Peter
Jan 13 '06 #24

Peter Otten wrote:
bo****@gmail.com wrote:
David Murmann wrote:

> # New attempts:
> from itertools import imap
> def flatten4(x, y):
> '''D Murman'''
> l = []
> list(imap(l.extend, izip(x, y)))
> return l well, i would really like to take credit for these, but they're
not mine ;) (credit goes to Michael Spencer). i especially like
flatten4, even if its not as fast as the phenomenally faster
flatten7.

Me too. And expand a bit on flatten4, I got this interesting result.

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools; a=zip(xrange(1000),xrange(1000))" "l=len(a);
li=[None]*l*2;li[::2]=range(1000); li[1::2]=range(1000)"
1000 loops, best of 3: 318 usec per loop

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools,psyco; a=zip(xrange(1000),xrange(1000));li=[]"
"filter(li.extend,a)"
1000 loops, best of 3: 474 usec per loop


For a fair comparison you'd have to time the zip operation.
Still 50% slower but it has the advantage that it works on all kinds of
sequence as they can be efficiently izip() together.


Creating a list via list/map/filter just for the side effect is not only bad
taste,

~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];ext=lst.extend'
'for i in a: ext(i)'
1000000 loops, best of 3: 1.23 usec per loop
~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];filter(lst.extend,
a)'
1000000 loops, best of 3: 1.63 usec per loop

it is also slower than an explicit loop. Don't do it.

ah, stand corrected.

Jan 13 '06 #25
Michael Spencer wrote:
Peter Otten wrote:
If you require len(xdata) == len(ydata) there's an easy way to move the
loop into C:

def flatten7():
n = len(xdata)
assert len(ydata) == n
result = [None] * (2*n)
result[::2] = xdata
result[1::2] = ydata
return result

$ python -m timeit 'from flatten import flatten6 as f' 'f()'
1000 loops, best of 3: 847 usec per loop
$ python -m timeit 'from flatten import flatten7 as f' 'f()'
10000 loops, best of 3: 43.9 usec per loop
Very nice observation, Peter.
Thank you :-)
Easily the "winner". It reminds me of
str.translate beating python loops in filtering applications:
http://groups.google.com/group/comp....3cdc374144a4bd What's more, you can generalize your approach to any number of sequences
and un-equal lengths, with only modest loss of speed:

def interleave(*args, **kw):
"""Peter Otten flatten7 (generalized by Michael Spencer)
Interleave any number of sequences, padding shorter sequences if kw
pad is supplied"""
dopad = "pad" in kw
pad = dopad and kw["pad"]
count = len(args)
lengths = map(len, args)
maxlen = max(lengths)
result = maxlen*count*[None]
for ix, input in enumerate(args):
try:
result[ix::count] = input
except ValueError:
if dopad:
result[ix::count] = input + [pad]*(maxlen-lengths[ix])
else:
raise
return result


You don't loose speed because the expensive operation is only executed if
list lengths differ. Two observations:

- list arithmetic like 'input + [pad]*(maxlen-lengths[ix])' should and can
be avoided

- You are padding twice -- once with None, and then with the real thing.

def interleave2(*args, **kw):
dopad = "pad" in kw
pad = kw.get("pad")
count = len(args)
lengths = map(len, args)
maxlen = max(lengths)
if not dopad and min(lengths) != maxlen:
raise ValueError
result = maxlen*count*[pad]
for ix, input in enumerate(args):
result[ix:len(input)*count:count] = input
return result

$ python -m timeit -s 'from interleave_spencer import xdata, ydata,
interleave as i;xdata=xdata[:-1]' 'i(xdata, ydata, pad=None)'
10000 loops, best of 3: 69.7 usec per loop
$ python -m timeit -s 'from interleave_spencer import xdata, ydata,
interleave2 as i;xdata=xdata[:-1]' 'i(xdata, ydata, pad=None)'
10000 loops, best of 3: 46.4 usec per loop

Not overwhelming, but I expect the difference to grow when the arguments
occupy a significant portion of the available memory.

Peter
Jan 13 '06 #26

Peter Otten wrote:
bo****@gmail.com wrote:
David Murmann wrote:

> # New attempts:
> from itertools import imap
> def flatten4(x, y):
> '''D Murman'''
> l = []
> list(imap(l.extend, izip(x, y)))
> return l well, i would really like to take credit for these, but they're
not mine ;) (credit goes to Michael Spencer). i especially like
flatten4, even if its not as fast as the phenomenally faster
flatten7.

Me too. And expand a bit on flatten4, I got this interesting result.

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools; a=zip(xrange(1000),xrange(1000))" "l=len(a);
li=[None]*l*2;li[::2]=range(1000); li[1::2]=range(1000)"
1000 loops, best of 3: 318 usec per loop

bonono@moresing:~/bonobo/psp$ python ~/lib/python2.4/timeit.py -s
"import itertools,psyco; a=zip(xrange(1000),xrange(1000));li=[]"
"filter(li.extend,a)"
1000 loops, best of 3: 474 usec per loop


For a fair comparison you'd have to time the zip operation.
Still 50% slower but it has the advantage that it works on all kinds of
sequence as they can be efficiently izip() together.


Creating a list via list/map/filter just for the side effect is not only bad
taste,

~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];ext=lst.extend'
'for i in a: ext(i)'
1000000 loops, best of 3: 1.23 usec per loop
~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];filter(lst.extend,
a)'
1000000 loops, best of 3: 1.63 usec per loop

it is also slower than an explicit loop. Don't do it.

Hi, but I found this result instead :

bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[None]*len(a)*2; e=l.extend" "l[::2]=b;l[1::2]=b"
100 loops, best of 3: 6.22 msec per loop
bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[]; e=l.extend" "filter(e,b)"
100 loops, best of 3: 7.25 msec per loop
bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[]; e=l.extend" "for x in b: e(x)"
100 loops, best of 3: 10.7 msec per loop
bonono@moresing:~$

So it seems to be faster than explicit loop. By localizing the l.extend
name binding, its speed is only 20% slower than the fastest method.
May be I have done something wrong in the test ?

Jan 13 '06 #27
bo****@gmail.com wrote:
Creating a list via list/map/filter just for the side effect is not only
bad taste,

~ $ python -m timeit -s'a = zip([range(1000)]*2)' 'lst=[];ext=lst.extend'
'for i in a: ext(i)'
1000000 loops, best of 3: 1.23 usec per loop
~ $ python -m timeit -s'a = zip([range(1000)]*2)'
'lst=[];filter(lst.extend, a)'
1000000 loops, best of 3: 1.63 usec per loop

it is also slower than an explicit loop. Don't do it.

Hi, but I found this result instead :

bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[None]*len(a)*2; e=l.extend" "l[::2]=b;l[1::2]=b"
100 loops, best of 3: 6.22 msec per loop
bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[]; e=l.extend" "filter(e,b)"
100 loops, best of 3: 7.25 msec per loop
bonono@moresing:~$ python ~/lib/python2.4/timeit.py -s "a=range(10000);
b=zip(*[a]*2); l=[]; e=l.extend" "for x in b: e(x)"
100 loops, best of 3: 10.7 msec per loop
bonono@moresing:~$

So it seems to be faster than explicit loop. By localizing the l.extend
name binding, its speed is only 20% slower than the fastest method.
May be I have done something wrong in the test ?


I hate to admit it but /my/ post had a bug:

zip([range(1000)]*2) should have been zip(*[range(1000)]*2).

filter() may be ugly, but faster it is.

Peter

Jan 13 '06 #28
Peter Otten wrote:
.......
- You are padding twice -- once with None, and then with the real thing.

def interleave2(*args, **kw):
dopad = "pad" in kw
pad = kw.get("pad")
count = len(args)
lengths = map(len, args)
maxlen = max(lengths)
if not dopad and min(lengths) != maxlen:
raise ValueError
result = maxlen*count*[pad]
for ix, input in enumerate(args):
result[ix:len(input)*count:count] = input
return result

$ python -m timeit -s 'from interleave_spencer import xdata, ydata,
interleave as i;xdata=xdata[:-1]' 'i(xdata, ydata, pad=None)'
10000 loops, best of 3: 69.7 usec per loop
$ python -m timeit -s 'from interleave_spencer import xdata, ydata,
interleave2 as i;xdata=xdata[:-1]' 'i(xdata, ydata, pad=None)'
10000 loops, best of 3: 46.4 usec per loop

Not overwhelming, but I expect the difference to grow when the arguments
occupy a significant portion of the available memory.

Peter

very nice indeed; another generalization is to allow truncation as well

def interleave(*args,**kw):
"""Peter Otten flatten7 (generalized by Michael Spencer and Robin Becker)
Interleave any number of sequences, padding shorter sequences if kw pad
is supplied or truncating if truncate=True is specified
interleave([1,3,5], [2,4,6]) == [1,2,3,4,5,6] True interleave([1,2,3]) == [1,2,3] True interleave(*[[1,2,3]]*10) == [1]*10+[2]*10+[3]*10 True interleave(range(0,1000,2),range(1,1000,2)) == range(1000) True interleave([1,2],[3,4,5]) Traceback (most recent call last):
...
ValueError: Minimum length=2 != Max length=3 interleave([1,3],[2,4,6], pad = None) == [1,2,3,4,None,6] True interleave([1,3],[2,4,6],truncate=True) == [1,2,3,4] True interleave([1,2],[3,4,5],pad='aaa',truncate=1)

Traceback (most recent call last):
...
AssertionError: Cannot specify both truncate=True and pad='aaa'
"""
dopad = "pad" in kw
pad = kw.get("pad")
dotrunc = bool(kw.get('truncate',False))
assert not (dotrunc and pad), \
'Cannot specify both truncate=True and pad=%r' % pad
count = len(args)
lengths = map(len,args)
maxlen = max(lengths)
minlen = min(lengths)
if dotrunc:
result = minlen*count*[None]
for ix, input in enumerate(args):
result[ix::count] = input[:minlen]
else:
if not dopad and minlen!=maxlen:
raise ValueError('Minimum length=%d != Max length=%d'
% (minlen,maxlen))
result = maxlen*count*[pad]
for ix, input in enumerate(args):
result[ix:len(input)*count:count] = input
return result

def _test():
import doctest, interleave
return doctest.testmod(interleave)

if __name__ == "__main__":
_test()

--
Robin Becker

Jan 13 '06 #29
Alex Martelli <al***@mail.comcast.net> wrote:
Nick Craig-Wood <ni**@craig-wood.com> wrote:
Except if you are trying to sum arrays of strings...
>>> sum(["a","b","c"], "")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]
>>>


I've no idea why this limitation is here... perhaps it is because pre
python2.4 calling += on strings was very slow?


No: when I implemented sum, I originally specialcased sum on strings to
map down to a ''.join -- Guido decided it was confusing and had no
advantage wrt calling ''.join directly so he made me put in that
check.


Hmm, interesting...

I agree with Guido about the special case, but I disagree about the
error message. Not being able to use sum(L,"") reduces the
orthogonality of sum for no good reason I could see.

However I only noticed that when trying to do the recent python
challenge which probaby isn't the best use case ;-)

--
Nick Craig-Wood <ni**@craig-wood.com> -- http://www.craig-wood.com/nick
Jan 13 '06 #30
Nick Craig-Wood <ni**@craig-wood.com> wrote:
...
I agree with Guido about the special case, but I disagree about the
error message. Not being able to use sum(L,"") reduces the
orthogonality of sum for no good reason I could see.


Having sum(L,'') work but be O(N squared) would be an "attractive
nuisance" within the meaning of the law; it's bad enough that sum(L,[])
etc are that way, but at least beginners want to concatenate lists (or
tuples, arrays, etc) much less often than they want to concatenate
strings. sum is meant to work on _numbers_ -- the reason it takes that
second optional parameter is essentially to be able to specify the
``type'' of numbers. In retrospect it might have been better to have a
single argument (or interpret multiple ones like min or max do, maybe),
forcing the starting value to be integer 0.
Alex
Jan 13 '06 #31
> Michael Spencer wrote:
result[ix::count] = input + [pad]*(maxlen-lengths[ix])

Peter Otten rewrote: result[ix:len(input)*count:count] = input


Quite so. What was I thinking?
Michael

Jan 13 '06 #32
On Fri, 13 Jan 2006 07:48:39 -0800, al***@mail.comcast.net (Alex Martelli) wrote:
Nick Craig-Wood <ni**@craig-wood.com> wrote:
...
I agree with Guido about the special case, but I disagree about the
error message. Not being able to use sum(L,"") reduces the
orthogonality of sum for no good reason I could see.

I agree.
Having sum(L,'') work but be O(N squared) would be an "attractive Are you saying ''.join is O(N squared)? Or that sum would not be
allowed to used the same/equivalent implementation algorithms if it were allowed
to work on strings?
nuisance" within the meaning of the law; it's bad enough that sum(L,[])
etc are that way, but at least beginners want to concatenate lists (or
tuples, arrays, etc) much less often than they want to concatenate
strings. sum is meant to work on _numbers_ -- the reason it takes that If sum is meant to work on _numbers_, why not call it numsum, to suggest
the restriction? ISTM the usual assumption in python is non-restrictive duck typing,
and the expectation would be no more restriction than in reduce(operator.add, L, init_val).
second optional parameter is essentially to be able to specify the
``type'' of numbers. In retrospect it might have been better to have a
single argument (or interpret multiple ones like min or max do, maybe),
forcing the starting value to be integer 0.

So it was an accident that user types are permitted?
class C: ... def __getattr__(self, attr): print attr;raise AttributeError
... sum([C(),C()]) __coerce__
__radd__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for +: 'int' and 'instance' sum([C(),C()],C()) __coerce__
__add__
__coerce__
__radd__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for +: 'instance' and 'instance'

Yet user subclassing of str does not yield an acceptable user type?
S = type('S',(str,),{})
sum(S(i) for i in xrange(5)) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: unsupported operand type(s) for +: 'int' and 'S' sum(S(i) for i in xrange(5), S('start:')) File "<stdin>", line 1
SyntaxError: invalid syntax sum((S(i) for i in xrange(5)), S('start:')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]

Although, notice
class SI(str): ... def __add__(self, other): return SI(str.__add__(self, str(other)))
... __radd__ = __add__
... sum((SI(i) for i in xrange(5))) '001234'

But: sum((SI(i) for i in xrange(5)), SI('start:')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sum() can't sum strings [use ''.join(seq) instead]
vs reduce(operator.__add__, (SI(i) for i in xrange(5)), SI('start:'))

'start:01234'
Which seems to boil down to incomplete restrictions on duck typing.
Maybe Guido will want to complete it, but ISTM your original implementation
delegating string addition implementation to ''.join was reasonable.

Regards,
Bengt Richter
Jan 13 '06 #33

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

23
by: Francis Avila | last post by:
Below is an implementation a 'flattening' recursive generator (take a nested iterator and remove all its nesting). Is this possibly general and useful enough to be included in itertools? (I know...
10
by: bearophile | last post by:
This is my first Python program (it's an improvement of a recursive version by Luther Blissett). Given a list like this: , ]]] It produces the "flatted" version: I think this operation is...
3
by: Bengt Richter | last post by:
What am I missing? (this is from 2.4b1, so probably it has been fixed?) def flatten(list): l = for elt in list: ^^^^--must be expecting list instance or other sequence t = type(elt) if t...
18
by: Ville Vainio | last post by:
For quick-and-dirty stuff, it's often convenient to flatten a sequence (which perl does, surprise surprise, by default): ]]] -> One such implementation is at ...
181
by: Tom Anderson | last post by:
Comrades, During our current discussion of the fate of functional constructs in python, someone brought up Guido's bull on the matter: http://www.artima.com/weblogs/viewpost.jsp?thread=98196 ...
8
by: per9000 | last post by:
Hi all, I have a two-dimensional array of data, f.x int's. We can imagine that the array is "really large". Now I want the data in it and store this in a one-dimensional array. The obvious...
3
by: Sergio Correia | last post by:
Hi, I'm looking for an easy way to flatten a two level list like this spam = , , , ] Into something like eggs = There are *no* special cases (no empty sub-lists).
25
by: beginner | last post by:
Hi, I am wondering how do I 'flatten' a list or a tuple? For example, I'd like to transform or ] to . Another question is how do I pass a tuple or list of all the aurgements of a function to...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.