By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
438,427 Members | 1,354 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 438,427 IT Pros & Developers. It's quick & easy.

Efficiently iterating over part of a list

P: n/a
If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]
--
Steven D'Aprano

Oct 13 '06 #1
Share this Question
Share on Google+
6 Replies


P: n/a
Steven D'Aprano wrote:
If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]

I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)
Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James
James
Oct 13 '06 #2

P: n/a
James Stroud wrote:
Steven D'Aprano wrote:
>If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do
people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]


I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)
Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James
James
I mean

for i,item in enumerate(alist):
Oct 13 '06 #3

P: n/a
Steven D'Aprano wrote:
[snip]
The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]

for x in itertools.islice(alist, 1, len(alist)):
HTH
Ziga

Oct 13 '06 #4

P: n/a
Steven D'Aprano wrote:
Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x*=*alist[i]
For small start values you can use itertools.islice(), e. g:

for x in islice(alist, 1, None):
# use x

You'd have to time at what point the C-like idiom (which I would have no
qualms using throughout) becomes faster.

Peter

Oct 13 '06 #5

P: n/a
Steven D'Aprano <st***@REMOVEME.cybersource.com.auwrote:
The important thing to notice is that alist[1:] makes a copy. What if
the list has millions of items and duplicating it is expensive? What
do people do in that case?
I think you are worrying prematurely.

On my system slicing one element off the front of a 10,000,000 element list
takes 440mS. The same operation on 1,000,000 elements taks 41mS. Iterating
through the sliced list:

for x in r[1:]:
y = x+1

takes 1.8s and 157mS respectively, so the slicing is only a quarter of the
time for even this minimal loop. As soon as you do anything much inside the
loop you can forget the slice cost.

Remember that copying the list never copies the elements in the list, it
just copies pointers and bumps ref counts. Copying a list even if it has
millions of items is not usually expensive compared with the costs of
manipulating all the items in the list.

So the first thing you do is not to worry about this until you know it is
an issue. Once you know for a fact that it is a problem, then you can look
at optimising it with fancy lazy slicing techniques, but not before.
Oct 13 '06 #6

P: n/a
Steven D'Aprano <st***@REMOVEME.cybersource.com.auwrites:
for i in range(1, len(alist)):
x = alist[i]
a2 = iter(alist)
a2.next() # throw away first element
for x in a2:
...
Oct 13 '06 #7

This discussion thread is closed

Replies have been disabled for this discussion.