473,322 Members | 1,398 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Efficiently iterating over part of a list

If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]
--
Steven D'Aprano

Oct 13 '06 #1
6 7457
Steven D'Aprano wrote:
If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]

I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)
Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James
James
Oct 13 '06 #2
James Stroud wrote:
Steven D'Aprano wrote:
>If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do
people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]


I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)
Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James
James
I mean

for i,item in enumerate(alist):
Oct 13 '06 #3
Steven D'Aprano wrote:
[snip]
The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist[i]

for x in itertools.islice(alist, 1, len(alist)):
HTH
Ziga

Oct 13 '06 #4
Steven D'Aprano wrote:
Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x*=*alist[i]
For small start values you can use itertools.islice(), e. g:

for x in islice(alist, 1, None):
# use x

You'd have to time at what point the C-like idiom (which I would have no
qualms using throughout) becomes faster.

Peter

Oct 13 '06 #5
Steven D'Aprano <st***@REMOVEME.cybersource.com.auwrote:
The important thing to notice is that alist[1:] makes a copy. What if
the list has millions of items and duplicating it is expensive? What
do people do in that case?
I think you are worrying prematurely.

On my system slicing one element off the front of a 10,000,000 element list
takes 440mS. The same operation on 1,000,000 elements taks 41mS. Iterating
through the sliced list:

for x in r[1:]:
y = x+1

takes 1.8s and 157mS respectively, so the slicing is only a quarter of the
time for even this minimal loop. As soon as you do anything much inside the
loop you can forget the slice cost.

Remember that copying the list never copies the elements in the list, it
just copies pointers and bumps ref counts. Copying a list even if it has
millions of items is not usually expensive compared with the costs of
manipulating all the items in the list.

So the first thing you do is not to worry about this until you know it is
an issue. Once you know for a fact that it is a problem, then you can look
at optimising it with fancy lazy slicing techniques, but not before.
Oct 13 '06 #6
Steven D'Aprano <st***@REMOVEME.cybersource.com.auwrites:
for i in range(1, len(alist)):
x = alist[i]
a2 = iter(alist)
a2.next() # throw away first element
for x in a2:
...
Oct 13 '06 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Bart Nessux | last post by:
ip_list = inputFile = file('ips.txt', 'r') ip_list.append(inputFile.read()) inputFile.close() for i in ip_list: print "/sbin/ifconfig %s netmask 255.255.252.0 broadcast 128.173.123.255 up" %i...
7
by: Dave Hansen | last post by:
OK, first, I don't often have the time to read this group, so apologies if this is a FAQ, though I couldn't find anything at python.org. Second, this isn't my code. I wouldn't do this. But a...
6
by: Gustaf Liljegren | last post by:
I ran into this problem today: I got an array with Account objects. I need to iterate through this array to supplement the accounts in the array with more data. But the compiler complains when I...
4
RMWChaos
by: RMWChaos | last post by:
The next episode in the continuing saga of trying to develop a modular, automated DOM create and remove script asks the question, "Where should I put this code?" Alright, here's the story: with a...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.