469,359 Members | 1,606 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,359 developers. It's quick & easy.

breaking a list into smaller lists

I understand how a Python list can be broken into smaller lists by using
slice like this:

new_small_list_1 = BIG_LIST[0:1001] #0-1000
new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
....
....
....

However, I was wondering if there's an easier way to do this. For
example, say I have a list that contains the names of 100,000 files that
I wish to open and read. Because of file handle limits, I can only open
roughly 1000 files at a time while the OS is doing other things.

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart
Jul 18 '05 #1
7 3374
Bart Nessux wrote:
Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart


big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]

replace 10 with any value you need, but it is better not to keep all 100 000
filenames in memory if it isn't necessary

--
Maciej "Fiedzia" Dziardziel (fiedzia (at) fiedzia (dot) prv (dot) pl)
www.fiedzia.prv.pl
Jul 18 '05 #2
Maciej Dziardziel wrote:
Bart Nessux wrote:

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart

big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]

replace 10 with any value you need, but it is better not to keep all 100 000
filenames in memory if it isn't necessary


Thank you for this example. It works very well.
Jul 18 '05 #3
Maciej Dziardziel wrote:
Bart Nessux wrote:
Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart


big = range(87)

print [ big[i*10:i*10+10] for i in xrange(len(big)/10+int((len(big) %
10)>0) ) ]


I would add that you should definitely use the // operator in this case
as this code will break in 2.4.

Also, I would leave the modulo part out and write the thing as follows:

print [ big[i*10:(i+1)*10] for i in xrange((len(big)-1) // 10 + 1) ]

Reinhold

--
Wenn eine Linuxdistribution so wenig brauchbare Software wie Windows
mitbrächte, wäre das bedauerlich. Was bei Windows der Umfang eines
"kompletten Betriebssystems" ist, nennt man bei Linux eine Rescuedisk.
-- David Kastrup in de.comp.os.unix.linux.misc
Jul 18 '05 #4
I would add that you should definitely use the // operator in this case
as this code will break in 2.4.

2.4? It seems that in PEP: 238, it will break only in 3.0
Does anyone can confirm ?

" - Classic division will remain the default in the Python 2.x
series; true division will be standard in Python 3.0. "

Xavier

Jul 18 '05 #5
Xavier Combelle wrote:
I would add that you should definitely use the // operator in this case
as this code will break in 2.4.

2.4? It seems that in PEP: 238, it will break only in 3.0
Does anyone can confirm ?

" - Classic division will remain the default in the Python 2.x
series; true division will be standard in Python 3.0. "


Sorry, it's my fault, I didn't recall it properly.

Reinhold

--
Wenn eine Linuxdistribution so wenig brauchbare Software wie Windows
mitbrächte, wäre das bedauerlich. Was bei Windows der Umfang eines
"kompletten Betriebssystems" ist, nennt man bei Linux eine Rescuedisk.
-- David Kastrup in de.comp.os.unix.linux.misc
Jul 18 '05 #6
One could easilly break it into a list of small lists:

small = [ BIG_LIST[i:i+1000] for i in xrange(0,len(BIG_LIST),1000) ]

On Wed, Jun 16, 2004 at 07:04:10PM -0400, Bart Nessux wrote:
I understand how a Python list can be broken into smaller lists by using
slice like this:

new_small_list_1 = BIG_LIST[0:1001] #0-1000
new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
...
...
...

However, I was wondering if there's an easier way to do this. For
example, say I have a list that contains the names of 100,000 files that
I wish to open and read. Because of file handle limits, I can only open
roughly 1000 files at a time while the OS is doing other things.

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?

Thanks,
Bart


Jul 18 '05 #7

"Bart Nessux" <ba*********@hotmail.com> wrote in message
news:ca**********@solaris.cc.vt.edu...
I understand how a Python list can be broken into smaller lists by using
slice like this:

new_small_list_1 = BIG_LIST[0:1001] #0-1000
this is 1001 items: you probably want 0:1000
new_small_list_2 = BIG_LIST[1001:2001] #1001-2000
new_small_list_3 = BIG_LIST[2001:3001] #2001-3000
these are 1000 items, but with off-by-one correction, 1000:2000, etc

However, I was wondering if there's an easier way to do this. For
example, say I have a list that contains the names of 100,000 files that
I wish to open and read. Because of file handle limits, I can only open
roughly 1000 files at a time while the OS is doing other things.

Is there a way to turn this big list into 100 small lists (each
containing 1000 files) with one simple line of code?


using *your* pattern, corrected, but untested and written while slightly
tired:
[biglist[i:i+1000] for i in range(len(biglist)/1000)]

Terry J. Reedy


Jul 18 '05 #8

This discussion thread is closed

Replies have been disabled for this discussion.

By using this site, you agree to our Privacy Policy and Terms of Use.