getting n items at a time from a generator

[Kugutsumen]

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
"""generate N items from a generator."""
chunk = []
count = 0
while True:
try:
item = items.next()
count += 1
except StopIteration:
yield chunk
break
chunk.append(item)
if not (count % size):
yield chunk
chunk = []
count = 0

>>t = (i for i in range(30))
c = chunk(7, t)
for i in c:

.... print i
....
[0, 1, 2, 3, 4, 5, 6]
[7, 8, 9, 10, 11, 12, 13]
[14, 15, 16, 17, 18, 19, 20]
[21, 22, 23, 24, 25, 26, 27]
[28, 29]

In my real world project, I have over 250 million items that are too
big to fit in memory and that processed and later used to update
records in a database... to minimize disk IO, I found it was more
efficient to process them by batch or "chunk" of 50,000 or so. Hence

Is this the proper way to do this?

[Paul Hankin]

On Dec 27, 11:34*am, Kugutsumen <[e-mail]:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
* * """generate N items from a generator."""
* * chunk = []
* * count = 0
* * while True:
* * * * try:
* * * * * * item = items.next()
* * * * * * count += 1
* * * * except StopIteration:
* * * * * * yield chunk
* * * * * * break
* * * * chunk.append(item)
* * * * if not (count % size):
* * * * * * yield chunk
* * * * * * chunk = []
* * * * * * count = 0

The itertools module is always a good place to look when you've got a
complicated generator.

import itertools
import operator

def chunk(N, items):
"Group items in chunks of N"
def clump((n, _)):
return n // N
for _, group in itertools.groupby(enumerate(items), clump):
yield itertools.imap(operator.itemgetter(1), group)

for ch in chunk(7, range(30)):
print list(ch)
I've changed chunk to return a generator rather than building a list
which is probably only going to be iterated over. But if you prefer
the list version, replace 'itertools.imap' with 'map'.

--
Paul Hankin

[Kugutsumen]

On Dec 27, 7:07*pm, Paul Hankin <[e-mail]:

On Dec 27, 11:34*am, Kugutsumen <[e-mail]:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
* * """generate N items from a generator."""
* * chunk = []
* * count = 0
* * while True:
* * * * try:
* * * * * * item = items.next()
* * * * * * count += 1
* * * * except StopIteration:
* * * * * * yield chunk
* * * * * * break
* * * * chunk.append(item)
* * * * if not (count % size):
* * * * * * yield chunk
* * * * * * chunk = []
* * * * * * count = 0

The itertools module is always a good place to look when you've got a
complicated generator.

import itertools
import operator

def chunk(N, items):
* * "Group items in chunks of N"
* * def clump((n, _)):
* * * * return n // N
* * for _, group in itertools.groupby(enumerate(items), clump):
* * * * yield itertools.imap(operator.itemgetter(1), group)

for ch in chunk(7, range(30)):
* * print list(ch)

I've changed chunk to return a generator rather than building a list
which is probably only going to be iterated over. But if you prefer
the list version, replace 'itertools.imap' with 'map'.

--
Paul Hankin

Thanks, I am going to take a look at itertools.
I prefer the list version since I need to buffer that chunk in memory
at this point.

[Terry Jones]

>>>>"Kugutsumen" == Kugutsumen <ku********@gmail.comwrites:
KugutsumenOn Dec 27, 7:07*pm, Paul Hankin <[e-mail]:

>On Dec 27, 11:34*am, Kugutsumen <[e-mail]:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

KugutsumenThanks, I am going to take a look at itertools. I prefer the
Kugutsumenlist version since I need to buffer that chunk in memory at
Kugutsumenthis point.

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

def chop(iterable, length=2):
return izip(*(iter(iterable),) * length)

Terry

[Kugutsumen]

On Dec 27, 7:24*pm, Terry Jones <[e-mail]:

>>>"Kugutsumen" == Kugutsumen *<[e-mail]:

KugutsumenOn Dec 27, 7:07*pm, Paul Hankin <[e-mail]:

On Dec 27, 11:34*am, Kugutsumen <[e-mail]:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

KugutsumenThanks, I am going to take a look at itertools. *I prefer the
Kugutsumenlist version since I need to buffer that chunk in memory at
Kugutsumenthis point.

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

* * def chop(iterable, length=2):
* * * * return izip(*(iter(iterable),) * length)

Terry

Thanks Terry,

However, chop ignores the remainder of the data in the example.

>>t = (i for i in range(30))
c =chop (t, 7)
for ch in c:

... print ch
...
(0, 1, 2, 3, 4, 5, 6)
(7, 8, 9, 10, 11, 12, 13)
(14, 15, 16, 17, 18, 19, 20)
(21, 22, 23, 24, 25, 26, 27)

k

[Tim Roberts]

Kugutsumen <ku********@gmail.comwrote:

>
I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.

[Raymond Hettinger]

* * def chop(iterable, length=2):

* * * * return izip(*(iter(iterable),) * length)

Is this *always* guaranteed by the language to work?

Yes!

Users requested this guarantee, and I agreed. The docs now explicitly
guarantee this behavior.
Raymond

[Raymond Hettinger]

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

>

* * def chop(iterable, length=2):
* * * * return izip(*(iter(iterable),) * length)

However, chop ignores the remainder of the data in the example.

There is a recipe in the itertools docs which handles the odd-length
data at the end:

def grouper(n, iterable, padvalue=None):
"grouper(3, 'abcdefg', 'x') --('a','b','c'), ('d','e','f'),
('g','x','x')"
return izip(*[chain(iterable, repeat(padvalue, n-1))]*n)
Raymond

[Paul Rubin]

Tim Roberts <ti**@probo.comwrites:

I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?

I'm in favor.

[Shane Geiger]

Paul Rubin wrote:

Tim Roberts <ti**@probo.comwrites:

>I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?

I'm in favor.

I am ecstatic about the idea of getting n items at a time from a
generator! This would eliminate the use of less elegant functions to do
this sort of thing which I would do even more frequently if it were
easier.

Is it possible that this syntax for generator expressions could be adopted?

>>sentence = 'this is a senTence WiTH'
generator = (word.capitalize() for word in sentence.split())
print generator.next(3,'PadValue')

('This','Is','A')

>>print generator.next(3,'PadValue')

('Sentence','With','PadValue')

>>generator.next(3,'PadValue')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

>>>

While on the topic of generators:

Something else I have longed for is assignment within a while loop. (I
realize this might be more controversial and might have been avoided on
purpose, but I wasn't around for that discussion.)

>>sentence = 'this is a senTence WiTH'
generator = (word.capitalize() for word in sentence.split())
while a,b,c = generator.next(3,'PadValue'):

.... print a,b,c
....
This Is A
Sentence With PadValue

>>>

--
Shane Geiger
IT Director
National Council on Economic Education
sg*****@ncee.net | 402-438-8958 | http://www.ncee.net

Leading the Campaign for Economic and Financial Literacy