getting n items at a time from a generator

[Kugutsumen]

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
"""generate N items from a generator."""
chunk = []
count = 0
while True:
try:
item = items.next()
count += 1
except StopIteration:
yield chunk
break
chunk.append(it em)
if not (count % size):
yield chunk
chunk = []
count = 0

>>t = (i for i in range(30))
c = chunk(7, t)
for i in c:

.... print i
....
[0, 1, 2, 3, 4, 5, 6]
[7, 8, 9, 10, 11, 12, 13]
[14, 15, 16, 17, 18, 19, 20]
[21, 22, 23, 24, 25, 26, 27]
[28, 29]

In my real world project, I have over 250 million items that are too
big to fit in memory and that processed and later used to update
records in a database... to minimize disk IO, I found it was more
efficient to process them by batch or "chunk" of 50,000 or so. Hence

Is this the proper way to do this?

[Paul Hankin]

On Dec 27, 11:34*am, Kugutsumen <kugutsu...@gma il.comwrote:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
* * """generate N items from a generator."""
* * chunk = []
* * count = 0
* * while True:
* * * * try:
* * * * * * item = items.next()
* * * * * * count += 1
* * * * except StopIteration:
* * * * * * yield chunk
* * * * * * break
* * * * chunk.append(it em)
* * * * if not (count % size):
* * * * * * yield chunk
* * * * * * chunk = []
* * * * * * count = 0

The itertools module is always a good place to look when you've got a
complicated generator.

import itertools
import operator

def chunk(N, items):
"Group items in chunks of N"
def clump((n, _)):
return n // N
for _, group in itertools.group by(enumerate(it ems), clump):
yield itertools.imap( operator.itemge tter(1), group)

for ch in chunk(7, range(30)):
print list(ch)
I've changed chunk to return a generator rather than building a list
which is probably only going to be iterated over. But if you prefer
the list version, replace 'itertools.imap ' with 'map'.

--
Paul Hankin

[Kugutsumen]

On Dec 27, 7:07*pm, Paul Hankin <paul.han...@gm ail.comwrote:

On Dec 27, 11:34*am, Kugutsumen <kugutsu...@gma il.comwrote:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

def chunk(size, items):
* * """generate N items from a generator."""
* * chunk = []
* * count = 0
* * while True:
* * * * try:
* * * * * * item = items.next()
* * * * * * count += 1
* * * * except StopIteration:
* * * * * * yield chunk
* * * * * * break
* * * * chunk.append(it em)
* * * * if not (count % size):
* * * * * * yield chunk
* * * * * * chunk = []
* * * * * * count = 0

The itertools module is always a good place to look when you've got a
complicated generator.

import itertools
import operator

def chunk(N, items):
* * "Group items in chunks of N"
* * def clump((n, _)):
* * * * return n // N
* * for _, group in itertools.group by(enumerate(it ems), clump):
* * * * yield itertools.imap( operator.itemge tter(1), group)

for ch in chunk(7, range(30)):
* * print list(ch)

I've changed chunk to return a generator rather than building a list
which is probably only going to be iterated over. But if you prefer
the list version, replace 'itertools.imap ' with 'map'.

--
Paul Hankin

Thanks, I am going to take a look at itertools.
I prefer the list version since I need to buffer that chunk in memory
at this point.

[Terry Jones]

>>>>"Kugutsumen " == Kugutsumen <ku********@gma il.comwrites:
KugutsumenOn Dec 27, 7:07*pm, Paul Hankin <paul.han...@gm ail.comwrote:

>On Dec 27, 11:34*am, Kugutsumen <kugutsu...@gma il.comwrote:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

KugutsumenThank s, I am going to take a look at itertools. I prefer the
Kugutsumenlist version since I need to buffer that chunk in memory at
Kugutsumenthis point.

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

def chop(iterable, length=2):
return izip(*(iter(ite rable),) * length)

Terry

[Kugutsumen]

On Dec 27, 7:24*pm, Terry Jones <[e-mail] ote:

>>>"Kugutsume n" == Kugutsumen *<kugutsu...@gm ail.comwrites:

KugutsumenOn Dec 27, 7:07*pm, Paul Hankin <paul.han...@gm ail.comwrote:

On Dec 27, 11:34*am, Kugutsumen <kugutsu...@gma il.comwrote:

I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

KugutsumenThank s, I am going to take a look at itertools. *I prefer the
Kugutsumenlist version since I need to buffer that chunk in memory at
Kugutsumenthis point.

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

* * def chop(iterable, length=2):
* * * * return izip(*(iter(ite rable),) * length)

Terry

Thanks Terry,

However, chop ignores the remainder of the data in the example.

>>t = (i for i in range(30))
c =chop (t, 7)
for ch in c:

... print ch
...
(0, 1, 2, 3, 4, 5, 6)
(7, 8, 9, 10, 11, 12, 13)
(14, 15, 16, 17, 18, 19, 20)
(21, 22, 23, 24, 25, 26, 27)

k

[Tim Roberts]

Kugutsumen <ku********@gma il.comwrote:

>
I am relatively new the python language and I am afraid to be missing
some clever construct or built-in way equivalent to my 'chunk'
generator below.

I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.

[Raymond Hettinger]

* * def chop(iterable, length=2):

* * * * return izip(*(iter(ite rable),) * length)

Is this *always* guaranteed by the language to work?

Yes!

Users requested this guarantee, and I agreed. The docs now explicitly
guarantee this behavior.
Raymond

[Raymond Hettinger]

Also consider this solution from O'Reilly's Python Cookbook (2nd Ed.) p705

>

* * def chop(iterable, length=2):
* * * * return izip(*(iter(ite rable),) * length)

However, chop ignores the remainder of the data in the example.

There is a recipe in the itertools docs which handles the odd-length
data at the end:

def grouper(n, iterable, padvalue=None):
"grouper(3, 'abcdefg', 'x') --('a','b','c'), ('d','e','f'),
('g','x','x')"
return izip(*[chain(iterable, repeat(padvalue , n-1))]*n)
Raymond

[Paul Rubin]

Tim Roberts <ti**@probo.com writes:

I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?

I'm in favor.

[Shane Geiger]

Paul Rubin wrote:

Tim Roberts <ti**@probo.com writes:

>I have to say that I have found this to be a surprisingly common need as
well. Would this be an appropriate construct to add to itertools?

I'm in favor.

I am ecstatic about the idea of getting n items at a time from a
generator! This would eliminate the use of less elegant functions to do
this sort of thing which I would do even more frequently if it were
easier.

Is it possible that this syntax for generator expressions could be adopted?

>>sentence = 'this is a senTence WiTH'
generator = (word.capitaliz e() for word in sentence.split( ))
print generator.next( 3,'PadValue')

('This','Is','A ')

>>print generator.next( 3,'PadValue')

('Sentence','Wi th','PadValue')

>>generator.nex t(3,'PadValue')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

>>>

While on the topic of generators:

Something else I have longed for is assignment within a while loop. (I
realize this might be more controversial and might have been avoided on
purpose, but I wasn't around for that discussion.)

>>sentence = 'this is a senTence WiTH'
generator = (word.capitaliz e() for word in sentence.split( ))
while a,b,c = generator.next( 3,'PadValue'):

.... print a,b,c
....
This Is A
Sentence With PadValue

>>>

--
Shane Geiger
IT Director
National Council on Economic Education
sg*****@ncee.ne t | 402-438-8958 | http://www.ncee.net

Leading the Campaign for Economic and Financial Literacy