By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,908 Members | 1,850 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,908 IT Pros & Developers. It's quick & easy.

Python Generators

P: n/a
mpc
HI all,
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

#!/usr/bin/env python
import os,struct,time
def generate_headers(cachedir):
for name, blocksize in cachefiles:
pathname = os.path.join(cachedir,name)
f = open(pathname,"rb")
f.seek(4096)
while True:
header = f.read(36)
if not header: break
fields = struct.unpack(">9I",header)
if fields[0] == 0x00010008: yield f, fields
fp = f.tell()
offset = fp % blocksize
if offset:
f.seek(blocksize - offset,1)
f.close()
def concatenate(sequences):
for seq in sequences:
for item in seq:
yield item

all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
cachefiles = [('_CACHE_001_',256),('_CACHE_002_',1024),('_CACHE_ 003_',
4096)]
n = 0
while True:
n += 1
time.sleep(0.5)
headers = concatenate(generate_headers(path) for path in
all_caches)
for h in headers:
print h,n
# this doesn't work either

while True:
n += 1
time.sleep(0.5)
for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n
# however if I hard code the path
path = "FFCache"
while True:
n += 1
headers = generate_headers(path)
for h in headers:
print h,n
#but of course i do not wish to hard code the path.
Mar 15 '08 #1
Share this Question
Share on Google+
4 Replies


P: n/a
On 15 Mar, 21:35, mpc <mcoh...@gmail.comwrote:

generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?

I am not sure what you are trying to do, but it seems a bit confused.

>>def concat(seq):
for s in seq: yield s
>>seq = xrange(3)
>>for n in xrange(5):
h = concat(s for s in seq)
for i in h: print i,n
0 0
1 0
2 0
0 1
1 1
2 1
0 2
1 2
2 2
0 3
1 3
2 3
0 4
1 4
2 4

Generators work they way they should. Even when one is used as
argument for another.


Mar 15 '08 #2

P: n/a
mpc wrote:
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?
A generator or generator expression does indeed only run once.
>>for i in range(3):
.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:
>>for i in range(3):
.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.
while True:
n += 1
time.sleep(0.5)
all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n
(Untested, because you didn't bother to provide a self-contained example)

Peter
Mar 15 '08 #3

P: n/a
mpc wrote:
I am trying to write a while loop that will iterate over generators to
capture all the headers of FFCache directories. However, the
generators embedded within the argument of another generator do not
seem to re-initiate. the example below loops through and initiates the
generator embedded in the argument only once. Can anyone explain while
the generator will not re-initiate, and suggest a simple fix?
A generator or generator expression does indeed only run once.
>>gen = (i for i in range(5) if i%2)
for i in range(3):
.... print "---", i, "---"
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---

--- 2 ---

The fix is to use a list or list comprehension, or make a new generator
every time you need one:
>>for i in range(3):
.... print "---", i, "---"
.... gen = (i for i in range(5) if i%2)
.... for k in gen: print k,
.... print
....
--- 0 ---
1 3
--- 1 ---
1 3
--- 2 ---
1 3

At first glance I would guess that in your case all_caches is the culprit
that has to be moved into the 'while True: ...' loop.
while True:
n += 1
time.sleep(0.5)
all_caches = (path for path,dirlist,filelist in os.walk("/Users/") if
'_CACHE_MAP_' in filelist)
for path in all_caches:
headers = generate_headers(path)
for h in headers:
print h,n
(Untested, because you didn't bother to provide a self-contained example)

Peter
Mar 15 '08 #4

P: n/a
mpc wrote:

<snip>
def concatenate(sequences):
for seq in sequences:
for item in seq:
yield item
You should check out itertools.chain(). It does this. You call it like
"chain(seq1, seq2, ...)" instead of "chain(sequences)" though, which may
be a problem for you.

The rest of itertools might be interesting too:

<http://docs.python.org/lib/module-itertools.html>

<snip>
--
Mar 16 '08 #5

This discussion thread is closed

Replies have been disabled for this discussion.