It's easy to write a function that wraps a generator and provides a
callback. E. g.:
import os, sys
def walk(path, visit):
""" Emulate os.path.walk() (simplified) using os.walk()"""
for dir, folders, files in os.walk(path):
visit(dir, folders + files)
if __name__ == "__main__":
walk(".", lambda d, f: sys.stdout.write(d + "\n"))
However, I did not succeed in turning the old os.path.walk(), i. e. a
function taking a callback, into a generator. Is there a general way to do
it without having to store all intermediate results first?
Peter
PS. No, I don't have a use case. Threads welcome if all else fails :-) 6 1796
Hi, However, I did not succeed in turning the old os.path.walk(), i. e. a function taking a callback, into a generator. Is there a general way to do it without having to store all intermediate results first?
This works for me:
import os.path
def path_gen(start):
res = []
def cb(r, dir, names):
for n in names:
r.append(n)
os.path.walk(start, cb, res)
for n in res:
yield n
g = path_gen("/etc")
for n in g:
print n
Diez
> def path_gen(start): res = [] def cb(r, dir, names): for n in names: r.append(n)
os.path.walk(start, cb, res) for n in res: yield n
g = path_gen("/etc")
for n in g: print n
Just found out that lists support extend, which allows the ugly loop for
appending names in cb to be written this way:
r.extend(names)
Diez
Diez B. Roggisch wrote: However, I did not succeed in turning the old os.path.walk(), i. e. a function taking a callback, into a generator. Is there a general way to do it without having to store all intermediate results first? This works for me:
import os.path
def path_gen(start): res = [] def cb(r, dir, names): for n in names: r.append(n)
os.path.walk(start, cb, res)
At this point, you have stored all results in res, and thus did not play by
the rules :-)
for n in res: yield n
g = path_gen("/etc")
for n in g: print n
Both os.path.walk() and os.walk() basically need memory for the names in
*one* directory; when you take os.walk() to model os.path.walk() that
doesn't change, but the other way round - bang, memory usage explosion.
This is a strange asymmetry, and I was wondering if I've overlooked a simple
way to transform a callback into a generator, but I now tend to the
assumption that you *need* threads: One thread with the callback puts names
or name lists into the queue until it's full (some arbitrary limit that
also determines the total memory needed), another thread (the generator)
consumes names from the queue.
Peter
On Wed, 03 Dec 2003 18:48:41 +0100, Peter Otten <__*******@web.de> wrote: It's easy to write a function that wraps a generator and provides a callback. E. g.:
import os, sys
def walk(path, visit): """ Emulate os.path.walk() (simplified) using os.walk()""" for dir, folders, files in os.walk(path): visit(dir, folders + files)
if __name__ == "__main__": walk(".", lambda d, f: sys.stdout.write(d + "\n"))
However, I did not succeed in turning the old os.path.walk(), i. e. a function taking a callback, into a generator. Is there a general way to do it without having to store all intermediate results first?
Peter
PS. No, I don't have a use case. Threads welcome if all else fails :-)
I suspect that's what is necessary currently, until we get a yield that can suspend a
whole stack of frames at a yield inside nested calls to functions. Then it would
just be a matter of putting a yield in a callback routine and starting the
walk from the base generator-making function/method/whatever.
Maybe deep generators could be created via an __iter__ method of the function type
as an alternative/extension to has-yield-in-function-code-body magic.
Exit from the generator wouldn't happen until you exited the base frame.
Yield in a nested function call would just suspend right there.
Easier said than done, of course ;-)
Regards,
Bengt Richter
Hi, This is a strange asymmetry, and I was wondering if I've overlooked a simple way to transform a callback into a generator, but I now tend to the assumption that you *need* threads: One thread with the callback puts names or name lists into the queue until it's full (some arbitrary limit that also determines the total memory needed), another thread (the generator) consumes names from the queue.
I also thought about that - its the only thing that allows for real lazyness
- you can wait in the callback until the generator clears a semaphore. But
if the context-changes are worth the effort is questionable.
Diez
Diez B. Roggisch wrote: This is a strange asymmetry, and I was wondering if I've overlooked a simple way to transform a callback into a generator, but I now tend to the assumption that you *need* threads: One thread with the callback puts names or name lists into the queue until it's full (some arbitrary limit that also determines the total memory needed), another thread (the generator) consumes names from the queue.
I also thought about that - its the only thing that allows for real lazyness - you can wait in the callback until the generator clears a semaphore. But if the context-changes are worth the effort is questionable.
There is another aspect I forgot to mention. Your approach would only start
to yield results after all results are found, i. e. in the example you
would have to scan the whole harddisk even when the first visited file
might have been the one you want.
So I think it is best to either not change the callback approach or use
threads as outlined by Jimmy Retzlaff.
Peter This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Francis Avila |
last post by:
A little annoyed one day that I couldn't use the statefulness of
generators as "resumable functions", I came across Hettinger's PEP 288
(http://www.python.org/peps/pep-0288.html, still listed as...
|
by: Wai Yip Tung |
last post by:
I'm attempting to turn some process than uses callback to return result
into a more user friendly generator. I'm hitting some road block so any
pointer would be appriciated.
Let say there is an...
|
by: ma740988 |
last post by:
// file sltest.h
#ifndef SLTEST_H
#define SLTEST_H
class CallbackBase // herb shutters gotW source ..
{
public:
virtual void operator()() const { };
virtual ~CallbackBase() = 0;
};
|
by: Felix Kater |
last post by:
Hi,
in a given library I register callback functions with this function:
bool set_callback(int index, int (*callback_function)(long));
I need the callback function to also pass the index...
|
by: Robert |
last post by:
After failing on a yield/iterator-continuation problem in Python (see
below) I tried the Ruby (1.8.2) language first time on that construct:
The example tries to convert a block callback interface...
|
by: Kirk McDonald |
last post by:
Let's say I have a function that takes a callback function as a
parameter, and uses it to describe an iteration:
def func(callback):
for i in :
callback(i)
For the sake of argument, assume...
|
by: DaTurk |
last post by:
Hi,
I'm coding up an application that has a native c++ layer,
asynchronously calling callback in a CLI layer.
We originally did this with static inline methods in the CLI layer,
but this...
|
by: Tim Spens |
last post by:
--- On Fri, 6/27/08, Tim Spens <t_spens@yahoo.comwrote:
I think I know where the problem is but I'm unsure how to fix it. When I call Register_Handler(...) from python via...
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
| |