473,320 Members | 1,821 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Help me understand this iterator

Hi,

I've found this script over at effbot
(http://effbot.org/librarybook/os-path.htm), and I can't get my head
around its inner workings. Here's the script:

import os

class DirectoryWalker:
# a forward iterator that traverses a directory tree

def __init__(self, directory):
self.stack = [directory]
self.files = []
self.index = 0

def __getitem__(self, index):
while 1:
try:
file = self.files[self.index]
self.index = self.index + 1
except IndexError:
# pop next directory from stack
self.directory = self.stack.pop()
self.files = os.listdir(self.directory)
self.index = 0
else:
# got a filename
fullname = os.path.join(self.directory, file)
if os.path.isdir(fullname) and not
os.path.islink(fullname):
self.stack.append(fullname)
return fullname

for file in DirectoryWalker("."):
print file

Now, if I look at this script step by step, I don't understand:
- what is being iterated over (what is being called by "file in
DirectoryWalker()"?);
- where it gets the "index" value from;
- where the "while 1:"-loop is quitted.

Thanks in advance,

Mathieu

Oct 31 '06 #1
8 1173
LaundroMat wrote:
Hi,

I've found this script over at effbot
(http://effbot.org/librarybook/os-path.htm), and I can't get my head
around its inner workings. Here's the script:

import os

class DirectoryWalker:
# a forward iterator that traverses a directory tree

def __init__(self, directory):
self.stack = [directory]
self.files = []
self.index = 0

def __getitem__(self, index):
while 1:
try:
file = self.files[self.index]
self.index = self.index + 1
except IndexError:
# pop next directory from stack
self.directory = self.stack.pop()
self.files = os.listdir(self.directory)
self.index = 0
else:
# got a filename
fullname = os.path.join(self.directory, file)
if os.path.isdir(fullname) and not
os.path.islink(fullname):
self.stack.append(fullname)
return fullname

for file in DirectoryWalker("."):
print file

Now, if I look at this script step by step, I don't understand:
- what is being iterated over (what is being called by "file in
DirectoryWalker()"?);
- where it gets the "index" value from;
- where the "while 1:"-loop is quitted.
With

dw = DirectoryWalker(".")

the for loop is equivalent to

index = 0 # internal variable, not visible from Python
while True:
try:
file = dw[index] # invokes dw.__getitem__(index)
except IndexError:
break
print file

This is an old way of iterating over a sequence which is only used when the
iterator-based approach

dwi = iter(dw) # invokes dw.__iter__()
while True:
try:
file = dwi.next()
except StopIteration:
break
print file

fails.

Peter
Oct 31 '06 #2
LaundroMat wrote:
Now, if I look at this script step by step, I don't understand:
- what is being iterated over (what is being called by "file in
DirectoryWalker()"?);
as explained in the text above the script, this class emulates a
sequence. it does this by implementing the __getindex__ method:

http://effbot.org/pyref/__getitem__
- where it gets the "index" value from;
from the call to __getitem__ done by the for-in loop.
- where the "while 1:"-loop is quitted.
the loop stops when the stack is empty, and pop raises an IndexError
exception.

note that this is an old example; code written for newer versions of
Python would probably use a recursing generator instead (see the source
code for os.walk in the standard library for an example).

</F>

Oct 31 '06 #3
On Tue, 31 Oct 2006 03:36:08 -0800, LaundroMat wrote:
Hi,

I've found this script over at effbot
(http://effbot.org/librarybook/os-path.htm), and I can't get my head
around its inner workings.
[snip code]
Now, if I look at this script step by step, I don't understand:
- what is being iterated over (what is being called by "file in
DirectoryWalker()"?);
What is being iterated over is the list of files in the current directory.
In Unix land (and probably DOS/Windows as well) the directory "." means
"this directory, right here".

- where it gets the "index" value from;
When Python see's a line like "for x in obj:" it does some special
magic. First it looks to see if obj has a "next" method, that is, it
tries to call obj.next() repeatedly. That's not the case here --
DirectoryWalker is an old-style iterator, not one of the fancy new ones.

Instead, Python tries calling obj[index] starting at 0 and keeps going
until an IndexError exception is raised, then it halts the for loop.

So, think of it like this: pretend that Python expands the following code:

for x in obj:
block

into something like this:

index = 0
while True: # loop forever
try:
x = obj[index]
block # can use x in block
except IndexError:
# catch the exception and escape the while loop
break
index = index + 1
# and now we're done, continue the rest of the program

That's not exactly what Python does, of course, it is much more efficient,
but that's a good picture of what happens.

- where the "while 1:"-loop is quitted.

The while 1 loop is escaped when the function hits the return statement.

--
Steven.

Oct 31 '06 #4
LaundroMat wrote:

[me hitting send too soon]
Now, if I look at this script step by step, I don't understand:
- where the "while 1:"-loop is quitted.
class DirectoryWalker:
# a forward iterator that traverses a directory tree

def __init__(self, directory):
self.stack = [directory]
self.files = []
self.index = 0

def __getitem__(self, index):
while 1:
try:
file = self.files[self.index]
self.index = self.index + 1
except IndexError:
# pop next directory from stack
self.directory = self.stack.pop()
If self.stack is empty, pop() will raise an IndexError which terminates both
the 'while 1' loop in __getitem__() and the enclosing 'for file in ...'
loop
self.files = os.listdir(self.directory)
self.index = 0
else:
# got a filename
fullname = os.path.join(self.directory, file)
if os.path.isdir(fullname) and not
os.path.islink(fullname):
self.stack.append(fullname)
return fullname
The return statement feeds the next file to the for loop.

Peter

Oct 31 '06 #5
Thanks all, those were some great explanations. It seems I have still
still a long way for me to go before I grasp the intricacies of this
language.

That 'magic index' variable bugs me a little however. It gives me the
same feeling as when I see hard-coded variables. I suppose the
generator class has taken care of this with its next() method (although
- I should have a look - __next__() probable takes self and index as
its arguments). Although I'm very fond of the language (as a
non-formally trained hobbyist developer), that "magic" bit is a tad
disturbing.

Still, thanks for the quick and complete replies!

Oct 31 '06 #6
Ack, I get it now. It's not the variable's name ("index") that is
hard-coded, it's just that the for...in... loop sends an argument by
default. That's a lot more comforting.

Oct 31 '06 #7
LaundroMat wrote:
That 'magic index' variable bugs me a little however. It gives me the
same feeling as when I see hard-coded variables.
what magic index? the variable named "index" is an argument to the
method it's used in.

</F>

Oct 31 '06 #8
On Oct 31, 3:53 pm, Fredrik Lundh <fred...@pythonware.comwrote:
LaundroMat wrote:
That 'magic index' variable bugs me a little however. It gives me the
same feeling as when I see hard-coded variables.what magic index? the variable named "index" is an argument to the
method it's used in.
Yes, I reacted too quickly. Sorry.

Oct 31 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

45
by: Joh | last post by:
hello, i'm trying to understand how i could build following consecutive sets from a root one using generator : l = would like to produce : , , , ,
4
by: Rex_chaos | last post by:
Hi there, I am writing an iterator for a container. Just like a typical iterator, template <typename ADT> class MyIter { ... MyContainer<ADT>& refc; public:
3
by: CoolPint | last post by:
I have implemented a generic priority queue below and tested it works fine, but I have one small problem I cannot understand. I have type parameter F which determines the priority so that users can...
5
by: Shane | last post by:
Thanks in advance for the help. I'm new to the STL and havig a bit of trouble figuring out the whole iterator thing. I am using the <list> template and I can insert elements into the list just...
6
by: woosu | last post by:
Hello ladies and gentlemen. I have a relatively simple problem that I've been unable to solve. In the interest of learning C++, I've decided to write a simple game. The basis for the game is...
36
by: felixnielsen | last post by:
What i really wanna do, is defining my own types, it doesnt really matter why. Anyway, i have run into some problems 1) typedef unsigned short U16; U16 test = 0xffffffff; // There should be a...
18
by: Nobody | last post by:
I've been looking for a job for a while now, and have run into this interview question twice now... and have stupidly kind of blown it twice... (although I've gotten better)... time to finally...
6
by: StephQ | last post by:
I need to implement an algorithm that takes as input a container and write some output in another container. The containers involved are usually vectors, but I would like not to rule out the...
7
by: DJ Dharme | last post by:
Hi, I really like to use stl as much as possible in my code. But I found it really hard to understand by looking into there source code. I have no idea about what iterator traits, heaps and...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.