473,770 Members | 1,677 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Strange behavior with iterables - is this a bug?

Ok, I am confused about this one. I'm not sure if it's a bug or a
feature.. but
=============== =============== == RESTART
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in f1 for i2 in f2 for i3 in f3] [('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c')] l1 = ['a\n','b\n','c\ n']
l2 = ['a\n','b\n','c\ n']

l3 = ['a\n','b\n','c\ n']
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in l1 for i2 in l2 for i3 in l3]

[('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c'), ('a', 'b', 'a'),
('a', 'b', 'b'), ('a', 'b', 'c'), ('a', 'c', 'a'), ('a', 'c', 'b'),
('a', 'c', 'c'), ('b', 'a', 'a'), ('b', 'a', 'b'), ('b', 'a', 'c'),
('b', 'b', 'a'), ('b', 'b', 'b'), ('b', 'b', 'c'), ('b', 'c', 'a'),
('b', 'c', 'b'), ('b', 'c', 'c'), ('c', 'a', 'a'), ('c', 'a', 'b'),
('c', 'a', 'c'), ('c', 'b', 'a'), ('c', 'b', 'b'), ('c', 'b', 'c'),
('c', 'c', 'a'), ('c', 'c', 'b'), ('c', 'c', 'c')]

explanation of code: the files word1.txt, word2.txt and word3.txt are
all identical conataining the letters a,b and c one letter per line.
The lists I've added the "\n" so that the lists are identical to what
is returned by the file objects. Just eliminating any possible
differences.
If you notice, when using the file objects I don't get the proper set
of permutations. I was playing around with doing this via recursion,
etc. But nothing was working so I made a simplest case nesting. Still
no go.
Why does this not work with the file objects? Or any other class I''ve
made which implements __iter__ and next?

Seems like a bug to me, but maybe I am missing something. Seems to
happen in 2.3 and 2.4.

May 30 '06 #1
7 1053

<ak*********@gm ail.com> wrote in message
news:11******** **************@ u72g2000cwu.goo glegroups.com.. .
Ok, I am confused about this one. I'm not sure if it's a bug or a
feature.. but
=============== =============== == RESTART
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in f1 for i2 in f2
for i3 in f3] [('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c')]


A file is something like an iterator and something like an iterable. At
this point, the internal cursur for f3 points at EOF. To reiterate thru
the file, you must rewind in the inner loops. So try (untest by me)

def initf(fil):
f.seek(0)
return f

and ...for i2 in initf(f2) for i3 in initf(f3)

l1 = ['a\n','b\n','c\ n']
l2 = ['a\n','b\n','c\ n']

l3 = ['a\n','b\n','c\ n']
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in l1 for i2 in l2
for i3 in l3]

[('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c'), ('a', 'b', 'a'),
('a', 'b', 'b'), ('a', 'b', 'c'), ('a', 'c', 'a'), ('a', 'c', 'b'),
('a', 'c', 'c'), ('b', 'a', 'a'), ('b', 'a', 'b'), ('b', 'a', 'c'),
('b', 'b', 'a'), ('b', 'b', 'b'), ('b', 'b', 'c'), ('b', 'c', 'a'),
('b', 'c', 'b'), ('b', 'c', 'c'), ('c', 'a', 'a'), ('c', 'a', 'b'),
('c', 'a', 'c'), ('c', 'b', 'a'), ('c', 'b', 'b'), ('c', 'b', 'c'),
('c', 'c', 'a'), ('c', 'c', 'b'), ('c', 'c', 'c')]

explanation of code: the files word1.txt, word2.txt and word3.txt are
all identical conataining the letters a,b and c one letter per line.
The lists I've added the "\n" so that the lists are identical to what
is returned by the file objects. Just eliminating any possible
differences.


But lists are not file objects and you did not eliminate the crucial
difference in reiterability. Try your experiment with StringIO objects,
which are more nearly identical to file objects.

Terry Jan Reedy

May 30 '06 #2
On Tue, May 30, 2006 at 01:11:26PM -0700, ak*********@gma il.com wrote:
[...]
=============== =============== == RESTART
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in f1 for i2 in f2 for i3 in f3] [('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c')] l1 = ['a\n','b\n','c\ n']
l2 = ['a\n','b\n','c\ n']

l3 = ['a\n','b\n','c\ n']
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in l1 for i2 in l2 for i3 in l3] [('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c'), ('a', 'b', 'a'),
('a', 'b', 'b'), ('a', 'b', 'c'), ('a', 'c', 'a'), ('a', 'c', 'b'),
('a', 'c', 'c'), ('b', 'a', 'a'), ('b', 'a', 'b'), ('b', 'a', 'c'),
('b', 'b', 'a'), ('b', 'b', 'b'), ('b', 'b', 'c'), ('b', 'c', 'a'),
('b', 'c', 'b'), ('b', 'c', 'c'), ('c', 'a', 'a'), ('c', 'a', 'b'),
('c', 'a', 'c'), ('c', 'b', 'a'), ('c', 'b', 'b'), ('c', 'b', 'c'),
('c', 'c', 'a'), ('c', 'c', 'b'), ('c', 'c', 'c')]

explanation of code: the files word1.txt, word2.txt and word3.txt are
all identical conataining the letters a,b and c one letter per line.
The lists I've added the "\n" so that the lists are identical to what
is returned by the file objects. Just eliminating any possible
differences.


You're comparing file, which is ITERATOR, and list, which is ITERABLE,
not ITERATOR. To get the result you want, use this instead;
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in open('word1.txt ')
for i2 in open('word2.txt ')
for i3 in open('word3.txt ')]

FIY, to get the same buggy(?) result using list, try this instead;
l1 = iter(['a\n','b\n','c\ n'])
l2 = iter(['a\n','b\n','c\ n'])
l3 = iter(['a\n','b\n','c\ n'])
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in l1 for i2 in l2 for i3 in l3] [('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c')]

-Inyeol Lee
May 30 '06 #3
ak*********@gma il.com wrote:
Ok, I am confused about this one. I'm not sure if it's a bug or a
feature.. but

List comprehension is a great shortcut, but when the shortcut starts
causing trouble, better to go with the old ways. You need to reopen each
file each time you want to iterate through it. You should be able to
understand the difference between these two bits of code.

The first bit opens each file but uses (two of them) multiple times.
Reading from a file at EOF returns an empty sequence.

The second bit opened the file each time you want to reuse it. That
works correctly.

And that suggest the third bit of correctly working code which uses list
comprehension.

# Fails because files are opened once but reused
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
for i1 in f1:
for i2 in f2:
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Works because files are reopened for each reuse:
f1 = open('word1.txt ')
for i1 in f1:
f2 = open('word2.txt ')
for i2 in f2:
f3 = open('word3.txt ')
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Also works because files are reopened for each use:
print [(i1.strip(),i2. strip(),i3.stri p())
for i1 in open('word1.txt ')
for i2 in open('word2.txt ')
for i3 in open('word3.txt ')]

Hope that's clear!

Gary Herron



=========== =============== ====== RESTART
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in f1 for i2 in f2 for i3 in f3]

[('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c')]

l1 = ['a\n','b\n','c\ n']
l2 = ['a\n','b\n','c\ n']

l3 = ['a\n','b\n','c\ n']
print [(i1.strip(),i2. strip(),i3.stri p(),) for i1 in l1 for i2 in l2 for i3 in l3]

[('a', 'a', 'a'), ('a', 'a', 'b'), ('a', 'a', 'c'), ('a', 'b', 'a'),
('a', 'b', 'b'), ('a', 'b', 'c'), ('a', 'c', 'a'), ('a', 'c', 'b'),
('a', 'c', 'c'), ('b', 'a', 'a'), ('b', 'a', 'b'), ('b', 'a', 'c'),
('b', 'b', 'a'), ('b', 'b', 'b'), ('b', 'b', 'c'), ('b', 'c', 'a'),
('b', 'c', 'b'), ('b', 'c', 'c'), ('c', 'a', 'a'), ('c', 'a', 'b'),
('c', 'a', 'c'), ('c', 'b', 'a'), ('c', 'b', 'b'), ('c', 'b', 'c'),
('c', 'c', 'a'), ('c', 'c', 'b'), ('c', 'c', 'c')]

explanation of code: the files word1.txt, word2.txt and word3.txt are
all identical conataining the letters a,b and c one letter per line.
The lists I've added the "\n" so that the lists are identical to what
is returned by the file objects. Just eliminating any possible
differences.
If you notice, when using the file objects I don't get the proper set
of permutations. I was playing around with doing this via recursion,
etc. But nothing was working so I made a simplest case nesting. Still
no go.
Why does this not work with the file objects? Or any other class I''ve
made which implements __iter__ and next?

Seems like a bug to me, but maybe I am missing something. Seems to
happen in 2.3 and 2.4.


May 30 '06 #4
DOH!!

thanks a lot. had to be something stupid on my part.

Now I get it :)

May 30 '06 #5

Gary Herron wrote:
List comprehension is a great shortcut, but when the shortcut starts
causing trouble, better to go with the old ways. You need to reopen each
file each time you want to iterate through it. You should be able to
understand the difference between these two bits of code.

The first bit opens each file but uses (two of them) multiple times.
Reading from a file at EOF returns an empty sequence.

The second bit opened the file each time you want to reuse it. That
works correctly.

And that suggest the third bit of correctly working code which uses list
comprehension.

# Fails because files are opened once but reused
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
for i1 in f1:
for i2 in f2:
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Works because files are reopened for each reuse:
f1 = open('word1.txt ')
for i1 in f1:
f2 = open('word2.txt ')
for i2 in f2:
f3 = open('word3.txt ')
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Also works because files are reopened for each use:
print [(i1.strip(),i2. strip(),i3.stri p())
for i1 in open('word1.txt ')
for i2 in open('word2.txt ')
for i3 in open('word3.txt ')]

Hope that's clear!

Gary Herron

My original problem was with recursion. I explicitly nested it out to
try and understand the behavior - and foolishly looked in the wrong
spot for the problem, namely that file is not reitreable. In truth I
was never concerned about file objects, the problem was failing with my
own custom iterators (wich also were not reiterable) and I switched to
file, to eliminate possible code deficiencies on my own part. I was
simply chasing down the wrong problem. As was pointed out to me in a
nother thread - the cleanest implementation which would allow me to use
one copy of the file (in my example the files are identical) would be
to use a trivial iterator class that opens the file, uses tell to track
position and seek to set position, and returns the appropriate line for
that instance - thus eliminating unnecessary file opens and closes.

May 31 '06 #6
ak*********@gma il.com wrote:
Gary Herron wrote:

List comprehension is a great shortcut, but when the shortcut starts
causing trouble, better to go with the old ways. You need to reopen each
file each time you want to iterate through it. You should be able to
understand the difference between these two bits of code.

The first bit opens each file but uses (two of them) multiple times.
Reading from a file at EOF returns an empty sequence.

The second bit opened the file each time you want to reuse it. That
works correctly.

And that suggest the third bit of correctly working code which uses list
comprehension .

# Fails because files are opened once but reused
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
for i1 in f1:
for i2 in f2:
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Works because files are reopened for each reuse:
f1 = open('word1.txt ')
for i1 in f1:
f2 = open('word2.txt ')
for i2 in f2:
f3 = open('word3.txt ')
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Also works because files are reopened for each use:
print [(i1.strip(),i2. strip(),i3.stri p())
for i1 in open('word1.txt ')
for i2 in open('word2.txt ')
for i3 in open('word3.txt ')]

Hope that's clear!

Gary Herron

My original problem was with recursion. I explicitly nested it out to
try and understand the behavior - and foolishly looked in the wrong
spot for the problem, namely that file is not reitreable. In truth I
was never concerned about file objects, the problem was failing with my
own custom iterators (wich also were not reiterable) and I switched to
file, to eliminate possible code deficiencies on my own part. I was
simply chasing down the wrong problem. As was pointed out to me in a
nother thread - the cleanest implementation which would allow me to use
one copy of the file (in my example the files are identical) would be
to use a trivial iterator class that opens the file, uses tell to track
position and seek to set position, and returns the appropriate line for
that instance - thus eliminating unnecessary file opens and closes.

I see.

I wouldn't call "tell" and "seek" clean. Here's another suggestion. Use
l1 = open(...).readl ines()
to read the whole file into a (nicely reiterable) list residing in
memory, and then iterate through the list as you wish. Only if your
files are MANY megabytes long would this be a problem with memory
consumption. (But if they were that big, you wouldn't be trying to find
all permutations would you!)

Gary Herron

May 31 '06 #7
My original concern and reason for goint the iterator/generator route
was exactly for large large lists :) Unnecessary in this example, but
exactly what I was exploring. I wouldn't be using list comprehension
for generating the permutiations. Where all this came from was
creating a generator/iterator to handle very large permutations.

Gary Herron wrote:
ak*********@gma il.com wrote:
Gary Herron wrote:

List comprehension is a great shortcut, but when the shortcut starts
causing trouble, better to go with the old ways. You need to reopen each
file each time you want to iterate through it. You should be able to
understand the difference between these two bits of code.

The first bit opens each file but uses (two of them) multiple times.
Reading from a file at EOF returns an empty sequence.

The second bit opened the file each time you want to reuse it. That
works correctly.

And that suggest the third bit of correctly working code which uses list
comprehension .

# Fails because files are opened once but reused
f1 = open('word1.txt ')
f2 = open('word2.txt ')
f3 = open('word3.txt ')
for i1 in f1:
for i2 in f2:
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Works because files are reopened for each reuse:
f1 = open('word1.txt ')
for i1 in f1:
f2 = open('word2.txt ')
for i2 in f2:
f3 = open('word3.txt ')
for i3 in f3:
print (i1.strip(),i2. strip(),i3.stri p())

and

# Also works because files are reopened for each use:
print [(i1.strip(),i2. strip(),i3.stri p())
for i1 in open('word1.txt ')
for i2 in open('word2.txt ')
for i3 in open('word3.txt ')]

Hope that's clear!

Gary Herron

My original problem was with recursion. I explicitly nested it out to
try and understand the behavior - and foolishly looked in the wrong
spot for the problem, namely that file is not reitreable. In truth I
was never concerned about file objects, the problem was failing with my
own custom iterators (wich also were not reiterable) and I switched to
file, to eliminate possible code deficiencies on my own part. I was
simply chasing down the wrong problem. As was pointed out to me in a
nother thread - the cleanest implementation which would allow me to use
one copy of the file (in my example the files are identical) would be
to use a trivial iterator class that opens the file, uses tell to track
position and seek to set position, and returns the appropriate line for
that instance - thus eliminating unnecessary file opens and closes.

I see.

I wouldn't call "tell" and "seek" clean. Here's another suggestion. Use
l1 = open(...).readl ines()
to read the whole file into a (nicely reiterable) list residing in
memory, and then iterate through the list as you wish. Only if your
files are MANY megabytes long would this be a problem with memory
consumption. (But if they were that big, you wouldn't be trying to find
all permutations would you!)

Gary Herron


May 31 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

18
1618
by: genc ymeri | last post by:
Hi, I just noticed that while trying to run this code : MessageBox.Show("Hello"); the message box pops up but shows no string/message in it. (expecting the "Hello" string). Even the "OK" string in the button itself in dialog box doesn't show. I tried to write "Hello" string in the caption/text of the button or the
38
2111
by: Steven Bethard | last post by:
> >>> aList = > >>> it = iter(aList) > >>> zip(it, it) > > That behavior is currently an accident. >http://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1121416
1
1507
by: Alexander Inochkin | last post by:
Hi! I found same strange behavior of ASP.NET. It is possible this is the bug. Follow the steps:
0
3574
by: ivb | last post by:
Hi all, I am using DB2 8.1.11.1 on NT with ASP.NET 1.1 When application make connection to database (via ADO.NET), it set "Connection timeout" parameter to 30 seconds. After, when my webpage requests database, and query execution time exceeds 30 seconds, the following error reported: ===
6
2273
by: Joseph Geretz | last post by:
Writing an Outlook AddIn with C#. For the user interface within Outlook I'm adding matching pairs of Toolbar buttons and Menu items. All of the buttons and menu items are wired up to send events to the same method (aka delegate?). I use the Tag property within this method to determine what user action is taking place. Very simple: When adding toolbar button: tbButton.Click += new...
3
1518
by: sara | last post by:
Very strange behavior, but I suspect some is A2K and some might be for me to correct. Just trying to see if anyone can help and advise. We have a database that's been running for a few years with no problems. We continuously add queries and reports. We're up to about 700 queries (no, not all are used and most are parameter queries - the business asks a LOT of questions!), and under 250 reports and fewer than 15 forms. Data is...
1
2978
by: Nicholas Palmer | last post by:
Hi all, Got a question about the AspCompat=true page property. First a little background. We have an ASP.NET app that uses two COM components. The first is the Microsoft OWC 11 components and the second is a custom VB6 COM component. So I was reading about AspCompat=true and it seemed like it would be a good fit for our app. From what I can tell both of the COM components that we are using are STA and we are creating the components in...
19
1756
by: david | last post by:
I took old code and decided to modify it a bit, and I just noticed that it does not compile at all and before server one of severs (main) crashed in the system it was working fine (I am really sure that I remember that). Now I am getting this error: $ make g++ -Wall -ansi -pedantic -c pirma_lib.cpp g++ -Wall -ansi -pedantic -c pirma.cpp In file included from pirma.cpp:2:
10
1065
by: Pekka Laukkanen | last post by:
Hello, just noticed this: Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) on darwin Type "help", "copyright", "credits" or "license" for more information. {1: 2} {True: False} {1: False}
0
9439
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10237
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10071
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10017
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9882
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7431
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5326
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5467
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
3589
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.