472,334 Members | 2,212 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,334 software developers and data experts.

File Closing Problem in 2.3 and 2.4, Not in 2.5

Greetings:

Please forgive me if this is the wrong place for this post. I couldn't find a more acceptable forum. If there is one, please point me in the right direction.

I am part of a small team writing a table-driven automated testing framework for embedded software. The tables, which contain rows of keywords and data that drive the testing, are stored as plain-text "Comma-Separated Value" or .csv files. Each table can call other tables, which means that multiple files may be open at a time.

The framework includes a Parser class. The program takes the name of the top-level table as a parameter, creates an instance of the Parser and passes the table name to it. The Parser instance opens the .csv file with that name, reads each line of the file (row of the table) and takes the appropriate action. When it encounters a row referencing another table, it creates a new Parser instance and passes it the name of the new table, suspending its own operation until the new Parser instance completes.

In this way, a tree of Parser instances is created, each with a single open file object. (BTW, recursive and circular references are not allowed.) When each Parser instance comes to the end of its table, the instance is explicitly destroyed, presumably destroying any objects it holds, AND closing its open file.

Because of the nature of the framework and the testing we do, this Parser tree never gets very tall: four or five levels at the most. The same table may be invoked dozens or hundreds of times, however, with different sets of data each time.

This is where things go wrong. After about 500 table invocations, the framework starts refusing to process any more tables, responding with the following error:

[Errno 24] Too many open files: 'HostCommandDevNotReady.csv'

We can correct the behavior by explicitly closing each Parser's table file object before exiting the Parser code. This indicates that Python is failing to free up some file-related internal resource when the Parser object is destroyed. This behavior occurs on Python 2.3 and 2.4 for Windows, but not on Python 2.3 for Linux, and not on the Windows version of Python2.5.

This is why I didn't just post a report to Bug Tracker: the problem seems to have been fixed. I did search through the archive of Windows related bugs, but found no mention of this type of a bug. What I want to know is:

* has anyone else encountered a problem like this,
* how was the problem corrected,
* can the fix be retro-fitted to 2.5 and 2.4?

Thanks in advance for any information you can provide.

Regards,
*
Barry
ba***********@psc.com
541-302-1107
________________________
We who cut mere stones must always be envisioning cathedrals.
-Quarry worker's creed
Jan 6 '07 #1
3 1425
Carroll, Barry schrieb:
What I want to know is:

* has anyone else encountered a problem like this, * how was the
problem corrected, * can the fix be retro-fitted to 2.5 and 2.4?
From your description, I suspect an error in your code. Your description
indicates that you don't expect to have more than five files open
simultaneously. Yet, the error message "Too many open files" occurs when
you open many more files (in the order of hundreds of files).

It is very unlikely that there is a bug in Python where it would fail to
close a file when .close() is explicitly invoked on it (as your
description suggests that you do), so if you get that error message, it
can only mean that you fail to close some files.

Notice that you may have other files open, as well, and that those also
count towards the limit.

As a debugging utility, you can use Sysinternal's process explorer.
Make the program halt (not exit) when the exception occurs (e.g. by
having it sleep(1) in a loop), then view all open handles in the
process explorer (check the menu if it doesn't display them initially).

Regards,
Martin
Jan 6 '07 #2
Martin v. Lwis wrote:
Carroll, Barry schrieb:
>What I want to know is:

* has anyone else encountered a problem like this, * how was the
problem corrected, * can the fix be retro-fitted to 2.5 and 2.4?

From your description, I suspect an error in your code. Your description
indicates that you don't expect to have more than five files open
simultaneously. Yet, the error message "Too many open files" occurs when
you open many more files (in the order of hundreds of files).

It is very unlikely that there is a bug in Python where it would fail to
close a file when .close() is explicitly invoked on it (as your
description suggests that you do), so if you get that error message, it
can only mean that you fail to close some files.

Notice that you may have other files open, as well, and that those also
count towards the limit.

As a debugging utility, you can use Sysinternal's process explorer.
Make the program halt (not exit) when the exception occurs (e.g. by
having it sleep(1) in a loop), then view all open handles in the
process explorer (check the menu if it doesn't display them initially).

Regards,
Martin
I agree with Martin .. this code to close is solid.

Make certain you are really closing the files when you think you should.
I am pretty sure you are not. Look at the code that closes the files
closely. Put a print statement in the block that is supposed to close
the files (may bee even a raw_input("closing file" + afile) statement).

My guess is that you won't see the print statements trigger when you
though they should .. they may be out of "the loop" you thought that
they were in.
Jan 6 '07 #3
Martin v. Lwis wrote:
Carroll, Barry schrieb:
What I want to know is:

* has anyone else encountered a problem like this, * how was the
problem corrected, * can the fix be retro-fitted to 2.5 and 2.4?

From your description, I suspect an error in your code. Your description
indicates that you don't expect to have more than five files open
simultaneously. Yet, the error message "Too many open files" occurs when
you open many more files (in the order of hundreds of files).

It is very unlikely that there is a bug in Python where it would fail to
close a file when .close() is explicitly invoked on it (as your
description suggests that you do), so if you get that error message, it
can only mean that you fail to close some files.
I don't understand: the OP's description suggests nothing of the sort
to me. What he said was:
"""
In this way, a tree of Parser instances is created, each with a single
open file object. (BTW, recursive and circular references are not
allowed.) When each Parser instance comes to the end of its table, the
instance is explicitly destroyed, presumably destroying any objects it
holds, AND closing its open file.
"""
which I interpret as: he is doing del parser_instance, and *presuming*
(incorrectly) that attributes of parser_instance (including an open
file object) are magically whisked away instantly, instead of
later/maybe. He later says he explicitly closed the files, which fixed
what he alleges (incorrectly) to be a bug.

To the OP:
(1) The del statement doesn't "destroy" anything. It unbinds the name
from the object in the current namespace, and decrements the object's
reference count. Only if the reference count is then zero will the
janitor be called in.
(2) Check the reference count on the parser_instance just before you
del it. You could be retaining a reference somewhere.
(3) Explicitly close all non-lightweight objects like files (even
read-only ones) and sockets rather than hoping they will go away.

HTH,
John

Jan 7 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

22
by: Bryan | last post by:
i'm curious to know how others handle the closing of files. i seem to always end up with this pattern even though i rarely see others do it. f1...
2
by: muser | last post by:
Karl this is albeit your program, in the program that you wrote for me I learned quite alot, i didn't realise I could return NULL for instance or...
6
by: Affan Syed | last post by:
Hi, I am getting this weird problem. I know what i am doing is strange.. i am using C++ vectors and fopen, but for some reason if i used ofstream...
2
by: Atulvid | last post by:
Hi, I created a test file for my application by writing structures to a binary file. To make sure that I will get correct data for the pointers...
4
by: Dameon | last post by:
Hi All, I have a process where I'd like to search the contents of a file(in a dir) for all occurences (or the count of) of a given string. My...
6
by: Steven D'Aprano | last post by:
Closing a file can (I believe) raise an exception. Is that documented anywhere? I've spent a lot of frustrating time trying to track this down,...
5
by: buu | last post by:
I have an app made in vb.net with a part of it made in vc.net. after finishing a data processing with a file, I would like to delete that file, but...
1
by: ofuuzo1 | last post by:
Hi, Is there anyway I can append a new element to an existing xml without first loading the existing file into a variable, adding the new element...
1
by: Hamayun Khan | last post by:
Hi All I have text files having queries like below. INSERT INTO tblJobScrap (,,,,,,,,,) VALUES...
0
by: teenabhardwaj | last post by:
How would one discover a valid source for learning news, comfort, and help for engineering designs? Covering through piles of books takes a lot of...
0
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
0
by: CD Tom | last post by:
This happens in runtime 2013 and 2016. When a report is run and then closed a toolbar shows up and the only way to get it to go away is to right...
0
by: CD Tom | last post by:
This only shows up in access runtime. When a user select a report from my report menu when they close the report they get a menu I've called Add-ins...
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
0
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
2
by: Matthew3360 | last post by:
Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it...
0
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...
0
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.