Misleading error message when opening a file (on Windows XP SP 2)

[Claudio Grondi]

Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

>>f = file('veryBigFile.dat','r')
f = file('veryBigFile.dat','r+')

Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Claudio Grondi

[Marc 'BlackJack' Rintsch]

In <ec**********@newsreader2.netcologne.de>, Claudio Grondi wrote:

>
Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

>>f = file('veryBigFile.dat','r')
>>f = file('veryBigFile.dat','r+')

You mention the file size and gave a "speaking" name to that file -- does
the file size matter?

Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

It's the error number Windows returns for that operation.

Ciao,
Marc 'BlackJack' Rintsch

[Claudio Grondi]

Marc 'BlackJack' Rintsch wrote:

In <ec**********@newsreader2.netcologne.de>, Claudio Grondi wrote:

>>Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

>>f = file('veryBigFile.dat','r')
f = file('veryBigFile.dat','r+')

You mention the file size and gave a "speaking" name to that file -- does
the file size matter?

Yes, it does.
I haven't tested it yet, but I suppose 2 or 4 GByte threshold value.

>

>>Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

It's the error number Windows returns for that operation.

So you just try to say:
"it's not Python fault - it's just another bug of the damn Microsoft
Windows operating system", right?

Claudio Grondi

[Tim Peters]

[Claudio Grondi]

Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

>>f = file('veryBigFile.dat','r')
>>f = file('veryBigFile.dat','r+')

Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

[Fredrik Lundh]

Tim Peters wrote:

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

dir bigfile.dat

2006-08-28 11:46 5 000 000 000 bigfile.dat

>>f = file("bigfile.dat", "r")
f = file("bigfile.dat", "r+")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'bigfile.dat'

>>f = file("bigfile.dat", "rb")
f = file("bigfile.dat", "r+b")

(typing f.read() here is a nice way to lock up the machine ;-)

</F>

[Claudio Grondi]

Tim Peters wrote:

[Claudio Grondi]

>Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

> >>f = file('veryBigFile.dat','r')
f = file('veryBigFile.dat','r+')

Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

I knew already that 'r+b' fixes it. Yes, you have won the bet :) .

I suppose, like you do, that because there is a difference between text
and binary files on Windows and the text files are e.g. opened being
buffered using a 32-bit buffer pointer, this fails on too large NTFS files.

I could also imagine that Python tries to buffer the text file and fails
because it uses the wrong pointer size when asking Windows for the
content. I have not yet looked into the C-code of Python - any hint
which file I should take a closer look at?
Just curious to see for myself, that the bug is on the Windows side.

Claudio Grondi

[Fredrik Lundh]

Tim Peters wrote:

>Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

however, if you use the C level API, you get EINVAL (which presumably means
that the CRT cannot open this file in text mode), not ENOENT. this is also true
for older versions of Python:

Python 2.1.1 (#20, Aug 23 2001, 11:27:17) [MSC 32 bit (Intel)] on win32

>>f = open("bigfile.dat")
f = open("bigfile.dat", "r+")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
IOError: [Errno 22] Invalid argument: 'bigfile.dat'

which probably means that this fix

http://www.python.org/sf/538827

is partially responsible for the misleading error message.

(the cause of this seems to be that when you open a text file for updating, the
CRT check if there's a chr(26) at the end of the file, but the 32-bit lseek API
doesn't support seeking to positions larger than 2^31-2)

</F>

[Claudio Grondi]

Fredrik Lundh wrote:

Tim Peters wrote:

>>>Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

however, if you use the C level API, you get EINVAL (which presumably means
that the CRT cannot open this file in text mode), not ENOENT. this is also true
for older versions of Python:

Python 2.1.1 (#20, Aug 23 2001, 11:27:17) [MSC 32 bit (Intel)] on win32

>>>>f = open("bigfile.dat")
f = open("bigfile.dat", "r+")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
IOError: [Errno 22] Invalid argument: 'bigfile.dat'

which probably means that this fix

http://www.python.org/sf/538827

is partially responsible for the misleading error message.

(the cause of this seems to be that when you open a text file for updating, the
CRT check if there's a chr(26) at the end of the file, but the 32-bit lseek API
doesn't support seeking to positions larger than 2^31-2)

</F>

Using MSVC++ .NET 2003 compiler (if I did it all the right way):

fstm = fopen("bigfile.dat","r+");
if(fstm == 0) { printf( " ErrNo: %i \n", errno ); }
// ^-- prints :
// on "r+" with too large file: 22 (EINVAL-Invalid argument)
// on non-existing file : 2 (ENOENT-no such file)
// on bad mode string spec. : 0 (??? why not EINVAL ...)

So there _is_ a way to distinguish the different problems occurred while
opening the file. The error message comes from Python (errnomodule.c),
not from Windows(errno.h). Concluding from this it becomes evident for
me, that this misleading error message is Python fault (even if
originated by misleading errno values set after fopen in the MSVC++
environment and Windows), right?
Probably also in Python 2.5?

Claudio Grondi

[Georg Brandl]

Claudio Grondi wrote:

Tim Peters wrote:

>[Claudio Grondi]

>>Here an example of what I mean
(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
large file):

>>f = file('veryBigFile.dat','r')
>>f = file('veryBigFile.dat','r+')

Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
f = file('veryBigFile.dat','r+')
IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

Is it a BUG or a FEATURE?

Assuming the file exists and isn't read-only, I bet it's a Windows
bug, and that if you open in binary mode ("r+b") instead I bet it goes
away (this wouldn't be the first large-file text-mode Windows bug).

I knew already that 'r+b' fixes it. Yes, you have won the bet :) .

I suppose, like you do, that because there is a difference between text
and binary files on Windows and the text files are e.g. opened being
buffered using a 32-bit buffer pointer, this fails on too large NTFS files.

I could also imagine that Python tries to buffer the text file and fails
because it uses the wrong pointer size when asking Windows for the
content. I have not yet looked into the C-code of Python - any hint
which file I should take a closer look at?

That would be Objects/fileobject.c. And no, on just open()ing the file,
Python does nothing more than fopen() (or some Windows equivalent).

Georg