strange file.write() behavior on windows: $ConvertToNonre sident, $ReplaceAttribu te2

[welch]

while taking some rough disk performance measures on windows machines,
and snooping with FileMon, i've noticed some odd behavior

here's the little nul-writer i'm running:
def writeTest(nBloc ks, blockSize):
"""write nBlocks*blockSi ze nuls to a file"""
f = open("writeTest .out", "wb")
nulBlock = '\0'*blockSize
while nBlocks:
f.write(nulBloc k)
nBlocks -= 1
f.close()

when i invoke this with a blockSize of 64K or less, no surprises.
here's a row of FileMon output for the 64K block size:
2:44:48 PM python.exe:3064 WRITE C:\writeTest.ou t SUCCESS Offset:
214106112 Length: 65536

when i invoke this with larger blocksizes (say, 655360), FileMon output
looks like this:
2:37:46 PM python.exe:3064 WRITE C:\writeTest.ou t Offset: 140902400
Length: 655360
2:37:46 PM python.exe:3064 WRITE C:\$ReplaceAttr ibute2 SUCCESS Offset:
140902400 Length: 65536
....(the $ReplaceAttribu te2 line repeats several times, writing 65536
and advancing the offset by 65536 each time)

sometimes, depending on the phase of the moon, instead of
$ReplaceAttribu te2 the write goes to $ConvertToNonre sident. within a
single run of writeTest the write always goes to the same one,
whichever one that is. these runs are always slower (sometimes greatly)
than writing the same with 64K blocks, even though many fewer
file.write()'s are being issued because of the larger block size.

finally, where you'd expect an even 10 of the WRITE
C:\$ReplaceAttr ibute2 lines per WRITE C:\writeTest.ou t lines in the
example above, instead FileMon reports 8 lines for the first, 6 for the
second, 8 for the third, 6 for the fourth, etc... i've no idea if this
means FileMon is missing messages, but this pattern is absolutely
consistent across every run i've made, on both xp and win2k3 server
machines.

a look at the python fileobject.c source shed no light for me, anyone
know what could be going on? the "equivalent " c version of writeTest
using fwrite() shows a succession of 1K blocks writing to the named
file (even when fwrite is given 64K blocks), and no mysterious
$ReplaceAttribu te2 lines

-- w

[jepler]

This is really just a snide joke at the expense of Microsoft, but have you
checked the MSDN documentation about ConvertToNonres ident or ReplaceAttribut e2?

Your search - site:msdn.micro soft.com ConvertToNonres ident OR
ReplaceAttribut e2 - did not match any documents.
-- google.com

Internally, Python also simply calls fwrite(). Are you using the same C
runtime for both sets of tests?

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDe8alJd0 1MZaTXX0RAjpDAK CPpv7J3sJ21Yszx DjknKG0dEvjRwCf QfP5
oU/o2FKjl0cDqqr3dD ufeyU=
=C/YG
-----END PGP SIGNATURE-----

[jepler]

This is really just a snide joke at the expense of Microsoft, but have you
checked the MSDN documentation about ConvertToNonres ident or ReplaceAttribut e2?

Your search - site:msdn.micro soft.com ConvertToNonres ident OR
ReplaceAttribut e2 - did not match any documents.
-- google.com

Internally, Python also simply calls fwrite(). Are you using the same C
runtime for both sets of tests?

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDe8alJd0 1MZaTXX0RAjpDAK CPpv7J3sJ21Yszx DjknKG0dEvjRwCf QfP5
oU/o2FKjl0cDqqr3dD ufeyU=
=C/YG
-----END PGP SIGNATURE-----

[Fredrik Lundh]

"welch" <we************ ***@gmail.com> wrote:

while taking some rough disk performance measures on windows machines,
and snooping with FileMon, i've noticed some odd behavior sometimes, depending on the phase of the moon, instead of
$ReplaceAttribu te2 the write goes to $ConvertToNonre sident. within a
single run of writeTest the write always goes to the same one,
whichever one that is. these runs are always slower (sometimes greatly)
than writing the same with 64K blocks, even though many fewer
file.write()'s are being issued because of the larger block size.

finally, where you'd expect an even 10 of the WRITE
C:\$ReplaceAttr ibute2 lines per WRITE C:\writeTest.ou t lines in the
example above, instead FileMon reports 8 lines for the first, 6 for the
second, 8 for the third, 6 for the fourth, etc... i've no idea if this
means FileMon is missing messages, but this pattern is absolutely
consistent across every run i've made, on both xp and win2k3 server
machines.

a look at the python fileobject.c source shed no light for me, anyone
know what could be going on? the "equivalent " c version of writeTest
using fwrite() shows a succession of 1K blocks writing to the named
file (even when fwrite is given 64K blocks), and no mysterious
$ReplaceAttribu te2 lines

you're seeing activities by the Windows caching system (c:\$ stuff is various
NTFS data streams). unless you're writing low-level drivers, you don't really
have to care about what it does, and when it does what...

</F>

[Fredrik Lundh]

"welch" <we************ ***@gmail.com> wrote:

while taking some rough disk performance measures on windows machines,
and snooping with FileMon, i've noticed some odd behavior sometimes, depending on the phase of the moon, instead of
$ReplaceAttribu te2 the write goes to $ConvertToNonre sident. within a
single run of writeTest the write always goes to the same one,
whichever one that is. these runs are always slower (sometimes greatly)
than writing the same with 64K blocks, even though many fewer
file.write()'s are being issued because of the larger block size.

finally, where you'd expect an even 10 of the WRITE
C:\$ReplaceAttr ibute2 lines per WRITE C:\writeTest.ou t lines in the
example above, instead FileMon reports 8 lines for the first, 6 for the
second, 8 for the third, 6 for the fourth, etc... i've no idea if this
means FileMon is missing messages, but this pattern is absolutely
consistent across every run i've made, on both xp and win2k3 server
machines.

a look at the python fileobject.c source shed no light for me, anyone
know what could be going on? the "equivalent " c version of writeTest
using fwrite() shows a succession of 1K blocks writing to the named
file (even when fwrite is given 64K blocks), and no mysterious
$ReplaceAttribu te2 lines

you're seeing activities by the Windows caching system (c:\$ stuff is various
NTFS data streams). unless you're writing low-level drivers, you don't really
have to care about what it does, and when it does what...

</F>