Python 2.5.1 broken os.stat module

[Joe Salmeri]

I just upgraded from Python 2.4.2 to Python 2.5.1 and have found some
unexpected behavior that appears to be a bug in the os.stat module.

My OS is Windows XP SP2 + all updates.

I have several programs that have worked flawlessly on all previous Python
versions for years and they are now producing incorrect results in the code
that uses os.stat.

Searching through the 2.5.1 release notes I found the following:

Use Win32 API to implement os.stat/fstat. As a result, subsecond
timestamps
are reported, the limit on path name lengths is removed, and stat
reports
WindowsError now (instead of OSError).

*********************
* Overview of the problem:
*********************

Reviewing my code it seems that it should still work with the 2.5.1 os.stat
changes however that does not appear to be the case.

Timestamps reported by os.stat() are no longer correct and the results are
not even consistent.

In my first test case ALL 3 timestamps reported by Python are 1 hour less
than the actual timestamp on the file.

In my second test case the created and last accessed timestamps reported by
Python are correct but the last write timestamp is 1 hour less than the
actual timestamp on the file.

As a third test I looked at the last write timestamps on approximately
21,000 files.
Python determined wrong last write timestamp on approximately 1581 files.

Assuming there is no error in the following code that prints out the
timestamps using the new return value from os.stat() then it would appear
that the 2.5.1 os.stat changes have a bug.

print 'Creation Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_CTIME]))
print 'Last Access Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_ATIME]))
print 'Last Write Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_MTIME]))
*********************
* Detailed test results
*********************

To demonstrate the problem I have created the following test.

Here are the files that will be used in my test and their associated
timestamps as reported the the dir command.

01/02/2003 12:34 PM 0 broke_test
03/06/2007 05:24 PM 3,497,177 broke_test2
05/31/2007 04:35 PM 254 runtest.cmd
05/31/2007 04:31 PM 513 broke.py

The file named broke_test has a timestamp of 01/02/2003 12:34:56 (dir does
not show seconds).

The runtest.cmd script shows the created, last accessed, and last write
timestamps as reported by the dir command (and also verified via right
clicking on the file and selecting properties in Explorer.

------ >>START runtest.cmd script <<<<
@Echo Off

echo Create TimeStamp reported by dir command
dir /tc %1

echo Last Access TimeStamp reported by dir command
dir /ta %1

echo Last Write TimeStamp reported by dir command
dir /tw %1

echo Python 2.5.1 timestamp info
broke.py %1

------ >>END runtest.cmd script <<<<

The broke.py script prints out the created, last accessed, last write
timestamps as Python sees them.

------ >>START broke.py script <<<<
import sys
import os
import stat
import time

file_name = sys.argv[1]

file_stats = os.stat(file_name)

print
print 'File Name : %s' % (file_name)
print
print 'Creation Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_CTIME]))
print 'Last Access Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_ATIME]))
print 'Last Write Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_MTIME]))
print

------ >>END broke.py script <<<<

#
# Test 1 on file broke_test
#

runtest broke_test

Create TimeStamp reported by dir command

01/02/2003 12:34 PM 0 broke_test

Last Access TimeStamp reported by dir command

01/02/2003 12:34 PM 0 broke_test

Last Write TimeStamp reported by dir command

01/02/2003 12:34 PM 0 broke_test

Python 2.5.1 timestamp info

File Name : broke_test

Creation Time: 01/02/2003 11:34:56 -- Python results are WRONG hours
reported are 1 hour short
Last Access Time: 01/02/2003 11:34:56 -- Python results are WRONG hours
reported are 1 hour short
Last Write Time: 01/02/2003 11:34:56 -- Python results are WRONG hours
reported are 1 hour short

#
# Test 2 on file broke_test2
#

runtest broke_test2

Create TimeStamp reported by dir command

05/31/2007 04:26 PM 3,497,177 broke_test2

Last Access TimeStamp reported by dir command

05/31/2007 04:26 PM 3,497,177 broke_test2

Last Write TimeStamp reported by dir command

03/06/2007 05:24 PM 3,497,177 broke_test2

Python 2.5.1 timestamp info

File Name : broke_test2

Creation Time: 05/31/2007 16:26:36 -- Python results are correct
Last Access Time: 05/31/2007 16:26:38 -- Python results are correct
Last Write Time: 03/06/2007 16:24:21 -- Python results are WRONG hours
reported are 1 hour short

[Tony Meyer]

On Jun 1, 9:16 am, "Joe Salmeri" <[e-mail]:

I just upgraded from Python 2.4.2 to Python 2.5.1 and have found some
unexpected behavior that appears to be a bug in the os.stat module.

Have you read this thread?

http://groups.google.com/group/comp....66283a8253cafb

I suspect that it explains your problem.

Cheers,
Tony

[Joe Salmeri]

Hi Tony,

I still believe there is a problem.

I was searching for os.stat problems so I hadn't seen that one yet. (THANKS)

I just read that thread but it seems that the conclusion was that this was a
bug in a Microsoft c runtime library.

Here's why I think there is still a problem:

I created a file and specifically set the created date, last accessed date
and last write date to

01/02/2003 12:34:56

After setting the date and time on the file I verified the following:

The dir /tc command shows the correct Create date of 01/02/2003 12:34
The dir /ta command shows the correct Access date of 01/02/2003 12:34
The dir /tw command shows the correct Write date of 01/02/2003 12:34

Looking at the properties of the file in Windows Explorer also shows that
all 3 dates match.

When Python 2.4.2 os.stat is used to get those timestamps all 3 are CORRECT

When Python 2.5.1 os.stat is used to get those timestamps all 3 are WRONG
and they are off by exact 1 hour

01/02/2003 11:34:56 instead of 01/02/2003 12:34:56

In the case of my above test I know exactly what the timestamp on the file
is because I manually set it so that all 3 timestamps are the same.
Since Python 2.5.1 does not return the known values for that files
timestamps how can it not be a Python 2.5.1 bug?

Further testing shows that the results are somewhat inconsistent, many times
the create and access date are correct but the Last Write timestamp is
wrong. It is generally off by one hour but I have seen situations where it
was +1 hour and other situations where it was -1 hour.

I even found situations where the python timestamp was 1 minute later. (I
know about the 2 second timestamps on FAT, all my filesystems are NTFS). I
just found a situation where the python timestamp was 02:51 PM and the
windows timestamp was 02:12 PM. DST or timezone changes are not going to
make the results be off by 39 minutes? (My timezone is GMT - 5:00).

As an additional test I wrote the following Python program which takes a
directory name as a paramater.
It retrieves all 3 dates for all files in all directories and subdirectories
for the specified directory name.
First it gets the dates using Python and os.stat, then it runs the dir /tc,
dir /ta, and dir /tw commands on the file.
Finally it compares the dates and times returned by Python against the dates
and times that the Windows dir command is reporting.

If you run the program using Python 2.4.2 then ALL dates and times returned
by Python 2.4.2 correctly match the dates and times that Windows reports.

If you run the program using Python 2.5.1 against the same directory then
you get intermittient results.

As a quick test I ran it against a directory on my home directory.

Python 2.4.2 all dates matched between what the Windows dir command reported
and what Python 2.4.2 reported.

Python 2.5.1 had the following differences (more than 50% of the time it was
wrong)

Total Files Processed : 149

Matched All 3 Dates : 70
Did NOT Match All 3 Dates: 79

Matched Create Date : 69
Did NOT Match Create Date: 10

Matched Access Date : 59
Did NOT Match Access Date: 20

Matched Write Date : 11
Did NOT Match Write Date : 68

Here's the source for the check_dates.py program. Run it for youself and
see what your results are.

import os
import stat
import sys
import time

if len(sys.argv) == 1:
print
print '%s path_to_check' % (sys.argv[0])
print
raise SystemExit
else:
path_to_check = sys.argv[1]

file_count = 0

file_match_count = 0
file_no_match_count = 0

create_match_count = 0
create_no_match_count = 0

access_match_count = 0
access_no_match_count = 0

write_match_count = 0
write_no_match_count = 0

for root, dirs, files in os.walk(r'%s' % path_to_check):
for file in files:
file_count = file_count + 1
file_name = os.path.join(root, file)

create_ts_match = False
access_ts_match = False
write_ts_match = False

file_stats = os.stat(file_name)

python_create_ts = time.strftime('%m/%d/%Y %I:%M %p',
time.localtime(file_stats[stat.ST_CTIME]))
python_access_ts = time.strftime('%m/%d/%Y %I:%M %p',
time.localtime(file_stats[stat.ST_ATIME]))
python_write_ts = time.strftime('%m/%d/%Y %I:%M %p',
time.localtime(file_stats[stat.ST_MTIME]))

win_create_ts = os.popen('dir /a /tc "%s"' %
(file_name)).readlines()[5][0:20]
win_access_ts = os.popen('dir /a /ta "%s"' %
(file_name)).readlines()[5][0:20]
win_write_ts = os.popen('dir /a /tw "%s"' %
(file_name)).readlines()[5][0:20]

if python_create_ts == win_create_ts: create_ts_match = True
if python_access_ts == win_access_ts: access_ts_match = True
if python_write_ts == win_write_ts: write_ts_match = True

if create_ts_match and access_ts_match and write_ts_match:
# All timestamps match
file_match_count = file_match_count + 1
else:
file_no_match_count = file_no_match_count + 1
print
print 'File Name : %s' % (file_name)
print

if create_ts_match:
create_match_count = create_match_count + 1
else:
create_no_match_count = create_no_match_count + 1

print 'python_create_ts: %s' % (python_create_ts)
print 'win_create_ts : %s' % (win_create_ts)
print

if access_ts_match:
access_match_count = access_match_count + 1
else:
access_no_match_count = access_no_match_count + 1

print 'python_access_ts: %s' % (python_access_ts)
print 'win_access_ts : %s' % (win_access_ts)
print

if write_ts_match:
write_match_count = write_match_count + 1
else:
write_no_match_count = write_no_match_count + 1

print 'python_write_ts : %s' % (python_write_ts)
print 'win_write_ts : %s' % (win_write_ts)
print

#
# Show Count Results
#

print 'Total Files Processed : %s' % (file_count)
print
print 'Matched All 3 Dates : %s' % (file_match_count)
print 'Did NOT Match All 3 Dates: %s' % (file_no_match_count)
print
print 'Matched Create Date : %s' % (create_match_count)
print 'Did NOT Match Create Date: %s' % (create_no_match_count)
print
print 'Matched Access Date : %s' % (access_match_count)
print 'Did NOT Match Access Date: %s' % (access_no_match_count)
print
print 'Matched Write Date : %s' % (write_match_count)
print 'Did NOT Match Write Date : %s' % (write_no_match_count)
print



"Tony Meyer" <to********@gmail.comwrote in message
news:11*********************@x35g2000prf.googlegro ups.com...

On Jun 1, 9:16 am, "Joe Salmeri" <[e-mail]:

>I just upgraded from Python 2.4.2 to Python 2.5.1 and have found some
unexpected behavior that appears to be a bug in the os.stat module.

Have you read this thread?

http://groups.google.com/group/comp....66283a8253cafb

I suspect that it explains your problem.

Cheers,
Tony

[Joe Salmeri]

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

I created a file and specifically set the created date, last accessed date

and last write date to

01/02/2003 12:34:56

How did you do that?

In the case of my above test I know exactly what the timestamp on the file
is because I manually set it so that all 3 timestamps are the same.
Since Python 2.5.1 does not return the known values for that files
timestamps how can it not be a Python 2.5.1 bug?

The program you've been using to set the time stamp to the old date
may not be working correctly.

Further testing shows that the results are somewhat inconsistent, many times
the create and access date are correct but the Last Write timestamp is
wrong. It is generally off by one hour but I have seen situations where it
was +1 hour and other situations where it was -1 hour.

In these cases, always ask yourself whether the time-zone of that time
stamp is different from the current time zone. In particular, did
these old time stamps live in non-DST, and the current time zone is
a DST one?

I even found situations where the python timestamp was 1 minute later. (I
know about the 2 second timestamps on FAT, all my filesystems are NTFS). I
just found a situation where the python timestamp was 02:51 PM and the
windows timestamp was 02:12 PM. DST or timezone changes are not going to
make the results be off by 39 minutes? (My timezone is GMT - 5:00).

Right. If that is reproducable, it is a bug. Please create a zip file
containing this file, and submit a bug report to sf.net/projects/python.

Regards,
Martin

[Joe Salmeri]

""Martin v. Löwis"" <ma****@v.loewis.dewrote in message
news:46**********************@news.freenet.de...

>I created a file and specifically set the created date, last accessed
date
and last write date to

01/02/2003 12:34:56

How did you do that?

I used a "touch" utility to set the dates but let's try it a different way.

Let's use Python 2.5.1 to set the timestamps

---->set file timestamp
import pywintypes
import win32file
import sys

file_name = sys.argv[1]
new_timestamp = pywintypes.Time([2003, 1, 2, 12, 34, 56])

f = win32file.CreateFile(file_name, win32file.GENERIC_WRITE, 0, None,
win32file.OPEN_EXISTING, 0, 0)
win32file.SetFileTime(f, new_timestamp, new_timestamp, new_timestamp)
f.Close()

Next I created an empty file with notepad.

Then I used the above code to set the 3 timestamps.

Let's confirm that Windows has the timestamps set to what we set them too.

Here are the timestamps as reported by Windows:

dir /tc joe.txt

01/02/2003 12:34 PM 0 joe.txt

dir /ta joe.txt

01/02/2003 12:34 PM 0 joe.txt

dir /tw joe.txt

01/02/2003 12:34 PM 0 joe.txt

Looks like the python code worked to correctly set the timestamps
(Note you can also verify them using properties on the file in Windows
Explorer)

Here is some python code to print out the timestamps:

import os
import stat
import sys
import time

file_name = sys.argv[1]

file_stats = os.stat(file_name)

print 'Creation Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_CTIME]))
print 'Last Access Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_ATIME]))
print 'Last Write Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_MTIME]))

Now let's see what Python 2.4.2 says about the file

Creation Time: 01/02/2003 12:34:56
Last Access Time: 01/02/2003 12:34:56
Last Write Time: 01/02/2003 12:34:56

Looks like Python 2.4.2 is reporting the timestamps correctly

Now let's see what Python 2.5.1 says

Creation Time: 01/02/2003 11:34:56
Last Access Time: 01/02/2003 11:34:56
Last Write Time: 01/02/2003 11:34:56

All times are off by 1 hour!

Let's re-verify that Windows still says the timestamps are 01/02/2003
12:34:56.

dir /tc joe.txt

01/02/2003 12:34 PM 0 joe.txt

dir /ta joe.txt

01/02/2003 12:34 PM 0 joe.txt

dir /tw joe.txt

01/02/2003 12:34 PM 0 joe.txt

Windows still says the times are what we set them too.

Opening up Windows Explorer also confirms that the times are still
01/02/2003 12:34:56.

My text editor has a was to display the file names and timestamp info just
like a dir command

Let's see what it says:

h: joe .txt 0 1/02/03 12:34 \

The text editor is reporting the correct timestamps too.

So far everything else I try seems to report the timestamps correctly except
for Python 2.5.1.

It is difficult for me to believe that EVERYTHING else is wrong and Python
2.5.1 is correct, especially when you consider the fact that the code in
Python 2.5.1 that performs this functionality is not the same code that was
used in previous Python versions.

Normally if something is not working, the biggest suspect is the last thing
changed.

In my other msg I posted a program that compares the timestamps of a dir
/tc, dir /ta, and dir /tw for all files against what Python is reporting.

This allows us to easily verify what Windows says the timestamps are versus
what Python says they are.

Python 2.4.2 ALWAYS got it correct.

Python 2.5.1 gets it wrong as much as 50% of the time.

So unless you are saying that Windows is wrong, Python 2.5.1 has to be wrong
since it does not compare to what Windows reports. Since everything else I
have tried matches what Windows reports that leads me to believe that Python
2.5.1 has to be wrong.

>I even found situations where the python timestamp was 1 minute later.
(I
know about the 2 second timestamps on FAT, all my filesystems are NTFS).
I
just found a situation where the python timestamp was 02:51 PM and the
windows timestamp was 02:12 PM. DST or timezone changes are not going
to
make the results be off by 39 minutes? (My timezone is GMT - 5:00).

Right. If that is reproducable, it is a bug. Please create a zip file
containing this file, and submit a bug report to sf.net/projects/python.

Please see my other msg, I ran the program in that msg against the
Python\lib directory: I tried it on multiple machines and found the same
problems occurring. I already submitted a bug report with the sample code
that demonstrates the problem.

All Python 2.4.2 timestamps match what Windows reports

The Python 2.5.1 timestamps vary
Sometimes python is +1 minute
Sometimes python is -1 hour
Sometimes python is +59 minutes
Sometimes python is +42 minutes
Sometimes python is +37 minutes
Sometimes python +2 minutes
Sometimes python +30 minutes
Sometimes python +3 minutes
Sometimes python +33 minutes

There is no way a timezone difference could explain those variations.

Bottom line is that Python should be reporting what Windows reports.

Even if it turns out that Windows has a bug and is wrong I still think that
Python should be reporting the same thing because what Windows reports is
what is expected. If Python reports something different (even if it was
correct) the information is not useful if it differs from what everything
else is reporting.

I hope I have provided enough information for you to reproduce the bug so
that a solution can be found.

Thanks for your time investigating this.

[Terry Reedy]

"Joe Salmeri" <Jo********@hotmail.comwrote in message
news:y7******************************@comcast.com. ..
| I hope I have provided enough information for you to reproduce the bug so
| that a solution can be found.

If, when discussion is concluded here, you still think there is a bug, file
a report on SF. Make a concise summary in the comments section and attach
a file with the output from your experiments. You may have to submit the
attachment separately after first submitting the report.

Keep in mind that all the 'you's who might fix this are volunteers just
like you.

tjr

[Joe Salmeri]

Hi Terry,

If, when discussion is concluded here, you still think there is a bug,
file
a report on SF. Make a concise summary in the comments section and attach
a file with the output from your experiments. You may have to submit the
attachment separately after first submitting the report.

I have submitted the bug report and test script to SF.

I purposely used the Python\lib directory for my test so that anyone could
run the test and get the results.

Keep in mind that all the 'you's who might fix this are volunteers just
like you.r

Understood, that's why I went to such great time and effort to create the
scripts which demonstrate the problem.

I'd fix it if I could but think that it would be better for the person that
made the change in os.stat for Python 2.5.1 since they would be much more
aware of the extent of that change.

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

Reviewing my code it seems that it should still work with the 2.5.1 os.stat

changes however that does not appear to be the case.

Timestamps reported by os.stat() are no longer correct and the results are
not even consistent.

I don't think this is the case. The results Python reports are correct
and consistent.

In my first test case ALL 3 timestamps reported by Python are 1 hour less
than the actual timestamp on the file.

No. Instead, your *rendering* of the time-stamp is 1 hour less than
what you expect to see. This is because you render the time stamp
according to the current time zone (which is daylight-saving),
not according to the time-zone that was in effect when the file
was created (which was regular time).

Assuming there is no error in the following code that prints out the
timestamps using the new return value from os.stat() then it would appear
that the 2.5.1 os.stat changes have a bug.

print 'Creation Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_CTIME]))
print 'Last Access Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_ATIME]))
print 'Last Write Time: %s' % time.strftime('%m/%d/%Y %H:%M:%S',
time.localtime(file_stats[stat.ST_MTIME]))

It's not clear whether it's an error, however, localtime() does
something different from what dir does.

Regards,
Martin

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

>How did you do that?

>
I used a "touch" utility to set the dates but let's try it a different way.

What touch utility specifically? Is source code available of that tool?

Looks like Python 2.4.2 is reporting the timestamps correctly

Unfortunately, no. Python 2.4 (and earlier) has an error that the
time stamps it reports may be off by an hour. This error is offset
by the fact that your rendering through localtime() gives result
different by one hour also, giving you the impression that Python
2.4 returns the correct result.

Now let's see what Python 2.5.1 says

Creation Time: 01/02/2003 11:34:56
Last Access Time: 01/02/2003 11:34:56
Last Write Time: 01/02/2003 11:34:56

All times are off by 1 hour!

The rendered times, yes. They are still correct - they are just
off by 1 hour, because your rendering algorithm is different from
the one that dir uses. It was 11:34 at the time when the file
was created, as there was no daylight-saving time in effect
in January 2003. According to your current time zone (which does
have daylight-saving time), it was 12:34 when the file was
created.

h: joe .txt 0 1/02/03 12:34 \

The text editor is reporting the correct timestamps too.

Define "correct". According to your current time zone, or taking into
account that the DST offset was different when the file was created?

Python 2.4.2 ALWAYS got it correct.

Unfortunately, it didn't. That was one of the reasons to change it.

So unless you are saying that Windows is wrong, Python 2.5.1 has to be wrong
since it does not compare to what Windows reports.

Windows isn't wrong. It just uses a different algorithm to render
historical time stamps. Neither approach is right or wrong:
- if you use the current time zone, you display a time-stamp that
wasn't the reading of the wall clock when the action happened
- if you adjust for DST changes, you get ambiguous time stamps,
as "2:30 am" may denote two different points in time on the
day when daylight saving time returns to normal time.

The one true way to render time stamps is in UTC/GMT/Zulu time :-)

Even if it turns out that Windows has a bug and is wrong I still think that
Python should be reporting the same thing because what Windows reports is
what is expected. If Python reports something different (even if it was
correct) the information is not useful if it differs from what everything
else is reporting.

For the cases where there is a 1h difference, Python reports *exactly*
what Windows reports. Python 2.4 didn't.

Regards,
Martin

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

File Name : P:\sw\Python\Lib\cgi.pyc

>
python_access_ts: 05/31/2007 07:17 PM
win_access_ts : 05/31/2007 06:35 PM

File Name : P:\sw\Python\Lib\code.pyc

python_access_ts: 05/30/2007 07:59 PM
win_access_ts : 05/30/2007 07:22 PM

I have analyzed these cases in more detail, and found
that Windows (dir and Explorer) indeed reports incorrect
last-access times. This is documented in

http://www.microsoft.com/resources/d....mspx?mfr=true

In essence, there are two places where the last-access
time is stored: In the file itself, and in the directory.
While the stamp in the file is constantly updated, the
stamp in the directory is only updated if the difference
exceeds one hour. This is done for performance reasons.

Python 2.5 reports the true last access time, whereas
dir, Explorer, and Python 2.4 report an access time
that isn't quite the *last* access time.

To see this for yourself, you can open a file,
and watch the last-access time as reported by Windows
not change, whereas Python reports a recent access
(if the time stamp changes indeed, wait a few minutes,
and access the file again - if the file hasn't been
accessed for more than one hour, the first access
will always update the time stamp right away).

Regards,
Martin

[Joe Salmeri]

It's not clear whether it's an error, however, localtime() does

something different from what dir does.

Hi Martin,

First off thank you for spending your time to investigate this bug with me.

Thanks for pointing out the issue with Windows XP caching the access
timestamps,
I was not aware of that.

The reality is that I am most interested in the last write time because that
is
the timestamp that Windows displays when you issue a dir command (WITHOUT a
/t option),
what Windows Explorer displays in detail view, and what every other program
I have
found so far references.

I re-reviewed all differences found by the program I posted for the
Python\lib directory
date comparison and if you eliminate access times all differences for
created and last
write that were reported were a difference of 1 hour.

Based on a bunch of testing that I did and some references listed below I
believe
that you are CORRECT, Python 2.5.1 is reporting the correct times and
Windows is
reporting the times incorrectly.

The short explaination of this issue is that the timestamp shown when you do
a dir
on a file that is on an NTFS volume changes by an hour when DST starts and
also when
DST ends, even though the file has NOT been modified!!! Note that this only
happens
if you have the setting turned on to automatically adjust for DST (this is
the default
in Windows).

Even though Python 2.5.1 is correct this presents a NEW problem because it
differs
from what XP SP 2 (wrongly) reports. I am sure this will cause a lot of
confusion
for people as it already appears that others have stumbled on this
difference.

Here are some tests I did which show the problem that Windows has.

Setup Test Environment Virtual machine
XP SP2
Set TimeZone to GMT with automatically adjust for DST turned on
Set Date and Time to 01/02/2007 12:00:00 AM
Create file with notepad named joe.txt

This establishes our baseline:

joe.txt last written 01/02/2007 12:00 AM GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/02/2007 12:00 AM 14 joe.txt

dir /tw

01/02/2007 12:00 AM 14 joe.txt

Python 2.4.2

Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:08

Python 2.5.1

Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:08

Great so far everybody is in agreement

Change Environment
Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
DST still set to automatically adjust
No changes to date and time at this point

With these changes joe.txt should now report 01/01/2007 07:00 PM since I
am 5 hours behind GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/01/2007 07:00 PM 14 joe.txt

dir /tw

01/01/2007 07:00 PM 14 joe.txt

Python 2.4.2

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08

Great so far everyone is still in agreement

Change Environment
Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am
located)
DST still set to automatically adjust
Change date to todays date 06/02/2007 and time 02:00:00 PM

This test simulates my machine moving in time forward from 01/1/2007 to
06/02/2007
with no changes being made on the computer. Imagine I turned it off on
01/02/2007
and went on a long vacation and just came home on 06/02/2007 and turned
it back on.

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/01/2007 08:00 PM 14 joe.txt

dir /tw

01/01/2007 08:00 PM 14 joe.txt

Python 2.4.2

Creation Time: 01/01/2007 20:00:04
Last Write Time: 01/01/2007 20:00:08

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08
This is where the problem occurs.
This file was originally created 01/02/2007 12:00 AM GMT
which is the same locally as 01/01/2007 07:00 PM EST
but now everything EXCEPT Python 2.51
is reporting that the file changed on 01/01/2007 08:00 PM EST

Well that couldn't have happened since no changes were made (remember I
was on vacation :-)).

Everything was fine until our current date time enters DST.
At that point the file now shows that it was last written an hour later.

The last write timestamp for the file should not change or be affected
because of the start/end of DST has occurred on my computer.

To the end user it appears the file was modifed when in fact it was not.

Python 2.5.1 is CORRECT
Windows is WRONG (dir /t, Explorer)
Python 2.4.2 is WRONG

One Possible Work Around for the problem:
Turn OFF the automatically adjust for DST setting
NOTE: Turning this off causes issues with time sync. See details at the
end.

With it turned off the timestamps are all correct
dir /tc

01/01/2007 07:00 PM 14 joe.txt

dir /tw

01/01/2007 07:00 PM 14 joe.txt
Python 2.4.2

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:08

I was concered that since I had the automatically adjust DST setting turned
on when
I originally started with the time set to GMT 12:00 AM that it might have
affected
my results so I reran the tests.

Setup Test Environment Virtual machine
XP SP2
Set TimeZone to GMT WITHOUT automatically adjust for DST
Set Date and Time to 01/02/2007 12:00:00 AM
Create file with notepad named joe.txt

This establishes our baseline:

joe.txt last written 01/02/2007 12:00 AM

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/02/2007 12:00 AM 16 joe.txt

dir /tw

01/02/2007 12:00 AM 16 joe.txt

Python 2.4.2

Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:09

Python 2.5.1

Creation Time: 01/02/2007 00:00:04
Last Write Time: 01/02/2007 00:00:09

Just like the first time everone is in agreement

Change Environment
Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
DST still NOT set to automatically adjust
No changes to date and time at this point

With these changes joe.txt should now report 01/01/2007 07:00 PM since I
am 5 hours behind GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/01/2007 07:00 PM 16 joe.txt

dir /tw

01/01/2007 07:00 PM 16 joe.txt

Python 2.4.2

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09

Just like the first time everyone still in agreement

Change Environment
Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am
located)
DST still NOT set to automatically adjust
Change date to todays date 06/02/2007 and time 02:00:00 PM

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

dir /tc

01/01/2007 07:00 PM 16 joe.txt

dir /tw

01/01/2007 07:00 PM 16 joe.txt

Python 2.4.2

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09

Everyone still in agreement.

This confirms that the bug is in Windows when the automatically adjust
for DST
setting is turned on. Windows should NOT be using the DST setting to
adjust
the file times. It makes it look like the files were modified when in
fact
they have not.

If you turn it on after the last test then you get the following results
which
again prove that Python 2.5.1 is correct and that Windows and Python
2.4.2 are
wrong.

dir /tc

01/01/2007 08:00 PM 16 joe.txt

dir /tw

01/01/2007 08:00 PM 16 joe.txt

Python 2.4.2

Creation Time: 01/01/2007 20:00:04
Last Write Time: 01/01/2007 20:00:09

Python 2.5.1

Creation Time: 01/01/2007 19:00:04
Last Write Time: 01/01/2007 19:00:09

The real issue now is that even though Python 2.5.1 is correct it is
confusing to
someone that is trying to compare their results to what Windows reports.

If you turn of automatically adjust DST and run the program I provided to
compare
the dates between windows and python in the other email message against the
Python\Lib directory then all create dates and all write dates between
Python and
Windows compare. Access dates do not match but you have already explained
why that
can occur (caching).

As shown above, one way to fix this is to turn OFF the automatically adjust
for
DST setting in Windows.

The problem with turning it OFF is that if you have automatic time sync
turned ON
when the time sync occurs and automatically adjust for DST is turned OFF and
it is DST
your time will be an hour off after the sync :-(.

Also any computers that are syncing their time with this computer that have
automatically adjust for DST turned ON will have their sync'd times one hour
off since this computer had automatically adjust for DST turned OFF.

The following link goes into even greater detail as to the source of the
problem
in Windows:

http://search.cpan.org/~shay/Win32-U...UTCFileTime.pm

Based on the above article it does not seem like Microsoft will be fixing
this
which leaves us stuck because Python and Windows are not reporting the same
values when DST starts/ends and you have automatically adjust for DST turned
ON.

Having this difference between Python and Windows makes it difficult to
write
code that depends on those values being in sync.

It makes me wonder if there should be an option in localtime to "fudge" the
dates
so that they match what Windows reports....

Thanks to all that assisted in investigating this.

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

The short explaination of this issue is that the timestamp shown when

you do a dir on a file that is on an NTFS volume changes by an hour
when DST starts and also when DST ends, even though the file has NOT
been modified!!! Note that this only happens if you have the setting
turned on to automatically adjust for DST (this is the default in
Windows).

You are misinterpreting what you are seeing. Windows is not claiming
that the modification time changes when DST starts. Instead, it is
claiming that the *same* time now has a *different* textual
representation, because the computer now has moved to a different
time zone.

When you explicitly change the time zone of your computer, you will
notice that all file time stamps immediately change also.

Even though Python 2.5.1 is correct this presents a NEW problem
because it differs from what XP SP 2 (wrongly) reports. I am sure
this will cause a lot of confusion for people as it already appears
that others have stumbled on this difference.

However, please understand that the problem is *not* caused by the
results returned from os.stat. Instead, the difference comes from
your choice of calling localtime() to break down the timestamp
to day, hour, and minute. If you write a different function
windows_localtime, that behaves the same way that the Windows
rendering behaves, you will get the same results.

To the end user it appears the file was modifed when in fact it was
not.

That's because the user is misinterpreting the data that Windows
reports. Windows isn't saying the file changed, Windows says that
the timezone changed.

To understand this, you have to truly grasp the notion of absolute
time. June 2, 2007, 9:40 is *not* a point in time. Instead, you
have to provide a timezone also to make it point in time. So Windows
is incorrect not giving the time zone; otherwise, it is correct.

This file was originally created 01/02/2007 12:00 AM GMT
which is the same locally as 01/01/2007 07:00 PM EST
but now everything EXCEPT Python 2.51
is reporting that the file changed on 01/01/2007 08:00 PM EST

Not true. Everything else is reporting that the time stamp is

01/01/2007 08:00 PM EDT

(where EDT is the Eastern Daylight Time).

Having this difference between Python and Windows makes it difficult
to write code that depends on those values being in sync.

No. Just write a windows_localtime function, and be done.

Regards,
Martin

[Joe Salmeri]

""Martin v. Löwis"" <ma****@v.loewis.dewrote in message
news:46***********************@news.freenet.de...

>The short explaination of this issue is that the timestamp shown when
you do a dir on a file that is on an NTFS volume changes by an hour
when DST starts and also when DST ends, even though the file has NOT
been modified!!! Note that this only happens if you have the setting
turned on to automatically adjust for DST (this is the default in
Windows).

You are misinterpreting what you are seeing. Windows is not claiming
that the modification time changes when DST starts. Instead, it is
claiming that the *same* time now has a *different* textual
representation, because the computer now has moved to a different
time zone.

When you explicitly change the time zone of your computer, you will
notice that all file time stamps immediately change also.

It appears that you may have missed part of my tests. Sorry it was such a
long reply but I was trying to provide a lot of detail so that others had a
clear understanding of what was going on.

I understand that the actual UTC time value for the file has *not* changed.

That local textual representation of the modification timestamp is what
people and programs look at and work with and is what is displayed by the
dir command and what Explorer displays.

Changing the timezone will defintely change the "textual representation" of
all timestamps just as you say (even though the actual UTC value has *not*
changed), however, when DST starts/ends the "textual representation" of the
timestamps on some files WILL ALSO CHANGE when the automatically adjust for
DST setting is turned on.

That is *not* my opinion or interpretation, it is clearly documented by
Microsoft and is also documented in the long article that I provided the
link for in my reply yesterday. It was also demonstrated in the second test
I provided yesterday and is also demonstrated in the article I provided the
link to.

In the following comments I am referring to the textual representation of
the timestamp that the user is working with (I understand that the interal
UTC timestamp for the file has *not* changed).

The issue occurs when the timestamp for the file is in a period when DST was
not in effect and DST is in effect for the current time OR when the
timestamp for the file is in a period when DST was in effect and DST is not
in effect for the current time.

While it is true that I did change the timezone in the first part of the
test (to establish the baseline), in the second part of the tests where the
difference you are referring to occured, the timezone was *not* changed, the
only thng that occured was that DST started.

In that test the file had a timestamp of 01/01/2007 07:00 PM as shown by the
dir command results.

When the date progressed to a point after DST had started Windows now
reports the timestamp on that *unmodified* file now is 01/01/2007 08:00 PM.

I did not change the timezone, the only thing that occurred was DST started.

The internal UTC timestamp did NOT change, the issue is in Windows textual
representation of that UTC timestamp to my local timezone and DST values,
Windows now reports that the modification timestamp on the file as 1 hour
later (again the actual UTC timestamp did *not* change).

To the end user the file "appears" to have been modified an hour later than
it was the day before even though it was not changed.

Over the years this issue has caused all sorts of problems for backup
programs and CVS (greatly detailed in the article I provided the link for).

>Even though Python 2.5.1 is correct this presents a NEW problem
because it differs from what XP SP 2 (wrongly) reports. I am sure
this will cause a lot of confusion for people as it already appears
that others have stumbled on this difference.

However, please understand that the problem is *not* caused by the
results returned from os.stat. Instead, the difference comes from
your choice of calling localtime() to break down the timestamp
to day, hour, and minute. If you write a different function
windows_localtime, that behaves the same way that the Windows
rendering behaves, you will get the same results.

>To the end user it appears the file was modifed when in fact it was
not.

That's because the user is misinterpreting the data that Windows
reports. Windows isn't saying the file changed, Windows says that
the timezone changed.

You mixed up my tests, in that case as shown above the timezone did *not*
change, the only thing that changed was that DST started and the file was
created during a time when DST was not in effect.

>Having this difference between Python and Windows makes it difficult
to write code that depends on those values being in sync.

No. Just write a windows_localtime function, and be done.

For those that are interested here is a function that you can use in place
of time.localtime that will return values that match what Windows
(incorrectly) reports.

There may be a better way to do this but so far I this code seems to work.

I ran a test against the Python\lib directory using the check_dates.py
program previously posted with it modified to use the localtime_windows
function in place of the time.localtime function calls and all create and
write timestamps returned now match what Windows (incorrectly) reports.

import datetime
import time

def localtime_windows(seconds):
value = time.localtime(seconds)
now = time.localtime()

# Compare DST in value versus DST now

if value[8] == 0 and now[8] == 1:
# When original date is NOT in DST but now date is in DST add 1 hour
new_value = datetime.datetime(value.tm_year, value.tm_mon,
value.tm_mday, value.tm_hour, value.tm_min, value.tm_sec) +
datetime.timedelta(hours=1)
value = new_value.timetuple()
elif value[8] == 1 and now[8] == 0:
# When original date is in DST and now date is NOT in DST subtract 1
hour
new_value = datetime.datetime(value.tm_year, value.tm_mon,
value.tm_mday, value.tm_hour, value.tm_min, value.tm_sec) +
datetime.timedelta(hours=-1)
value = new_value.timetuple()

return value

[Terry Reedy]

"Joe Salmeri" <Jo********@hotmail.comwrote in message
news:NY******************************@comcast.com. ..
| ""Martin v. Löwis"" <ma****@v.loewis.dewrote in message
| news:46***********************@news.freenet.de...
| >The short explaination of this issue is that the timestamp shown when
| >you do a dir on a file that is on an NTFS volume changes by an hour
| >when DST starts and also when DST ends, even though the file has NOT
| >been modified!!! Note that this only happens if you have the setting
| >turned on to automatically adjust for DST (this is the default in
| >Windows).
| >
| You are misinterpreting what you are seeing. Windows is not claiming
| that the modification time changes when DST starts. Instead, it is
| claiming that the *same* time now has a *different* textual
| representation, because the computer now has moved to a different
| time zone.

Joe, read the last sentence again. Most of your reply shows that you have
missed this basic point.

A 'time zone', as far as the math (and Windows) is concerned, *IS* an
offset from GMT. The shift from 'standard' to 'daylight' time is a
one-hour change in the offset. That has the same effect as moving to the
standard time zone one hour to the east. So Eastern Daylight Time is the
same as Atlantic Standard Time (or whatever Canadians call the next zone
for their maritime provinces). Similarly, 'Central Daylight Time' is an
alias for 'Eastern Standard Time'. CDT and EST both translate to -5 (I
believe) in Windows timezone lookup dictionary. Because there are multiple
names and abbreviations for most or all timezones, Windows has to
separately keep track of which one the user expects to see.

[In my opinion, the Congressional mandate that we pretend to live somewhere
to the east for over half the year is as arrogant and stupid as would be a
mandate that we use 'adjusted Fahrenheit' (F - 10 degrees) during the
summer so that temperatures 'seemed' lower. ;-]

| When you explicitly change the time zone of your computer, you will
| notice that all file time stamps immediately change also.

What Martin meant, of course, is the presented text representation thereof.

| It appears that you may have missed part of my tests. Sorry it was such
a
| long reply but I was trying to provide a lot of detail so that others had
a
| clear understanding of what was going on.
|
| I understand that the actual UTC time value for the file has *not*
changed.
|
| That local textual representation of the modification timestamp is what
| people and programs look at and work with and is what is displayed by the
| dir command and what Explorer displays.

And without a timezone attached, that 'local representation' is as
ambiguous as any measurement without units. Is 45 hot or cool?

| Changing the timezone will defintely change the "textual representation"
of
| all timestamps just as you say (even though the actual UTC value has
*not*
| changed), however, when DST starts/ends the "textual representation" of
the
| timestamps on some files WILL ALSO CHANGE when the automatically adjust
for
| DST setting is turned on.

To repeat, turning DST on *IS* a timezone change. So you are talking about
the *same* thing.

| In the following comments I am referring to the textual representation of
| the timestamp that the user is working with (I understand that the
interal
| UTC timestamp for the file has *not* changed).
|
| The issue occurs when the timestamp for the file is in a period when DST
was
| not in effect and DST is in effect for the current time OR when the
| timestamp for the file is in a period when DST was in effect and DST is
not
| in effect for the current time.

Because the timezones change. Geographically, UTC+X is the area where the
relevant authorities mandate that clocks be set to offset X from UTC. When
the clocks jump, those areas jump! In the US, at least, the UTC-6 area,
for instance, jumps from where it was to what was previously the UTC-7
area. This spring, when there were different jump dates around the world,
the overall configuration of zones (which areas had the same time) changed.
Yes, a mess

| While it is true that I did change the timezone in the first part of the
| test (to establish the baseline), in the second part of the tests where
the
| difference you are referring to occured, the timezone was *not* changed,
the
| only thng that occured was that DST started.

Which *is* a timezone change.

| In that test the file had a timestamp of 01/01/2007 07:00 PM as shown by
the
| dir command results.

As Martin said, that is not a timestamp. It is an information-stripped
representation thereof.

| When the date progressed to a point after DST had started Windows now
| reports the timestamp on that *unmodified* file now is 01/01/2007 08:00
PM.

Because Windows changed the timezone offset by one hour. Again, not a
timestamp but an alternate info-deleted representation thereof.

| I did not change the timezone, the only thing that occurred was DST
started.

Which *is* a timezone change.

| The internal UTC timestamp did NOT change, the issue is in Windows
textual
| representation of that UTC timestamp to my local timezone and DST values,
| Windows now reports that the modification timestamp on the file as 1 hour
| later (again the actual UTC timestamp did *not* change).

It is Windows' fault that it deletes the time scale indication. If there
were a 'Congressional Stupidty' temperature scale (F-10) and Windows
silently changed the scale for reporting historical temperature stamps
(perhaps of max CPU temp each your, auto recorded on machines with such
measurements), it would 'report' that such had changed, even thought they
had not.

| That's because the user is misinterpreting the data that Windows
| reports. Windows isn't saying the file changed, Windows says that
| the timezone changed.
|
| You mixed up my tests, in that case as shown above the timezone did *not*
| change, the only thing that changed was that DST started and the file was
| created during a time when DST was not in effect.

Need I repeat ;-?

Terry Jan Reedy

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

It appears that you may have missed part of my tests. Sorry it was such a

long reply but I was trying to provide a lot of detail so that others had a
clear understanding of what was going on.

Please understand that it is *extremely* tedious to follow your
messages. It would have been much better if they had been short and
to the point.

Changing the timezone will defintely change the "textual representation" of
all timestamps just as you say (even though the actual UTC value has *not*
changed), however, when DST starts/ends the "textual representation" of the
timestamps on some files WILL ALSO CHANGE when the automatically adjust for
DST setting is turned on.

Right, and that's because switching to DST means that YOU MOVE TO A
DIFFERENT TIME ZONE.

EST != EDT.

While it is true that I did change the timezone in the first part of the
test (to establish the baseline), in the second part of the tests where the
difference you are referring to occured, the timezone was *not* changed, the
only thng that occured was that DST started.

And that meant that the time zone *did* change.

When the date progressed to a point after DST had started Windows now
reports the timestamp on that *unmodified* file now is 01/01/2007 08:00 PM.

I did not change the timezone, the only thing that occurred was DST started.

So you did change the timezone.

Over the years this issue has caused all sorts of problems for backup
programs and CVS (greatly detailed in the article I provided the link for).

That is very hard to believe. CVS and backup programs use the UTC time
stamp, and completely ignore the timezone. So when you agree that the
UTC time stamp did not change, CVS and the backup programs will work
just fine.

There was a long-standing problem with changing time-stamps ON FAT.
On a FAT file system, the actual UTC time stamps change when the
timezone changes (including the change to and from DST). *That*
was a big problem for backup programs and CVS, and is now fixed
with NTFS.

You mixed up my tests, in that case as shown above the timezone did *not*
change, the only thing that changed was that DST started and the file was
created during a time when DST was not in effect.

Right, so the timezone did change.

Regards,
Martin

[Joe Salmeri]

| You are misinterpreting what you are seeing. Windows is not claiming
| that the modification time changes when DST starts. Instead, it is
| claiming that the *same* time now has a *different* textual
| representation, because the computer now has moved to a different
| time zone.

Joe, read the last sentence again. Most of your reply shows that you have
missed this basic point.

My reply specifically indicated that I was referring to the textual
presentation that Windows is showing.

I understand that a DST change and a timezone change produce the same
results on the offset from GMT but I don't consider them the same thing.

As I attempted to describe the issue with the timestamps changing when DST
starts/ends is not my *opinion* it is behavior that Microsoft (and others)
have documented (and I agree with).

Consider this situation from the end user perspective.

I create a file on 01/02/2007 at 07:00 PM

Windows shows the last write time as 01/02/2007 07:00 PM

I never make any changes to the file.

Fast forward in time and it is now 06/03/2007.

When I do a directory on the file Windows now shows the last write time as
01/02/2007 08:00 PM.

I understand WHY this is happening, but like many others do not agree with
the behavior.

The reason Windows does this is because although my timeszone did not change
(as far as I am concerned) DST has started (a timezone change as far as you
are concerned) so it now shows the time as 1 hour more than it used to
because the GMT offset has changed.

From the end user perspective it appears this file has now been modified an
hour later than it showed it was modifed before DST started.

Now let's look at another common user activity.

I watch this TV show on 01/02/2007 at 07:00 PM. It comes on every day at
07:00 PM

Fast forward in time it is now 06/03/2007.

When I watch that same show it comes on at 07:00 PM (even though DST has
started) and if I look the TV listings it still shows that the older show on
01/02/2007 came on at 07:00 PM. It did not show the older show now on as
08:00 PM which is what Windows is doing with the timestamp.

When you compare those two situations it does not make sense that Windows
shows a different timestamp.

I would recommend reading the links in the articles where Microsoft
documents the situation and developers for the CVS project discuss having to
deal with the problems.

Here's a clip from the article:

This behaviour is responsible for a flood of questions to the
various support lists for CVS, following the first Sunday in April and the
last Sunday in October, with
scores of people complaining that CVS now reports erroneously that
their files have been modified. This is commonly known as the "red file bug"
because the
WinCVS shell uses red icons to indicate modified files.

[Joe Salmeri]

Please understand that it is *extremely* tedious to follow your

messages. It would have been much better if they had been short and
to the point.

Sometimes it seems that it is impossible to please people.

When messages are short then people complain that they did not get
sufficient details, or examples.

I provide details and tests so that anyone could easily attempt to duplicate
exactly what I said was happening and you complain that they are too long.

Baring any typo mistakes on my part, performing the tests should produce
the exact results that I stated were happening.

That is very hard to believe. CVS and backup programs use the UTC time
stamp, and completely ignore the timezone. So when you agree that the
UTC time stamp did not change, CVS and the backup programs will work
just fine.

The CVS problems were mentioned in the article I linked to and were
discussed by people involved in CVS development. Whether you choose to
believe all those people or not is up to you.

See my reply to Terry, from the end user perspective, the Windows behavior
does not make sense.

Short enough?

[Joe Salmeri]

""Martin v. Löwis"" <ma****@v.loewis.dewrote in message
news:46**************@v.loewis.de...

>It appears that you may have missed part of my tests. Sorry it was such
a
long reply but I was trying to provide a lot of detail so that others had
a
clear understanding of what was going on.

Please understand that it is *extremely* tedious to follow your
messages. It would have been much better if they had been short and
to the point.

>Changing the timezone will defintely change the "textual representation"
of
all timestamps just as you say (even though the actual UTC value has
*not*
changed), however, when DST starts/ends the "textual representation" of
the
timestamps on some files WILL ALSO CHANGE when the automatically adjust
for
DST setting is turned on.

Right, and that's because switching to DST means that YOU MOVE TO A
DIFFERENT TIME ZONE.

EST != EDT.

>While it is true that I did change the timezone in the first part of the
test (to establish the baseline), in the second part of the tests where
the
difference you are referring to occured, the timezone was *not* changed,
the
only thng that occured was that DST started.

And that meant that the time zone *did* change.

>When the date progressed to a point after DST had started Windows now
reports the timestamp on that *unmodified* file now is 01/01/2007 08:00
PM.

I did not change the timezone, the only thing that occurred was DST
started.

So you did change the timezone.

>Over the years this issue has caused all sorts of problems for backup
programs and CVS (greatly detailed in the article I provided the link
for).

That is very hard to believe. CVS and backup programs use the UTC time
stamp, and completely ignore the timezone. So when you agree that the
UTC time stamp did not change, CVS and the backup programs will work
just fine.

There was a long-standing problem with changing time-stamps ON FAT.
On a FAT file system, the actual UTC time stamps change when the
timezone changes (including the change to and from DST). *That*
was a big problem for backup programs and CVS, and is now fixed
with NTFS.

>You mixed up my tests, in that case as shown above the timezone did *not*
change, the only thing that changed was that DST started and the file was
created during a time when DST was not in effect.

Right, so the timezone did change.

Regards,
Martin

[Joe Salmeri]

There is a conflict with the answers that you and Terry have given.

The original issue I raised was that I Python 2.5.1 and Windows did not have
the same textual represenation for a localize date.

You have stood true to the statements that Python 2.5.1 is correct and that
previous versions were wrong.

I agree with you that Python 2.5.1 is correct which means that Windows is
showing the incorrect dates.

Since that point in time you and Terry have disagreed with my statements
regarding timezones and DST.

All dates in my below comments are the textual representation of those dates
from Windows or from Python 2.5.1

For the test file created at 01/02/2007 07:00 PM.

From that date up until just before DST starts both Windows AND Python 2.5.1
will report that date as 01/02/2007 07:00 PM.

Once DST starts Python 2.5.1 will still report that date as 01/02/2007 07:00
PM, however Windows will now report that
date as 01/02/2007 at 08:00 PM.

I agree and believe that Python 2.5.1 is reporting the correct textual
representation of that timestamp, however, using
*your* logic that a DST change *is a timezone change* that would mean that
Python 2.5.1 is wrong and should be reporting the
date 1 hour later just as Windows is reporting.

You cannot have it both ways.

If a DST change is the same as a timezone change then Python 2.5.1 should be
reporting the dates that Windows reports.
If a DST change is NOT the same as a timezone change then Python 2.5.1
should be reporting the dates that it currently is.

[Joe Salmeri]

Perspective is often the source of problems with communication.

You view timezones and DST as offsets from GMT. I understand and respect
that perspective.

When I think of timezones and DST I think of the timezone setting and the
DST setting in Windows.

These settings are two separate settings in Windows that can be configured
individually and independent of each other.

When I think of a time I think of

somedate, sometime, Eastern Standard Time (EST)

OR

somedate, sometime, Eastern Standard Time Daylight Saving Time (EST DST)

If you view things from my perspective then I would hope my comments become
more clear.

Last night I was talking to my wife about this discussion (I need to get a
life :-)) and she came up with an analogy
that I like even better than my TV analogy.

Suppose she has a baby 01/02/2007 07:00 PM.

Today, a time after DST has started for this region someone asks her when
her baby was born.

Here answer would be 01/02/2007 07:00 PM.

If we ask Windows that same question it would be 01/02/2007 08:00 PM.

Please see my other response to Martin because using your logic of "a DST
change is a timezone change" that would mean that the textual representation
of dates by Python 2.5.1 is WRONG. I don't believe that to be true but you
cannot have it both ways, if a DST change is a timezone change then Python
should show the same dates as Windows does, if it is not the same then
Python should show the dates that it is.

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

Joe Salmeri schrieb:

There is a conflict with the answers that you and Terry have given.

No, there isn't. See some of my earlier replies: both windows and python
are correct, despite the fact they give different results.

When Windows renders a time stamp, it always uses the current UTC
offset. That is the only way to guarantee that a time stamp uniquely
identifies points in time: i.e. two files with the same time stamp
where also modified at the same time.

OTOH, Python's localtime function tries to apply the UTC offset that
was likely in force at the time indicated by the time stamp. I say
likely, because it probably doesn't know exactly when DST started in
what year (I might be wrong - Python defers to some Microsoft API
to perform the conversion). While this apparently better matches what
you expect, it also is ambiguous: two files authored at 2:15 a.m.
might still be an hour apart, if that is on the day when DST switches
to standard time.

The original issue I raised was that I Python 2.5.1 and Windows did not have
the same textual represenation for a localize date.

You have stood true to the statements that Python 2.5.1 is correct and that
previous versions were wrong.

Yes - but not because localtime reported the wrong value, but because
stat() didn't report UTC times.

I agree with you that Python 2.5.1 is correct which means that Windows is
showing the incorrect dates.

And yet, I claim they are both correct - just different.

Since that point in time you and Terry have disagreed with my statements
regarding timezones and DST.

I don't see how. Terry claims the Windows result is correct, and so do I.

From that date up until just before DST starts both Windows AND Python 2.5.1
will report that date as 01/02/2007 07:00 PM.

EST, that is.

Once DST starts Python 2.5.1 will still report that date as 01/02/2007 07:00
PM, however Windows will now report that
date as 01/02/2007 at 08:00 PM.

EDT.

01/02/2007 7:00 PM EST == 01/02/2007 8:00 PM EDT

So they print indeed the same output, even though the text is different.

I agree and believe that Python 2.5.1 is reporting the correct textual
representation of that timestamp, however, using
*your* logic that a DST change *is a timezone change* that would mean that
Python 2.5.1 is wrong and should be reporting the
date 1 hour later just as Windows is reporting.

No. Python prints the stamp according to the time zone that was in force
when the file was created. Of course, it can't be sure (since the
computer may have lived in an entirely different time zone at the
time), but it assumes that the computer has not moved, and it assumes
to know when DST started in 2007 (which it probably does know correctly;
I wouldn't be so sure about 1981).

You cannot have it both ways.

Sure can.

If a DST change is the same as a timezone change then Python 2.5.1 should be
reporting the dates that Windows reports.

You are misunderstanding what localtime() does.

If a DST change is NOT the same as a timezone change then Python 2.5.1
should be reporting the dates that it currently is.

You are misunderstanding that Windows does *not* use localtime(),
but windows_localtime(). Both consider the current timezone, but in
a different way.

Regards,
Martin

[Joe Salmeri]

I have tried (unsuccessfully) to get you to view things from the end user
perspective.

I wish that you would consider looking at what the end user sees because
that is what really matters.

Without end users we would not need to develop software would we?

This entire conversation was VERY nicely summarized (by you) with the
following statement:

01/02/2007 7:00 PM EST == 01/02/2007 8:00 PM EDT

I agree with that statement 100%.

Python displays the date as 01/02/2007 7:00 PM EST
Windows displays the date as 01/02/2007 8:00 PM EDT.

Python and Windows are BOTH correct!

However, there is a problem. Windows does *not* display the date like that.

Windows displays the date as 01/02/2007 08:00 PM.

Windows does *not* give *any* indication when it made the switch from EST to
EDT!!!

All the end user sees is that the file said it was last changed at 07:00 PM
one day and then the next
day after DST started it now says the file was last changed at 08:00 PM.

The user is left scratching their head.

I prefer not to leave my end users in that situation.

From Python I could resolve this using strftime with %Z to show the timezone
/ EST / EDT but
from Windows you cannot get it to show you the timezone / DST value.

The only place in Windows that I have found that indicates EST / EDT is in
Control Panel
Date and Time Properties.

Since the dir command and Explorer do not display the timezone / DST
information the end
experience is confusion when the EST / EDT switch occurs.

You understand what happend, I understand what happen, but the typical user
does not.

[Terry Reedy]

"Joe Salmeri" <Jo********@hotmail.comwrote in message
news:D_******************************@comcast.com. ..
|I have tried (unsuccessfully) to get you to view things from the end user
| perspective.

But that perspective is not directly relevant to *your* topic line. When
you make a claim that os.stat is 'broken' and bugged, you are making a
claim about the *programmer* experience -- in particular, experiencing a
discrepancy between performance and reasonable expectation based on the
manuals. Martin and I are both concerned about that particular claim
(which now appears to be resolved in the negative).

>From the developer point of view, you, the programmer, are the user. What

you present to your users (who may or may not be the 'end' users) is your
responsibility.

| However, there is a problem. Windows does *not* display the date like
that.

Python developers cannot fix that. Besides which, Python is a
cross-platform language, originally developed on *nix.

| The user is left scratching their head.
|
| I prefer not to leave my end users in that situation.

Then do not. I am sure Martin also agrees. But if you want to discuss
writing applications for end users, perhaps you should start a new thread
with a different subject line. It would attract different respondents.

Terry Jan Reedy

[Joe Salmeri]

But that perspective is not directly relevant to *your* topic line. When

you make a claim that os.stat is 'broken' and bugged, you are making a
claim about the *programmer* experience -- in particular, experiencing a
discrepancy between performance and reasonable expectation based on the
manuals. Martin and I are both concerned about that particular claim
(which now appears to be resolved in the negative).

The subject line reflects what *originally* appeared to be the source of the
problem since a comparison of the results did not match previous Python
versions.

For other developers that run into this difference (which did not occur
before) I believe it is benefitial to them to understand why the difference
is now occurring and why BOTH Python and Windows are displaying CORRECT
results even though they do NOT match what they display.

For those same developers, I also posted a localtime_windows much which
allows them to massage the values returned by Python localtime so that they
match the values that Windows displays if they have the need and/or
requirement to do so.

Often times in development what "appears" to be the source of the problem
turns out that it is not the actual problem that is occurring.

It is still beneficial to have that link from what "appears" to be the
problem to the details of what "actually" is the source of the problem.

For another developer that runs into this difference their most likely
initial conclusion would be that something in Python 2.5.1 is broke
especially when they research the changes in Python 2.5.1 and find that
os.stat was changed. As a general rule, when something is broke, suspect
the last thing that you did even if it does not seem like it should be the
source of the problem.

Python developers cannot fix that. Besides which, Python is a
cross-platform language, originally developed on *nix.

Really, I thought they could fix anything :-)