vc-cvs-parse-entry

vc-cvs-parse-entry

[martin rudalics]

When I open array.el whose entry in my ~/lisp/CVS/Entries shows as

/array.el/1.29/Wed Feb  1 09:17:44 2006//

`vc-cvs-parse-entry' compares

(nth 5 (file-attributes "~/lisp/array.el")))

which gives (17376 28328) on my system, with

(apply 'encode-time
       (parse-time-string
	(concat "Wed Feb  1 09:17:44 2006" " +0000")))

which gives (17376 31928).

The test for equality fails and array.el is classified as "edited"
although I never edited that file.  Apparently the code messes with
daylight savings time in some inappropriate fashion.  Files modified
_after_ the last change in DST are classified correctly.


Note that

(decode-time (nth 5 (file-attributes "~/lisp/array.el")))

gives (44 17 9 1 2 2006 3 nil 3600) while

       (parse-time-string
	(concat "Wed Feb  1 09:17:44 2006" " +0000"))

gives (44 17 9 1 2 2006 3 nil 0) on my system.

Re: vc-cvs-parse-entry

[Stefan Monnier]

> When I open array.el whose entry in my ~/lisp/CVS/Entries shows as
> /array.el/1.29/Wed Feb  1 09:17:44 2006//

> `vc-cvs-parse-entry' compares

> (nth 5 (file-attributes "~/lisp/array.el")))

> which gives (17376 28328) on my system, with

> (apply 'encode-time
>        (parse-time-string
> 	(concat "Wed Feb  1 09:17:44 2006" " +0000")))

> which gives (17376 31928).

> The test for equality fails and array.el is classified as "edited"
> although I never edited that file.  Apparently the code messes with
> daylight savings time in some inappropriate fashion.  Files modified
> _after_ the last change in DST are classified correctly.

What system is that (w32?)?
What does `cvs status <file>' say?


        Stefan

Re: vc-cvs-parse-entry

[martin rudalics]

> What system is that (w32?)?

In GNU Emacs 22.0.50.1 (i386-mingw-windows98.3000)
  of 2006-08-27 on MACHNO
X server distributor `Microsoft Corp.', version 4.90.3000
configured using `configure --with-gcc (3.4)'

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: dea
   locale-coding-system: cp1252
   default-enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
   shell-dirtrack-mode: t
   show-paren-mode: t
   encoded-kbd-mode: t
   tooltip-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   unify-8859-on-encoding-mode: t
   utf-translate-cjk-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

> What does `cvs status <file>' say?

File: array.el         	Status: Up-to-date

    Working revision:	1.29
    Repository revision:	1.29	/cvsroot/emacs/emacs/lisp/array.el,v
    Sticky Tag:		(none)
    Sticky Date:		(none)
    Sticky Options:	(none)

Re: vc-cvs-parse-entry

[Stefan Monnier]

>> What system is that (w32?)?
> In GNU Emacs 22.0.50.1 (i386-mingw-windows98.3000)

Then it's a known problem that's pretty hard to fix:

w32 handles DST by changing the definition of "time 0", so during the
DST-change things behave as if all the files's timestamps had been changed
by 1 hour.

So there really is a 1-hour difference between what the CVS/Entries file
says and what the file's time stamp says.  We could try to be more clever
with such situations, but since it's basically a case of braindamage in the
w32 "spec" I'd rather not bother.
IIRC "cvs update" will fix things for you.


        Stefan


PS: Note that last I heard `cvs status' has to send the file over to the
    server in order to figure out that it's up-to-date, because it uses the
    exact same algorithm as vc-cvs.el to decide whether a file was touched
    or not.  You can figure that out with "cvs -t status":

% cvs -t status INSTALL.CVS 
  -> main: Session ID is Mg0eb0F9FW12DMKr
  -> main loop with CVSROOT=/sources/emacs
  -> open_connection_to_server (cvs.sv.gnu.org:/sources/emacs)
 -> Starting server: ssh-cvs cvs.sv.gnu.org cvs server 
  -> rename(CVS/Entries.Backup,CVS/Entries)
  -> unlink_file(CVS/Entries.Log)
S -> serve_directory (.)
S -> dirswitch (., /sources/emacs/emacs)
S -> do_cvs_command (status)
S -> server_notify()
===================================================================
File: INSTALL.CVS       Status: Up-to-date

   Working revision:    1.7
   Repository revision: 1.7     /sources/emacs/emacs/INSTALL.CVS,v
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      (none)

S -> server_pathname_check (INSTALL.CVS)
S -> Reader_Lock(/sources/emacs/emacs)
S -> Simple_Lock_Cleanup()
  -> close_connection_to_server ()
S -> Lock_Cleanup()
S -> Simple_Lock_Cleanup()
S -> server_cleanup()
% touch INSTALL.CVS 
% cvs -t status INSTALL.CVS
  -> main: Session ID is 0Ihv3mbkCYccDMKr
  -> main loop with CVSROOT=/sources/emacs
  -> open_connection_to_server (cvs.sv.gnu.org:/sources/emacs)
 -> Starting server: ssh-cvs cvs.sv.gnu.org cvs server 
  -> Sending file `INSTALL.CVS' to server
S -> serve_directory (.)
  -> Register(INSTALL.CVS, 1.7, Tue Aug 29 21:05:25 2006, ,  )
  -> rename(CVS/Entries.Backup,CVS/Entries)
  -> unlink_file(CVS/Entries.Log)
===================================================================
File: INSTALL.CVS       Status: Up-to-date

   Working revision:    1.7
   Repository revision: 1.7     /sources/emacs/emacs/INSTALL.CVS,v
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      (none)

S -> server_pathname_check (INSTALL.CVS)
S -> Reader_Lock(/sources/emacs/emacs)
S -> RCS_cmp_file( /sources/emacs/emacs/INSTALL.CVS,v, 1.7, (null), , INSTALL.CVS )
S -> RCS_checkout (/sources/emacs/emacs/INSTALL.CVS,v, 1.7, , , (function))
S -> server_register(INSTALL.CVS, 1.7, Tue Aug 29 21:05:27 2006, , , , )
S -> Register(INSTALL.CVS, 1.7, Tue Aug 29 21:05:27 2006, ,  )
S -> Simple_Lock_Cleanup()
S -> rename(CVS/Entries.Backup,CVS/Entries)
S -> unlink_file(CVS/Entries.Log)
  -> close_connection_to_server ()
S -> dirswitch (., /sources/emacs/emacs)
S -> do_cvs_command (status)
S -> server_notify()
S -> Lock_Cleanup()
S -> Simple_Lock_Cleanup()
S -> server_cleanup()
% 

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 29 Aug 2006 17:06:50 -0400
> Cc: emacs-devel@gnu.org
> 
> >> What system is that (w32?)?
> > In GNU Emacs 22.0.50.1 (i386-mingw-windows98.3000)
> 
> Then it's a known problem that's pretty hard to fix:
> 
> w32 handles DST by changing the definition of "time 0"

This is not really accurate, at least on newer versions of Windows;
see below.

> so during the DST-change things behave as if all the files's
> timestamps had been changed by 1 hour.
> 
> So there really is a 1-hour difference between what the CVS/Entries file
> says and what the file's time stamp says.  We could try to be more clever
> with such situations, but since it's basically a case of braindamage in the
> w32 "spec" I'd rather not bother.

I'm not familiar with the details, nor with vc-cvs in general, so
what's below might be nonsense; sorry if so.

Is the problem you describe above in Emacs or in the CVS client that
Emacs runs?  If the latter, I agree that some (or maybe all) Windows
ports of the CVS client don't DTRT in this case.

If this is the problem, the only way to fix it is to complain to the
maintainer(s) of the CVS port and maybe provide a patch to them.

However, if the problem is in Emacs, we might be able to fix it, if
someone describes what Emacs functions are involved.  This is because
Windows does maintain the DST change rules, and its implementation of
localtime is supposed to use those rules, if they are known to the OS,
when it computes the tm_isdst member of struct tm.  So, at least in
principle, the required information is available, and can be used to
DTRT in this case.  For example, in the GnuWin32 port of Diffutils, I
see that the file times are reported with correct TZ offsets, like
this:

  --- make.h~0   2006-02-16 03:54:43.000000000 +0200
  +++ make.h     2006-08-18 21:12:32.859375000 +0300

As you see, Diff correctly determines that the old version was before
the DST change, while the new was after it.  AFAICS, Diff uses
strftime (which in turn uses localtime) to get at this information.

> IIRC "cvs update" will fix things for you.

Yes, but it's _awfully_ slow.  I needed to write a program to move the
files' last write timestamp by N hours, to avoid the resultant lossage
on Windows, whereby "cvs up" after a DST change can take _hours_
because each file is sent upstream due to the time mismatch.

Again, sorry if what I said doesn't make sense in the context of this
discussion.

Re: vc-cvs-parse-entry

[martin rudalics]

 >>So there really is a 1-hour difference between what the CVS/Entries file
 >>says and what the file's time stamp says.  We could try to be more clever
 >>with such situations, but since it's basically a case of braindamage in the
 >>w32 "spec" I'd rather not bother.
 >
 >
 > I'm not familiar with the details, nor with vc-cvs in general, so
 > what's below might be nonsense; sorry if so.
 >
 > Is the problem you describe above in Emacs or in the CVS client that
 > Emacs runs?  If the latter, I agree that some (or maybe all) Windows
 > ports of the CVS client don't DTRT in this case.

Eli, please look at what your system tells you about the following two
Emacs files: "src/acldef.h" and "src/alloca.c".  I get:

Entries:       /acldef.h/1.2/Mon Sep  1 15:45:52 2003//
Emacs:                       Mon Sep  1 17:45:52 2003
Windows:                     Mon Sep  1 17:45:52 2003

Entries:      /alloca.c/1.29/Fri Jan 30 17:10:02 2004//
Emacs:                       Fri Jan 30 17:10:02 2004
Windows:                     Fri Jan 30 18:10:02 2004

Where Entries is the entry from CVS/Entries, Emacs is a prettified
version obtained from `file-attributes', and Windows is what WindowsME
tells me.  I fail to understand why the values for `alloca.c' differ for
Windows and Emacs.  If these values are identic for you (XP I presume)
than the problem is with Emacs for Windows98/FAT32.

Re: vc-cvs-parse-entry

[Stefan Monnier]

>> >> What system is that (w32?)?
>> > In GNU Emacs 22.0.50.1 (i386-mingw-windows98.3000)
>> Then it's a known problem that's pretty hard to fix:
>> w32 handles DST by changing the definition of "time 0"
> This is not really accurate, at least on newer versions of Windows;
> see below.

It may have been fixed since w98, but I'm pretty sure that's how w98 did it.

>> IIRC "cvs update" will fix things for you.

> Yes, but it's _awfully_ slow.  I needed to write a program to move the
> files' last write timestamp by N hours, to avoid the resultant lossage
> on Windows, whereby "cvs up" after a DST change can take _hours_
> because each file is sent upstream due to the time mismatch.

The "time mismatch" you see is in CVS, not in Emacs.  vc-cvs.el suffers from
the same time mismatch because it uses the same algorithm (except it's
written in elisp instead of C).  I don't think vc-cvs.el should try to be
more clever than CVS itself.

Note that if more recent versions of w32 fixed this problem, it should be
fixed for both CVS and vc-cvs.el (and pcl-cvs ;-).

> Again, sorry if what I said doesn't make sense in the context of this
> discussion.

It makes sense to me.


        Stefan

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Cc: rudalics@gmx.at,  emacs-devel@gnu.org
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 30 Aug 2006 17:01:38 -0400
> 
> >> >> What system is that (w32?)?
> >> > In GNU Emacs 22.0.50.1 (i386-mingw-windows98.3000)
> >> Then it's a known problem that's pretty hard to fix:
> >> w32 handles DST by changing the definition of "time 0"
> > This is not really accurate, at least on newer versions of Windows;
> > see below.
> 
> It may have been fixed since w98, but I'm pretty sure that's how w98 did it.

The NTFS filesystem stores file times in UTC, but other file systems
don't, even when the OS is Windows XP.  See:

  http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/file_times.asp

> >> IIRC "cvs update" will fix things for you.
> 
> > Yes, but it's _awfully_ slow.  I needed to write a program to move the
> > files' last write timestamp by N hours, to avoid the resultant lossage
> > on Windows, whereby "cvs up" after a DST change can take _hours_
> > because each file is sent upstream due to the time mismatch.
> 
> The "time mismatch" you see is in CVS, not in Emacs.  vc-cvs.el suffers from
> the same time mismatch because it uses the same algorithm (except it's
> written in elisp instead of C).

Given that Emacs running on NTFS will (or at least could) see true UTC
file times, I don't see why it should see any mismatch whatsoever.
Perhaps someone who has a working vc-cvs setup on NTFS could see if
this problem still exists there.

> I don't think vc-cvs.el should try to be more clever than CVS
> itself.

I don't see why we should replicate problems with ported CVS clients.

> Note that if more recent versions of w32 fixed this problem, it should be
> fixed for both CVS and vc-cvs.el (and pcl-cvs ;-).

Only if the CVS client uses the right code to get the file times; see
the above URL for the gory details.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Wed, 30 Aug 2006 19:51:09 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> 
> Eli, please look at what your system tells you about the following two
> Emacs files: "src/acldef.h" and "src/alloca.c".  I get:
> 
> Entries:       /acldef.h/1.2/Mon Sep  1 15:45:52 2003//
> Emacs:                       Mon Sep  1 17:45:52 2003
> Windows:                     Mon Sep  1 17:45:52 2003
> 
> Entries:      /alloca.c/1.29/Fri Jan 30 17:10:02 2004//
> Emacs:                       Fri Jan 30 17:10:02 2004
> Windows:                     Fri Jan 30 18:10:02 2004
> 
> Where Entries is the entry from CVS/Entries, Emacs is a prettified
> version obtained from `file-attributes', and Windows is what WindowsME
> tells me.  I fail to understand why the values for `alloca.c' differ for
> Windows and Emacs.

Probably some mismatch in adjusting for the daylight saving offset;
the URL I mentioned in my other message suggests some hints.

> If these values are identic for you (XP I presume) than the problem
> is with Emacs for Windows98/FAT32.

See the URL I mentioned in my other message: yes, there's a difference
between FAT and NTFS volumes in this respect.

However, for a valid and meaningful comparison, the test cases you've
chosen are not good enough.  First, the actual time stamps of files in
the local sandbox depend on the exact day one checks out the CVS tree
and/or resyncs with it.  Obviously, no two people do that on the same
day; for example, my CVS tree was checked out in June 2005, so
acldef.h bears a time stamp of June 18, 2005, which is very different
from yours.  It's impossible to compare times like this.

I don't have any suggestions for how to make a meaningful comparison,
though.  There are too many system calls involved, and too many
unknowns.

Re: vc-cvs-parse-entry

[martin rudalics]

 >>Note that if more recent versions of w32 fixed this problem, it should be
 >>fixed for both CVS and vc-cvs.el (and pcl-cvs ;-).
 >
 >
 > Only if the CVS client uses the right code to get the file times; see
 > the above URL for the gory details.

FWIW, CVS did "use the right code to get the file times".  Also files
reported by vc-cvs-parse-entry as "modified" are _not_ reported as
"modified" by a recently installed wincvs.  In fact. I've not been able
to reproduce the modification times reported by (X)Emacs with any other
application on my system.

 > However, for a valid and meaningful comparison, the test cases you've
 > chosen are not good enough.  First, the actual time stamps of files in
 > the local sandbox depend on the exact day one checks out the CVS tree
 > and/or resyncs with it.  Obviously, no two people do that on the same
 > day; for example, my CVS tree was checked out in June 2005, so
 > acldef.h bears a time stamp of June 18, 2005, which is very different
 > from yours.  It's impossible to compare times like this.

I see: These two files are among the few that had not been changed ever
since I checked out my CVS tree.  Hence there was a minimal chance it
would be the same for you.

 > I don't have any suggestions for how to make a meaningful comparison,
 > though.  There are too many system calls involved, and too many
 > unknowns.

Provided DST is on in your current time zone: Do you have a file on your
system with a modification date while DST was off?  Does your OS give
the same modification time as Emacs?  It doesn't for me.  Reading the
URL you suggested I think it's due to the following:

FindFirstFile retrieves the local time from the FAT file system and
converts it to UTC by using the current settings for the time zone and
daylight saving time. Therefore, if it is daylight saving time,
FindFirstFile takes daylight saving time into account, even if the file
time you are converting is in standard time.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Sat, 02 Sep 2006 15:45:42 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> 
>  >>Note that if more recent versions of w32 fixed this problem, it should be
>  >>fixed for both CVS and vc-cvs.el (and pcl-cvs ;-).
>  >
>  >
>  > Only if the CVS client uses the right code to get the file times; see
>  > the above URL for the gory details.
> 
> FWIW, CVS did "use the right code to get the file times".

By ``CVS'', do you mean ``the recently installed wincvs'' you
mentioned?  I use some old port of CVS, and it does have problems when
DST changes, even though I have an XP machine and an NTFS filesystem.
So I think the CVS port does have a say here, not only the filesystem.

> In fact. I've not been able to reproduce the modification times
> reported by (X)Emacs with any other application on my system.

Please show an example of these modification times, and how did you
compare that with whatever other applications on your system report
file times.

> Provided DST is on in your current time zone: Do you have a file on your
> system with a modification date while DST was off?

Yes, of course.  How about src/unexelf.c, for example?  It has the
modification time of March 18, 2006, 5:32PM on my system.

> Does your OS give the same modification time as Emacs?

The problem is with the ``OS'' part--how did you know what ``the OS''
thinks about that file's times?  What utilities/system calls and/or
Emacs functions you used to find that out?  I don't think we can have
any progress until we have answers for these questions, and understand
well what code is involved in reporting time stamps.  One problem is
that, for commands like DIR, we don't have any way of knowing what
they do, since their source code is not available.

What I did was to reset the time zone to GMT and disable the automatic
DST corrections, then look at what DIR (from cmd.exe) reports.  I then
set the time zone back to the normal setting, and looked at what DIR
displayed now.  My conclusion was that DIR lies about file times: it
reports them as if DST were in effect, even if the file was modified
when DST was off.  By contrast, the GnuWin32 port of ls.exe reported
the file times correctly, both for files modified under DST and files
modified outside DST.

We could make a similar check with Emacs primitives instead of ls.exe
and DIR, if needed.

But the trick with setting TZ to GMT probably won't work on FAT
volumes, since the file times are recorded as local time there.

Btw, I find the Date/Time decoder utility very useful when testing
these issues, see:

   http://www.digital-detective.co.uk/freetools/decode.asp

> It doesn't for me.  Reading the URL you suggested I think it's due
> to the following:
> 
> FindFirstFile retrieves the local time from the FAT file system and
> converts it to UTC by using the current settings for the time zone and
> daylight saving time. Therefore, if it is daylight saving time,
> FindFirstFile takes daylight saving time into account, even if the file
> time you are converting is in standard time.

It's possible.  You might be able test this hypothesis by unchecking
the "Automatically adjust clock for daylight saving changes" checkbox
in the "Time Zone" tab of the "Date and Time Properties" dialog that
pops up when you double-click the current time string displayed at the
right edge of your system tray.  (You will need to reboot the machine
after unchecking that option, so that Windows doesn't reuse the cached
values of DST offset, when it reports UTC file times.)

Re: vc-cvs-parse-entry

[martin rudalics]

 >>FWIW, CVS did "use the right code to get the file times".
 >
 >
 > By ``CVS'', do you mean ``the recently installed wincvs'' you
 > mentioned?

No. I mean Concurrent Versions System (CVS) 1.11.19 (client).

 > I use some old port of CVS, and it does have problems when
 > DST changes, even though I have an XP machine and an NTFS filesystem.
 > So I think the CVS port does have a say here, not only the filesystem.

Maybe it's related to what I read here

http://www.codeproject.com/datetime/dstbugs.asp

and there

http://www.devguy.com/fp/cfgmgmt/cvs/

 >>Provided DST is on in your current time zone: Do you have a file on your
 >>system with a modification date while DST was off?
 >
 >
 > Yes, of course.  How about src/unexelf.c, for example?  It has the
 > modification time of March 18, 2006, 5:32PM on my system.

It's the problem we had before, just in reverse.  I did a CVS update on
April 11th, 2006.

 >>Does your OS give the same modification time as Emacs?
 >
 >
 > The problem is with the ``OS'' part--how did you know what ``the OS''
 > thinks about that file's times?  What utilities/system calls and/or
 > Emacs functions you used to find that out?  I don't think we can have
 > any progress until we have answers for these questions, and understand
 > well what code is involved in reporting time stamps.  One problem is
 > that, for commands like DIR, we don't have any way of knowing what
 > they do, since their source code is not available.
 >
 > What I did was to reset the time zone to GMT and disable the automatic
 > DST corrections, then look at what DIR (from cmd.exe) reports.  I then
 > set the time zone back to the normal setting, and looked at what DIR
 > displayed now.

Did you change your local time in the BIOS?  Did you change the time
zone settings in

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation\

Or did you simply check the box about automatic correction from the
system tray?  That just dis-/enables setting local time after the first
start of Windows when DST has changed by consulting

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Time Zones\

My settings for this are

C4 FF FF FF 00 00 00 00
C4 FF FF FF 00 00 0A 00  <-- 0A means "october"
00 00 05 00 03 00 00 00  <-- 05 means last sunday, 03 at three o'clock
00 00 00 00 00 00 03 00  <-- 03 means "march"
00 00 05 00 02 00 00 00  <-- 05 means last sunday, 02 at two o'clock
00 00 00 00

Did you change that?

I have checked the box on the present partition and disabled it on all
other partitions.  Otherwise I'd change my local time whenever I switch
to another partition after DST changed.  (Un-)checking the box does not
have any impact on how file modification times are reported.

 > My conclusion was that DIR lies about file times: it
 > reports them as if DST were in effect, even if the file was modified
 > when DST was off.

Because you neither changed your local time nor the respective registry
settings, I presume.  (I think it's better to leave them alone, anyway.)

 > By contrast, the GnuWin32 port of ls.exe reported
 > the file times correctly, both for files modified under DST and files
 > modified outside DST.

Because ls refers to the UTC value stored on disk.  I don't have that.

My ls.exe (version 3.16 of GNU fileutils) doesn't report full time for
files modified before March, 9th, 2006 - for whatever reason.  However,
the times reported by ls for files last modified around March 15, 2006
match those reported by DIR and other applications.  They are _not_
identic to those reported by Emacs' file-attributes.

 > We could make a similar check with Emacs primitives instead of ls.exe
 > and DIR, if needed.
 >
 > But the trick with setting TZ to GMT probably won't work on FAT
 > volumes, since the file times are recorded as local time there.

But do ls.exe and DIR give the same information for src/unexelf.c on
your system?  And what would interest me most: Could you try to debug
`vc-cvs-parse-entry' while opening src/unexelf.c and look whether it
classifies the file as modified?

 > Btw, I find the Date/Time decoder utility very useful when testing
 > these issues, see:
 >
 >    http://www.digital-detective.co.uk/freetools/decode.asp

What values do you paste there?  Where do you get them from?

 > It's possible.  You might be able test this hypothesis by unchecking
 > the "Automatically adjust clock for daylight saving changes" checkbox
 > in the "Time Zone" tab of the "Date and Time Properties" dialog that
 > pops up when you double-click the current time string displayed at the
 > right edge of your system tray.  (You will need to reboot the machine
 > after unchecking that option, so that Windows doesn't reuse the cached
 > values of DST offset, when it reports UTC file times.)

This is not sufficient as I tried to explain above.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Sun, 03 Sep 2006 12:40:11 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
>  > I use some old port of CVS, and it does have problems when
>  > DST changes, even though I have an XP machine and an NTFS filesystem.
>  > So I think the CVS port does have a say here, not only the filesystem.
> 
> Maybe it's related to what I read here
> 
> http://www.codeproject.com/datetime/dstbugs.asp

This one partly describes what the MSDN article whose URL I posted
describes, and partly is simply wrong.  There _is_ a way of correctly
accounting for DST on NTFS, and the MSDN article says how to do that.
Presumably, library functions on Windows do that already, although I
didn't have time to check.

> and there
> 
> http://www.devguy.com/fp/cfgmgmt/cvs/

This seems to say that WinCVS figured out how to code around the
problem for FAT, but not for NTFS.

>  > What I did was to reset the time zone to GMT and disable the automatic
>  > DST corrections, then look at what DIR (from cmd.exe) reports.  I then
>  > set the time zone back to the normal setting, and looked at what DIR
>  > displayed now.
> 
> Did you change your local time in the BIOS?  Did you change the time
> zone settings in
> 
> HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation\
> 
> Or did you simply check the box about automatic correction from the
> system tray?

The latter.

> That just dis-/enables setting local time after the first
> start of Windows when DST has changed by consulting
> 
> HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Time Zones\

Note that I changed to GMT+0 _and_ disabled automatic DST correction.
Thus, no additional registry changes are necessary, at least not on
XP.

> My settings for this are
> 
> C4 FF FF FF 00 00 00 00
> C4 FF FF FF 00 00 0A 00  <-- 0A means "october"
> 00 00 05 00 03 00 00 00  <-- 05 means last sunday, 03 at three o'clock
> 00 00 00 00 00 00 03 00  <-- 03 means "march"
> 00 00 05 00 02 00 00 00  <-- 05 means last sunday, 02 at two o'clock
> 00 00 00 00
> 
> Did you change that?

GMT+0/no DST means the DST rules have no effect.

> I have checked the box on the present partition and disabled it on all
> other partitions.  Otherwise I'd change my local time whenever I switch
> to another partition after DST changed.  (Un-)checking the box does not
> have any impact on how file modification times are reported.

Because you are on a FAT volume.

>  > My conclusion was that DIR lies about file times: it
>  > reports them as if DST were in effect, even if the file was modified
>  > when DST was off.
> 
> Because you neither changed your local time nor the respective registry
> settings, I presume.

Local time changes automatically on XP when you change the time zone
(my PC synchronizes with a time server via NTP, if that matters).

> My ls.exe (version 3.16 of GNU fileutils) doesn't report full time for
> files modified before March, 9th, 2006 - for whatever reason.

That's the standard ls ``6 months is way in the past'' policy.  It
only displays full time for files modified ``recently''.

> However,
> the times reported by ls for files last modified around March 15, 2006
> match those reported by DIR and other applications.  They are _not_
> identic to those reported by Emacs' file-attributes.

I think file-attributes returns UTC times, since it calls `stat'.

> But do ls.exe and DIR give the same information for src/unexelf.c on
> your system?

I thought I explained that DIR reports it with a 1-hour error, due to
incorrect accounting for DST.

> And what would interest me most: Could you try to debug
> `vc-cvs-parse-entry' while opening src/unexelf.c and look whether it
> classifies the file as modified?

I will try when I have time, but I doubt it will be marked modified.

>  >    http://www.digital-detective.co.uk/freetools/decode.asp
> 
> What values do you paste there?  Where do you get them from?

It accepts values from `stat' or similar API functions.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Mon, 04 Sep 2006 00:00:04 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > My ls.exe (version 3.16 of GNU fileutils) doesn't report full time for
> > files modified before March, 9th, 2006 - for whatever reason.
> 
> That's the standard ls ``6 months is way in the past'' policy.  It
> only displays full time for files modified ``recently''.

I forgot to mention that you can use the --full-time switch to `ls' to
force it display the full time for all files.

Re: vc-cvs-parse-entry

[martin rudalics]

 > I forgot to mention that you can use the --full-time switch to `ls' to
 > force it display the full time for all files.

Sorry, I forgot that --full-time needs -l.  ls.exe (and thus dired with
`ls-lisp-use-insert-directory-program' non-nil) gets me:

-rw-rw-rw-   1 user     group        1188 Mon Sep 01 17:45:52 2003 acldef.h
-rw-rw-rw-   1 user     group       15285 Fri Jan 30 18:10:02 2004 alloca.c
                                                      ^^
DIR gets me:

ACLDEF   H           1.188  01.09.03  17:45 acldef.h
ALLOCA   C          15.285  30.01.04  18:10 alloca.c
                                       ^^

Emacs gets me (sorry, it's too tedious to turn off the locales):

(format-time-string
  "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/acldef.h"))) ->
"Mo Sep 01 17:45:52 2003 (MEZ) - Mitteleurop. Sommerzeit"

... "Sommerzeit" == DST

(format-time-string
  "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/acldef.h")) t) ->
"Mo Sep 01 15:45:52 2003 GMT"

(format-time-string
  "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/alloca.c"))) ->
"Fr Jän 30 17:10:02 2004 (MEZ) Mitteleuropäische Zeit"
            ^^
... no DST here

(format-time-string
  "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/alloca.c")) t) ->
"Fr Jän 30 16:10:02 2004 GMT"

BTW, stat (GNU coreutils) 5.3.0 gives the same results as Emacs, hence
the results delivered by stat and ls (GNU fileutils) 3.16 differ on my
system.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

> Date: Mon, 04 Sep 2006 11:17:18 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
> Sorry, I forgot that --full-time needs -l.  ls.exe (and thus dired with
> `ls-lisp-use-insert-directory-program' non-nil) gets me:
> 
> -rw-rw-rw-   1 user     group        1188 Mon Sep 01 17:45:52 2003 acldef.h
> -rw-rw-rw-   1 user     group       15285 Fri Jan 30 18:10:02 2004 alloca.c
>                                                       ^^
> DIR gets me:
> 
> ACLDEF   H           1.188  01.09.03  17:45 acldef.h
> ALLOCA   C          15.285  30.01.04  18:10 alloca.c
>                                        ^^
> 
> Emacs gets me (sorry, it's too tedious to turn off the locales):
> 
> (format-time-string
>   "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/acldef.h"))) ->
> "Mo Sep 01 17:45:52 2003 (MEZ) - Mitteleurop. Sommerzeit"
> 
> ... "Sommerzeit" == DST
> 
> (format-time-string
>   "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/acldef.h")) t) ->
> "Mo Sep 01 15:45:52 2003 GMT"
> 
> (format-time-string
>   "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/alloca.c"))) ->
> "Fr Jän 30 17:10:02 2004 (MEZ) Mitteleuropäische Zeit"
>             ^^
> ... no DST here
> 
> (format-time-string
>   "%a %b %d %H:%M:%S %Y %Z" (nth 5 (file-attributes "~/src/alloca.c")) t) ->
> "Fr Jän 30 16:10:02 2004 GMT"

Please tell which of these results are correct.  I don't know what are
the details of the DST rules in your locale.

> BTW, stat (GNU coreutils) 5.3.0 gives the same results as Emacs, hence
> the results delivered by stat and ls (GNU fileutils) 3.16 differ on my
> system.

The GnuWin32 ports use a different implementation of stat nowadays,
perhaps that's the cause for the different results.

Re: vc-cvs-parse-entry

[martin rudalics]

 > Please tell which of these results are correct.  I don't know what are
 > the details of the DST rules in your locale.

I can't.  I don't know how to get the original times from the server.


There is, however, one thing I know: ediff uses

(ediff-format-date (decode-time file-modtime))

where file-modtime is taken from (nth 5 (file-attributes filename)) to
insert modification times in listings.

Now I happen to keep around backups of ediff output.  In particular I
have a file called "cus-edit.patch".  Its modification time according to
Emacs is 2006-02-26 17:29:48.  (According to DIR its 26.02.06 18:29.)
The file contains the following ediff output:

*** cus-edit.el	Wed Feb  1 10:17:44 2006
--- cus-edit.el	Sun Feb 26 18:11:40 2006

With the modification time reported by Emacs I must have

- saved `cus-edit.el' at 18:11:40,

- run ediff, and

- saved the output of ediff to "cus-edit.patch" at 17:29:48.

That's paradoxal.  (I verified that with a couple of other files, of
course.)


There's a second clue.  I use (add-hook 'before-save-hook 'time-stamp).
Now I have a backup of a file containing the line

;; Time-stamp: "2006-02-03 18:32:33 martin"

the modification time of that file as reported by Emacs reads as
2006-02-03 17:32:32.


Hence I strongly conjecture that when DST is on, `file-attributes'
returns the wrong modification time for files saved when DST was off for
Windows98/FAT32.


 >>BTW, stat (GNU coreutils) 5.3.0 gives the same results as Emacs, hence
 >>the results delivered by stat and ls (GNU fileutils) 3.16 differ on my
 >>system.
 >
 >
 > The GnuWin32 ports use a different implementation of stat nowadays,
 > perhaps that's the cause for the different results.

ls (GNU coreutils) 5.3.0 does the same as stat (GNU coreutils) 5.3.0.
I conjecture that both report wrong times for Windows98/FAT32.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Tue, 05 Sep 2006 11:10:20 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
>  > Please tell which of these results are correct.  I don't know what are
>  > the details of the DST rules in your locale.
> 
> I can't.  I don't know how to get the original times from the server.

The server times have nothing to do with this: the files on your
machine get timestamped with your machine's time when they are updated
by "cvs up".

One simple way of figuring out whether Emacs lies about time-related
values is to reset your machine's date to when DST is not in effect,
create a file (noting what was the machine's wallclock time), then
revert the date to its correct value, and perform the experiments with
that file.  (It is best to reboot the machine each time you modify the
date.)  Can you try that?

> Hence I strongly conjecture that when DST is on, `file-attributes'
> returns the wrong modification time for files saved when DST was off for
> Windows98/FAT32.

But you didn't compare the result of file-attributes, you compared the
result of running file-attributes' value through format-time-string,
which converts the value to local time.  We need to separate these two
unknowns, I think.

>  >>BTW, stat (GNU coreutils) 5.3.0 gives the same results as Emacs, hence
>  >>the results delivered by stat and ls (GNU fileutils) 3.16 differ on my
>  >>system.
>  >
>  >
>  > The GnuWin32 ports use a different implementation of stat nowadays,
>  > perhaps that's the cause for the different results.
> 
> ls (GNU coreutils) 5.3.0 does the same as stat (GNU coreutils) 5.3.0.
> I conjecture that both report wrong times for Windows98/FAT32.

This comes as no surprise, since no doubt _all_ programs in the ported
Coreutils, including stat, were linked against the same implementation
of the library function `stat'.

Re: vc-cvs-parse-entry

[martin rudalics]

 > The server times have nothing to do with this: the files on your
 > machine get timestamped with your machine's time when they are updated
 > by "cvs up".

The two files in question were never affected by a cvs update after I
checked that out in early 2005.  Earlier you wrote:

 > How about src/unexelf.c, for example?  It has the
 > modification time of March 18, 2006, 5:32PM on my system.

According to your reasoning above wouldn't this imply you never did a
cvs update after March 2006?

 > One simple way of figuring out whether Emacs lies about time-related
 > values is to reset your machine's date to when DST is not in effect,
 > create a file (noting what was the machine's wallclock time), then
 > revert the date to its correct value, and perform the experiments with
 > that file.  (It is best to reboot the machine each time you modify the
 > date.)  Can you try that?

I wouldn't know what to reset.  I'm reluctant to change DST settings in
the registry.  (Un-)checking that box from the system tray affects the
behavior of my machine only when DST actually changes.  Changing the
time-zone settings of my machine won't affect filetimes and how they are
displayed (it might affect cvs update, though).

Note: For a computer that never moves from one time-zone to another it's
insignificant whether modification times are stored as UTC or local
time.  It's sufficient to maintain a set of rules that determine whether
a modification time has to be evaluated with DST on or off.  In my
present time-zone the respective dates are the last sundays in March and
October of a year.  Getting the correct modification time of a file must
be done by adding one hour to the modification time stored on disk
provided that time falls somewhere between these dates for the year in
question.  Obviously, for one hour each year, a file B I modified after
a file A will get an earlier modification time than A.  Since I don't
work at that time I never noticed that problem.

Clearly, the "storing local times only" philospohy must fail when I move
my machine from one time-zone to another: I cannot distinguish files
created in different time-zones since I have no notion of UTC.  Every
interpretation of a filetime occurs as if the file had been modified in
my present (virtual) time-zone.

 > But you didn't compare the result of file-attributes, you compared the
 > result of running file-attributes' value through format-time-string,
 > which converts the value to local time.  We need to separate these two
 > unknowns, I think.

On my system `file-attributes' represent local time.

 >>ls (GNU coreutils) 5.3.0 does the same as stat (GNU coreutils) 5.3.0.
 >>I conjecture that both report wrong times for Windows98/FAT32.
 >
 > This comes as no surprise, since no doubt _all_ programs in the ported
 > Coreutils, including stat, were linked against the same implementation
 > of the library function `stat'.

I suppose they changed that for XP and simply forgot about FAT Windows.
In fact, installing the latest GnuWin32 also broke something in building
Emacs on my machine (probably due to a change in cp.exe).  It took me
some time to restore my fragile balance of mingw, cygwin, unxutils and
gnuwin32 tools after that.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Sun, 10 Sep 2006 11:55:33 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
>  > The server times have nothing to do with this: the files on your
>  > machine get timestamped with your machine's time when they are updated
>  > by "cvs up".
> 
> The two files in question were never affected by a cvs update after I
> checked that out in early 2005.

Then read "cvs co" instead of "cvs up".  The point here is that the
files get their file stamps when you check them out or update them;
the timestamp has nothing to do with when the file was last checked
into the repository.  And the timestamps in CVS/Entries are just those
checkout/update times in UTC (or that's what the CVS client expects to
find there).

> Earlier you wrote:
> 
>  > How about src/unexelf.c, for example?  It has the
>  > modification time of March 18, 2006, 5:32PM on my system.
> 
> According to your reasoning above wouldn't this imply you never did a
> cvs update after March 2006?

Yes, see above.

>  > One simple way of figuring out whether Emacs lies about time-related
>  > values is to reset your machine's date to when DST is not in effect,
>  > create a file (noting what was the machine's wallclock time), then
>  > revert the date to its correct value, and perform the experiments with
>  > that file.  (It is best to reboot the machine each time you modify the
>  > date.)  Can you try that?
> 
> I wouldn't know what to reset.

The date in your machine's BIOS.

> I'm reluctant to change DST settings in the registry.

I cannot imagine why (since you'd be restoring it to its original
value right after that), but it's your machine.

Unfortunately, if I don't have someone with access to a Windows 9x
system who can work with me on this, the problem, if there is one,
will not be solved any time soon.

> Changing the time-zone settings of my machine won't affect filetimes
> and how they are displayed

According to the Microsoft docs, it will affect what system calls like
FindFirstFile return.  And since `stat' calls FindFirstFile, that
could very well explain what happens with CVS.  See below.

> Note: For a computer that never moves from one time-zone to another it's
> insignificant whether modification times are stored as UTC or local
> time.

I think it _is_ significant, since various system calls need to return
UTC times.  On NTFS, they simply return what they find on disk, while
on FAT, they need to decide how many hours to add/subtract to get UTC
from the stored local times.  And that's where the problem happens:
the offset used is the _current_ one, not the one that was in effect
when the file was actually last written to.

Sorry for repeating this, I thought we already understood the reasons
behind this problem.  But your assertion above tells me that perhaps
we still aren't clear about what happens.

> It's sufficient to maintain a set of rules that determine whether
> a modification time has to be evaluated with DST on or off.

It is only sufficient if those rules are actually _applied_ when
FindFirstFile and friends convert file times recorded by FAT to UTC.
According to what I read in the MS docs, they don't use those rules;
instead, they use the current offset, which could be different from
the correct one if the DST settings then and now are different.

> Clearly, the "storing local times only" philospohy must fail when I move
> my machine from one time-zone to another

Sending files to the CVS server _is_ moving them to a different time
zone.

> I cannot distinguish files
> created in different time-zones since I have no notion of UTC.  Every
> interpretation of a filetime occurs as if the file had been modified in
> my present (virtual) time-zone.

Sorry, I cannot figure out what you are saying here.

> On my system `file-attributes' represent local time.

Really?  Can you show me why you think so?

AFAICS, file-attributes returns the times it gets from `stat', and
`stat' (we use a replacement in w32.c) simply calls FindFirstFile.
Now, this URL in the Microsoft docs:

  http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/file_times.asp

says that FindFirstFile (and other similar system calls) return file
times in UTC, even on FAT filesystems.  Which to me makes perfect
sense, since `stat' on Posix systems returns UTC time stamps.

If you see something else, please tell me where in the Emacs code that
is involved in file-attributes you see the file times returned in
local time or converted to local time.

TIA

Re: vc-cvs-parse-entry

[martin rudalics]

 > Then read "cvs co" instead of "cvs up".  The point here is that the
 > files get their file stamps when you check them out or update them;
 > the timestamp has nothing to do with when the file was last checked
 > into the repository.  And the timestamps in CVS/Entries are just those
 > checkout/update times in UTC (or that's what the CVS client expects to
 > find there).

My ~/src/CVS/Entries contains the following lines:

/acldef.h/1.2/Mon Sep  1 15:45:52 2003//
/alloca.c/1.29/Fri Jan 30 17:10:02 2004//
/chpdef.h/1.2/Mon Sep  1 15:45:52 2003//
/hftctl.c/1.13/Mon Sep  1 15:45:52 2003//
/ioctl.h/1.2/Mon Sep  1 15:45:54 2003//
/ndir.h/1.4/Mon Sep  1 15:45:56 2003//
/param.h/1.2/Mon Sep  1 15:45:56 2003//
/point.h/1.2/Mon Sep  1 15:45:56 2003//
/pre-crt0.c/1.2/Mon Sep  1 15:45:56 2003//
/stamp-h.in/1.1/Mon Apr 15 13:14:46 2002//
/temacs.opt/1.1/Fri Oct  1 11:44:30 1999//
/unexhp9k800.c/1.10/Mon Sep  1 15:45:58 2003//
/vlimit.h/1.2/Mon Sep  1 15:45:58 2003//

I did "cvs co" on March 24, 2005.  I did not count the "cvs up" I did
since then, the last one was a week ago.  Can you explain how dates in
the range from 1999 till 2004 made it to my Entries?  IMHO, these dates
must reflect the corresponding dates in the repository.

 >>Changing the time-zone settings of my machine won't affect filetimes
 >>and how they are displayed
 >
 >
 > According to the Microsoft docs, it will affect what system calls like
 > FindFirstFile return.  And since `stat' calls FindFirstFile, that
 > could very well explain what happens with CVS.  See below.

But it does not.  Changing time-zone settings doesn't change anything
for filetimes reported by the programs I use.  The URL

http://search.cpan.org/~shay/Win32-UTCFileTime-1.45/lib/Win32/UTCFileTime.pm

tells me

"... that the problem with FindFirstFile() on a FAT drive actually seems
to be this: the file time retrieved is wrong by one hour if it was set
in the opposite DST season to the current system time.

This, of course, includes the case where (as in <"An example of the
problem">) a file is created in one DST season and is then read from the
opposite DST season (for example, create a file in summer (which
effectively sets a summer file time from summer) and then read it from
winter), but it also includes hitherto unseen cases in which
FindFirstFile() will even return the wrong time for a file time in the
same DST season as the current system time if the file time was set from
the opposite DST season to the current system time (for example, set a
summer file time from winter and then read it from summer). Also note
that, in other thus-far unseen cases, FindFirstFile() can return the
correct time for a file time in the opposite DST season to the current
system time if the file time was set from the same DST season as the
current system time (for example, set a winter file time from summer and
then read it from summer).

Thus, we appear to be concerned with when the file time was set relative
to the current system time, rather than what the file time is relative
to the current system time."

 >>Note: For a computer that never moves from one time-zone to another it's
 >>insignificant whether modification times are stored as UTC or local
 >>time.
 >
 >
 > I think it _is_ significant, since various system calls need to return
 > UTC times.  On NTFS, they simply return what they find on disk, while
 > on FAT, they need to decide how many hours to add/subtract to get UTC
 > from the stored local times.  And that's where the problem happens:
 > the offset used is the _current_ one, not the one that was in effect
 > when the file was actually last written to.

Indeed - that's why I said it's insignificant how values are _stored_.
I fully agree with the remainder of what you say here.

 > Sorry for repeating this, I thought we already understood the reasons
 > behind this problem.  But your assertion above tells me that perhaps
 > we still aren't clear about what happens.

I think we are.  You probably misunderstood my text.

 >>It's sufficient to maintain a set of rules that determine whether
 >>a modification time has to be evaluated with DST on or off.
 >
 >
 > It is only sufficient if those rules are actually _applied_ when
 > FindFirstFile and friends convert file times recorded by FAT to UTC.
 > According to what I read in the MS docs, they don't use those rules;
 > instead, they use the current offset, which could be different from
 > the correct one if the DST settings then and now are different.

Whoever "they" are.  FindFirstFile does not apply these rules.

 >>Clearly, the "storing local times only" philospohy must fail when I move
 >>my machine from one time-zone to another
 >
 >
 > Sending files to the CVS server _is_ moving them to a different time
 > zone.

I talked about moving my _machine_.

Moving a file is conceptually simple: Moving a file to my machine means
take its time measured in UTC, convert it to the value appropriate for
my local time-zone, and save it with that value.  Moving a file from my
machine to another one means calculate the UTC value from the filetime
and the time-zone of my machine.

 >>I cannot distinguish files
 >>created in different time-zones since I have no notion of UTC.  Every
 >>interpretation of a filetime occurs as if the file had been modified in
 >>my present (virtual) time-zone.
 >
 >
 > Sorry, I cannot figure out what you are saying here.

Time-zones are a "virtual" concept.  I can tell my computer to live in
an arbitrary time-zone, disregarding its physical location in the world.
This won't change the stored filetime of any file on my disk.  It will,
however, affect the value calculated by CVS and subsequently stored "as
filetime" on disk when I move a file from somewhere else to my machine.

 >>On my system `file-attributes' represent local time.
 >
 >
 > Really?  Can you show me why you think so?
 >
 > AFAICS, file-attributes returns the times it gets from `stat', and
 > `stat' (we use a replacement in w32.c) simply calls FindFirstFile.
 > Now, this URL in the Microsoft docs:
 >
 >   http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/file_times.asp
 >
 > says that FindFirstFile (and other similar system calls) return file
 > times in UTC, even on FAT filesystems.

It says:

"The FAT file system stores time values based on the local time of the
computer. For example, a file that is saved at 3:00pm PST in Washington
is seen as 6:00pm EST in New York on an NTFS volume, but it is seen as
3:00pm EST in New York on a FAT volume."

Please observe the use of the terms "stores" and "seen".  It's what I
tried to explain above: I set my computer's time-zone to PST, save a
file at 3:00 PST according to my computer's clock, set my computer's
time-zone to EST, the filetime shows 3:00 EST.  That's incorrect, and
the cause is that my filetimes are local time.  Your system should get
you 6:00 EST instead because your filetimes are UTC.

It further says:

"FindFirstFile retrieves the local time from the FAT file system and
converts it to UTC by using the current settings for the time zone and
daylight saving time. Therefore, if it is daylight saving time,
FindFirstFile takes daylight saving time into account, even if the file
time you are converting is in standard time."

Please observe the use of the term "current settings" here.  That's
precisely the problem I experience now: Today, "it is daylight saving
time" and the problematic files are "in standard time".

It finally says:

"GetFileTime retrieves cached UTC times from the FAT file system."

AFAICT, this means that FAT based Windows' cache the results of the
calculations I described above.  Hence, using GetFileTime might solve
the problem for FAT provided I invalidate the cache after each DST
change.

The URL I mentioned above claims that GetFileInformationByHandle
delivers correct filetimes even when DST changes in between.

 > Which to me makes perfect
 > sense, since `stat' on Posix systems returns UTC time stamps.

Apparently, it doesn't make sense for FAT.  I still wonder what the 3.16
Windows port of ls used instead, I never had the sources for that.

Re: vc-cvs-parse-entry

[Stefan Monnier]

> I did "cvs co" on March 24, 2005.  I did not count the "cvs up" I did
> since then, the last one was a week ago.  Can you explain how dates in
> the range from 1999 till 2004 made it to my Entries?  IMHO, these dates
> must reflect the corresponding dates in the repository.

Indeed, when you do `cvs co', cvs manually sets the mtime of the files it
creates to match the time when they were committed.  After that, cvs doesn't
fiddle with the mtime any more (i.e. the mtime does reflect the time at
which the file was later updated).

> This, of course, includes the case where (as in <"An example of the
> problem">) a file is created in one DST season and is then read from the
> opposite DST season (for example, create a file in summer (which
> effectively sets a summer file time from summer) and then read it from
> winter), but it also includes hitherto unseen cases in which
> FindFirstFile() will even return the wrong time for a file time in the
> same DST season as the current system time if the file time was set from
> the opposite DST season to the current system time (for example, set a
> summer file time from winter and then read it from summer). Also note
> that, in other thus-far unseen cases, FindFirstFile() can return the
> correct time for a file time in the opposite DST season to the current
> system time if the file time was set from the same DST season as the
> current system time (for example, set a winter file time from summer and
> then read it from summer).

Indeed, that's why it's hopeless: just do `cvs update' and get on with
your life.


        Stefan

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Mon, 11 Sep 2006 11:41:21 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
> My ~/src/CVS/Entries contains the following lines:
> 
> /acldef.h/1.2/Mon Sep  1 15:45:52 2003//
> /alloca.c/1.29/Fri Jan 30 17:10:02 2004//
> /chpdef.h/1.2/Mon Sep  1 15:45:52 2003//
> /hftctl.c/1.13/Mon Sep  1 15:45:52 2003//
> /ioctl.h/1.2/Mon Sep  1 15:45:54 2003//
> /ndir.h/1.4/Mon Sep  1 15:45:56 2003//
> /param.h/1.2/Mon Sep  1 15:45:56 2003//
> /point.h/1.2/Mon Sep  1 15:45:56 2003//
> /pre-crt0.c/1.2/Mon Sep  1 15:45:56 2003//
> /stamp-h.in/1.1/Mon Apr 15 13:14:46 2002//
> /temacs.opt/1.1/Fri Oct  1 11:44:30 1999//
> /unexhp9k800.c/1.10/Mon Sep  1 15:45:58 2003//
> /vlimit.h/1.2/Mon Sep  1 15:45:58 2003//
> 
> I did "cvs co" on March 24, 2005.  I did not count the "cvs up" I did
> since then, the last one was a week ago.  Can you explain how dates in
> the range from 1999 till 2004 made it to my Entries?  IMHO, these dates
> must reflect the corresponding dates in the repository.

Maybe, I don't know anymore.  I have 2 checkouts of the Emacs tree on
my system, one has the same times as you show above (well, almost:
acldef.h has the time stamp of 15:45:51, not 15:45:52, which is also
weird), while the other checkout has a very different time, which
seems to be the time I checked out the tree.

> http://search.cpan.org/~shay/Win32-UTCFileTime-1.45/lib/Win32/UTCFileTime.pm
> 
> tells me
> 
> "... that the problem with FindFirstFile() on a FAT drive actually seems
> to be this: the file time retrieved is wrong by one hour if it was set
> in the opposite DST season to the current system time.

I wasn't talking about DST, I was talking about the time-zone offset
from UTC.  Most time zones have non-zero offset between UTC and the
local time even when the DST is not in effect.

>  >>On my system `file-attributes' represent local time.
>  >
>  >
>  > Really?  Can you show me why you think so?
>  >
>  > AFAICS, file-attributes returns the times it gets from `stat', and
>  > `stat' (we use a replacement in w32.c) simply calls FindFirstFile.
>  > Now, this URL in the Microsoft docs:
>  >
>  >   http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/file_times.asp
>  >
>  > says that FindFirstFile (and other similar system calls) return file
>  > times in UTC, even on FAT filesystems.
> 
> It says:
> 
> "The FAT file system stores time values based on the local time of the
> computer. For example, a file that is saved at 3:00pm PST in Washington
> is seen as 6:00pm EST in New York on an NTFS volume, but it is seen as
> 3:00pm EST in New York on a FAT volume."
> 
> Please observe the use of the terms "stores" and "seen".  It's what I
> tried to explain above: I set my computer's time-zone to PST, save a
> file at 3:00 PST according to my computer's clock, set my computer's
> time-zone to EST, the filetime shows 3:00 EST.

What program(s) did you use to show the time?  I was only talking
about what FindFirstFile returns.  To see that, you need to display
the time values returned by FindFirstFile; if you use some other
program to interpret those values, you risk inaccuracies due to
conversions made by that other program.

> That's incorrect, and
> the cause is that my filetimes are local time.  Your system should get
> you 6:00 EST instead because your filetimes are UTC.

I was talking about what FindFirstFile returns, not about what's
recorded on disk.  There's no argument that NTFS records file times in
UTC, while FAT records them in local time.  The question is: does
FindFirstFile convert file times to UTC on FAT volumes; the MS docs
tells quite clearly that it does.  Can you please verify that
directly, and show the actual numbers returned by FindFirstFile on
your system?

> "FindFirstFile retrieves the local time from the FAT file system and
> converts it to UTC by using the current settings for the time zone and
> daylight saving time. Therefore, if it is daylight saving time,
> FindFirstFile takes daylight saving time into account, even if the file
> time you are converting is in standard time."
> 
> Please observe the use of the term "current settings" here.  That's
> precisely the problem I experience now: Today, "it is daylight saving
> time" and the problematic files are "in standard time".

But even if the DST settings are applied incorrectly, FindFirstFile
_does_ convert to UTC, according to the above quote.  For example, if
my standard time is UTC + 0200, then FindFirstFile should subtract 2
hours from the local times stored by the FAT filesystem, even if it
thinks the DST is not in effect.  If it thinks that DST is in effect,
then it should subtract 3 hours.  So there's an uncertainty between 2
and 3 hours, but there should be NO uncertainty about the fact that
FindFirstFile subtracts at least 2 hours in my timezone, trying to
convert local times to UTC.  Are we in agreement about this?

> "GetFileTime retrieves cached UTC times from the FAT file system."
> 
> AFAICT, this means that FAT based Windows' cache the results of the
> calculations I described above.  Hence, using GetFileTime might solve
> the problem for FAT provided I invalidate the cache after each DST
> change.

Again, you are talking about the DST, while I was talking about the
time-zone offset, even when there's no DST factor.  I agree that the
DST rules are applied incorrectly on Windows.

Re: vc-cvs-parse-entry

[martin rudalics]

 > Maybe, I don't know anymore.  I have 2 checkouts of the Emacs tree on
 > my system, one has the same times as you show above (well, almost:
 > acldef.h has the time stamp of 15:45:51, not 15:45:52, which is also
 > weird), while the other checkout has a very different time, which
 > seems to be the time I checked out the tree.

I'm afraid you don't have a backup of the files from the first checkout.
Otherwise you could have a look at their modification times.

The off by one error might have something to do with the resolution of
Windows time and/or some arithmetics around that - didn't we discuss a
similar subject before?  Anyway, it's not related to Emacs.

 >>http://search.cpan.org/~shay/Win32-UTCFileTime-1.45/lib/Win32/UTCFileTime.pm
 >>
 >>tells me
 >>
 >>"... that the problem with FindFirstFile() on a FAT drive actually seems
 >>to be this: the file time retrieved is wrong by one hour if it was set
 >>in the opposite DST season to the current system time.
 >
 >
 > I wasn't talking about DST, I was talking about the time-zone offset
 > from UTC.  Most time zones have non-zero offset between UTC and the
 > local time even when the DST is not in effect.

Yes, but my problem isn't with time-zones.  It's with DST just as
described there.

 >>That's incorrect, and
 >>the cause is that my filetimes are local time.  Your system should get
 >>you 6:00 EST instead because your filetimes are UTC.
 >
 >
 > I was talking about what FindFirstFile returns, not about what's
 > recorded on disk.  There's no argument that NTFS records file times in
 > UTC, while FAT records them in local time.  The question is: does
 > FindFirstFile convert file times to UTC on FAT volumes; the MS docs
 > tells quite clearly that it does.  Can you please verify that
 > directly, and show the actual numbers returned by FindFirstFile on
 > your system?

You earlier said that `stat' runs FindFirstFile.  Without options stat
(version 5.3.0) gives for my standard time-zone settings (CEST):

   File: `c:\\Programme\\Emacs\\src\\acldef.h'
   Size: 1188      	Blocks: 8          IO Block: 4096   regular file
Device: 181f1301h/404689665d	Inode: 48878265426772015  Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/  martin)   Gid: (    0/ UNKNOWN)
Access: 2006-09-13 00:00:00.000000000 +0200
Modify: 2003-09-01 17:45:52.000000000 +0200
Change: 2006-08-21 14:32:55.710000000 +0200

   File: `c:\\Programme\\Emacs\\src\\alloca.c'
   Size: 15285     	Blocks: 32         IO Block: 4096   regular file
Device: 181f1301h/404689665d	Inode: 48878269721739315  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/  martin)   Gid: (    0/ UNKNOWN)
Access: 2006-09-13 00:00:00.000000000 +0200
Modify: 2004-01-30 17:10:02.000000000 +0100
Change: 2006-08-21 14:32:55.740000000 +0200

and tersely

c:\Programme\Emacs\src\acldef.h 1188 8 81b6 0 0 181f1301 48878265426772015 1 0 0 1158098400 1062431152 1156163575 4096
c:\Programme\Emacs\src\alloca.c 15285 32 81ff 0 0 181f1301 48878269721739315 1 0 0 1158098400 1075479002 1156163575 4096

Changing my time-zone to GMT (_without_ changing the DST setttings) gets
me:

   File: `c:\\Programme\\Emacs\\src\\acldef.h'
   Size: 1188      	Blocks: 8          IO Block: 4096   regular file
Device: 181f1301h/404689665d	Inode: 48878265426772015  Links: 1
Access: (0666/-rw-rw-rw-)  Uid: (    0/  martin)   Gid: (    0/ UNKNOWN)
Access: 2006-09-13 00:00:00.000000000 +0100
Modify: 2003-09-01 17:45:52.000000000 +0100
Change: 2006-08-21 14:32:55.710000000 +0100

   File: `c:\\Programme\\Emacs\\src\\alloca.c'
   Size: 15285     	Blocks: 32         IO Block: 4096   regular file
Device: 181f1301h/404689665d	Inode: 48878269721739315  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/  martin)   Gid: (    0/ UNKNOWN)
Access: 2006-09-13 00:00:00.000000000 +0100
Modify: 2004-01-30 17:10:02.000000000 +0000
Change: 2006-08-21 14:32:55.740000000 +0100

and tersely

c:\Programme\Emacs\src\acldef.h 1188 8 81b6 0 0 181f1301 48878265426772015 1 0 0 1158102000 1062434752 1156167175 4096
c:\Programme\Emacs\src\alloca.c 15285 32 81ff 0 0 181f1301 48878269721739315 1 0 0 1158102000 1075482602 1156167175 4096

Can you tell something from that?

 > But even if the DST settings are applied incorrectly, FindFirstFile
 > _does_ convert to UTC, according to the above quote.  For example, if
 > my standard time is UTC + 0200, then FindFirstFile should subtract 2
 > hours from the local times stored by the FAT filesystem, even if it
 > thinks the DST is not in effect.  If it thinks that DST is in effect,
 > then it should subtract 3 hours.  So there's an uncertainty between 2
 > and 3 hours, but there should be NO uncertainty about the fact that
 > FindFirstFile subtracts at least 2 hours in my timezone, trying to
 > convert local times to UTC.  Are we in agreement about this?

Hopefully.  Looks like FindFirstFile performs some calculations that are
undone by stat afterwards.  But, honestly, I'm too silly to understand
that.  In particular, DCode when applied to the "terse" output of stat
gives results that differ from those produced by stat without options.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Thu, 14 Sep 2006 10:40:04 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
>  > I wasn't talking about DST, I was talking about the time-zone offset
>  > from UTC.  Most time zones have non-zero offset between UTC and the
>  > local time even when the DST is not in effect.
> 
> Yes, but my problem isn't with time-zones.  It's with DST just as
> described there.

Understood and agreed.  However, this confusion started when you
claimed that FindFirstFile did not return UTC times, but local times.
I hope now we agree that it does return UTC times, except that those
times are sometimes off by one hour due to the incorrect DST
correction.

>  > I was talking about what FindFirstFile returns, not about what's
>  > recorded on disk.  There's no argument that NTFS records file times in
>  > UTC, while FAT records them in local time.  The question is: does
>  > FindFirstFile convert file times to UTC on FAT volumes; the MS docs
>  > tells quite clearly that it does.  Can you please verify that
>  > directly, and show the actual numbers returned by FindFirstFile on
>  > your system?
> 
> You earlier said that `stat' runs FindFirstFile.  Without options stat
> (version 5.3.0) gives for my standard time-zone settings (CEST):

I meant the _function_ `stat' as implemented in Emacs's w32.c.  I did
not mean the _program_ `stat' that is part of Coreutils.

If you can step with a debugger through the function `stat' in w32.c
and see what times it returns for your files, it'd be great.

> Can you tell something from that?

Not really.  It's quite obvious that `stat' the program converts the
times to local time for display (witness the +0200 etc. time-zone
offsets it shows).  I need to see the raw times returned by the
FindFirstFile API.

Anyway, I'm not sure anymore whether we should pursue this issue any
further, especially since Stefan thinks we shouldn't try fixing this.

Re: vc-cvs-parse-entry

[Eli Zaretskii]

> Date: Thu, 14 Sep 2006 10:40:04 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC:  emacs-devel@gnu.org
> 
> In particular, DCode when applied to the "terse" output of stat
> gives results that differ from those produced by stat without
> options.

DCode interprets the times as UTC, while `stat' (the program) reports
local times.