Re: Time resolution in Emacs argument optional ones

Re: Time resolution in Emacs argument optional ones

[Eli Zaretskii]

This discussion was moved from bug#54764, because it sounds like a
general design issue is at stake here, and IMO we should discuss this
before making any changes.

I've removed the bug tracker and the Org and Gnulib lists from the
addressees, because I think this is strictly an Emacs design issue.
We can add them later if needed.  For now, please refrain from adding
them, to make the signal-to-noise ratio as best as possible.

> Date: Thu, 21 Apr 2022 16:56:25 -0700
> Cc: manikulin@gmail.com, emacs-orgmode@gnu.org, 54764@debbugs.gnu.org,
>  bug-gnulib@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
>  > don't we use time values for file timestamps?
> 
> Yes, but file timestamps should use the resolution of the file system, 
> which in general is different from the resolution of the system clock. 
> That's a separate matter, which would be the subject of a separate 
> patch. For now we can stick with what we already have in that department.

Sorry, I don't understand what that "for now" means.  We are talking
about the design of how time objects are represented in Emacs, so that
design should cover all the possible uses of such objects, certainly
those uses which we already know about.  And that definitely includes
file timestamps.

So it is NOT a separate matter.  It should be part of the discussion.

>  > And for Windows, all this does is measure the "resolution" of the
>  > Gnulib emulation of timespec functions on MS-Windows, it tells nothing
>  > about the real resolution of the system time values.
> 
> If Emacs Lisp code (which currently is based on the Gnulib code) can see 
> only (say) 1-microsecond timestamps, then it doesn't matter that the 
> underlying system clock has (say) 1-nanosecond precision. Of course it 
> would be better for Emacs to see the system timestamp resolution, and if 
> we can get the time of day on MS-Windows to a precision better than 1/64 
> second then I assume Emacs should do that. Once it does, the patch 
> should calculate a higher HZ value to tell users about the improved 
> resolution.

You again are talking about the current implementation, whereas I
would like to discuss the design and our goals.  There's no problem
whatsoever to provide high-resolution time stamps on MS-Windows, if we
decide that this is what we want.  The only reason we didn't do that
until now is that it wasn't deemed important.

>  > if the "time resolution" determined by this procedure
>  > is different between two systems, does it mean that two time values
>  > that are 'equal' on one of them could be NOT 'equal' on another?
> 
> Sure, but the traditional (HIGH LOW MICROSEC PICOSEC) representation has 
> the same issue. For example, if A's clock has 1 ms resolution and B's 
> clock has 10 ms resolution, A's (time-convert nil 'list) called twice 
> would return (say) the two timestamps (25184 64239 1000 0) and (25184 
> 64239 1001 0) at the same moments that B's calls would return (25184 
> 64239 1000 0) twice. A would say that the two timestamps differ; B would 
> say they're the same.

Then it's a subtle bug in our current code that needs to be solved.

> This sort of disagreement is inherent to how timestamp resolution works. 

I'm asking why timestamp resolution, as defined by HZ, at all matters
in Emacs.  Please answer this question before anything else, because
it is IMO central to this discussion.  In addition to the general
cause you think this is important, please provide a list of use cases
where you can show that using this definition of time resolution is
important, and not using that would lead to problems.

An alternative would be to decide on a unified resolution for time
objects in Emacs, independent on what the underlying system does, and
then use that everywhere.  Why not do that?

The next important question is how do we define "timestamp resolution"
so that it covers all the uses, including file timestamps, time values
that come from external systems, etc.  (Granted, the simple
alternative of using a single unified resolution that is fine-grained
enough to cover all the known uses will solve this problem as well,
and AFAIU solve it simply and cleanly.)

I think we should only change our code after we conclude the
discussion of the above-mentioned general issues, and agree on our
goals and how we will implement those goals.

Thanks.

Re: Time resolution in Emacs argument optional ones

[Paul Eggert]

On 4/21/22 22:23, Eli Zaretskii wrote:

> There's no problem
> whatsoever to provide high-resolution time stamps on MS-Windows, if we
> decide that this is what we want.  The only reason we didn't do that
> until now is that it wasn't deemed important.

An example of when this matters is that GNU 'ls' needs to know whether a 
file timestamp is in the future because 'ls -l' uses a different textual 
format for future timestamps than it does for recent-past timestamps, as 
a visual cue to the user that the user has a clock skew problem that can 
screw up GNU 'make' and similar programs. For example, if I run the 
shell script:

   touch now
   touch --date=@$(date +%s.%N | sed 's/........$/99999999/') future
   ls -l future now

Here's the output I should see, because 'future' has a timestamp in the 
future:

   -rw-rw-r--. 1 eggert eggert 0 Apr 22  2022 future
   -rw-rw-r--. 1 eggert eggert 0 Apr 22 11:13 now

However, if GNU 'ls' got the time of day with lower resolution than it 
got file timestamps, it would use the wrong textual format for a 
recently-created file's timestamp, because it would compare a lower-res 
time-of-day like 1650651200.73 to a higher-res file timestamp like 
1650651200.734627956 that was generated before the current time. The 
above example would therefore generate the incorrect output:

   -rw-rw-r--. 1 eggert eggert 0 Apr 22 11:13 future
   -rw-rw-r--. 1 eggert eggert 0 Apr 22 11:13 now

When Emacs needs to do something like what GNU 'ls' does, it can have 
the same issue.

Admittedly this is not the most important issue on Emacs's plate. Still, 
Emacs should do a good job and be compatible with other GNU tools.


>> This sort of disagreement is inherent to how timestamp resolution works.
> 
> I'm asking why timestamp resolution, as defined by HZ, at all matters
> in Emacs.

It matters for applications like GNU 'ls' as described above. It also 
matters for applications that want to display a timestamp accurately, 
but without excess information that bogs down performance and implies 
more precision than is actually present. For example, current GNU 'date' 
can use a format '%s.%-N' that outputs the current time without excess 
trailing zeros in the subseconds part. Emacs should be able to do 
something similar.


> please provide a list of use cases
> where you can show that using this definition of time resolution is
> important, and not using that would lead to problems.

Doing what GNU 'ls' does, and what GNU 'date' does, are two use cases. 
Another example is doing what GNU 'cp -u' does when comparing file 
timestamps. I expect there are others.


> An alternative would be to decide on a unified resolution for time
> objects in Emacs, independent on what the underlying system does, and
> then use that everywhere.  Why not do that?

We've already done that in Emacs with the (TICKS . HZ) representation.

The obvious fix here would be to use better time-of-day primitives on 
Microsoft Windows, in particular, primitives that don't use a syncopated 
clock. (By "syncopated clock" I mean one that doesn't beat on 1-second 
boundaries.) This would be simpler than adding support for syncopated 
clocks to the Emacs timestamp format, as syncopated clocks are too much 
trouble to document and implement and would complicated too much code 
for too little benefit.

Besides, a precision of only 1/64 second on the time of day is not 
adequate for quite a few applications these days, so this issue of 
low-resolution time-of-day should be fixed on MS-Windows regardless of 
how Emacs represents timestamps internally.


>> > Yes, but file timestamps should use the resolution of the file system, 
>> > which in general is different from the resolution of the system clock. 
>> > That's a separate matter, which would be the subject of a separate 
>> > patch. For now we can stick with what we already have in that department.
> 
> Sorry, I don't understand what that "for now" means.

I meant that we can stick with what we currently do for file timestamps, 
which is to generate Emacs Lisp timestamps with a greater resolution 
than what the file timestamps actually have. This loses information 
about the resolution, but does not lose information about the time values.

This doesn't mean I'm a fan of the current situation with file 
timestamps. It means only that we can address the current-time problem 
in one patchset, and the file timestamp issue in a later patchset. It 
will be better to separate the solutions in this way, than to try to 
solve the problem all at once.

> It should be part of the discussion.

OK.

Re: Time resolution in Emacs argument optional ones

[Corwin Brust]

On Fri, Apr 22, 2022 at 1:22 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> Besides, a precision of only 1/64 second on the time of day is not
> adequate for quite a few applications these days, so this issue of
> low-resolution time-of-day should be fixed on MS-Windows regardless of
> how Emacs represents timestamps internally.

Since this seems to be in reply to Eli's question about use-cases, can
you elaborate on some of the specific use-cases where 1/64th of a
second is insufficient?

Thanks very much!
Corwin

Re: Time resolution in Emacs argument optional ones

[Eli Zaretskii]

> Date: Fri, 22 Apr 2022 11:22:11 -0700
> Cc: manikulin@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> > please provide a list of use cases
> > where you can show that using this definition of time resolution is
> > important, and not using that would lead to problems.
> 
> Doing what GNU 'ls' does, and what GNU 'date' does, are two use cases. 
> Another example is doing what GNU 'cp -u' does when comparing file 
> timestamps. I expect there are others.

It is ironic that you wanted to leave file timestamps alone, but
almost all of the examples you show are related to comparing file
timestamps with other timestamps.

I think there's no problem whatsoever in what we do as long as we
compare times whose source is the same, i.e. has the same resolution.
For example, even with the 15.6 msec resolution of the system time
functions on MS-Windows, we have no problems with stuff like timers in
Emacs -- we just lose the capability of reliably running timers with
time intervals smaller then the resolution.  But we never err about
which system time is later.

It is only when comparing timestamps from different sources that the
problem might appear.  But to do such comparisons correctly we need to
have each time object state its resolution, otherwise you will never
know how to compare them.  We cannot simplify by assuming that one of
them is more accurate and use the resolution of the less accurate one,
because such assumptions will never hold.  For example, while it's
true that file timestamps are more accurate on MS-Windows than the
system clock, sometimes it's the other way around: when using FAT32
volumes, the file time resolution is 2 sec (and, btw, that affects
GNU/Linux systems as well, when they use such filesystems).  Moreover,
some filesystems have different time resolution for different times
(creation, modification, access) on the same type of volume.

So, then, to do this stuff correctly, we need to assign resolution to
each time object.  That resolution cannot be possibly determined by
measuring the system clock resolution, because doing that will at best
provide the figure for one class of time objects.  ("At best" because,
as we saw, the current implementation of measuring that resolution
fails miserably on MS-Windows, producing a very incorrect value.)
What is the plan for other kinds of time objects?  For example, how
would we determine the resolution of file times -- do we have the
technology of telling that for each time field on each type of
filesystem we want to support?  And what about other types of time
objects, for example, those that come via Tramp from remote hosts --
how do we measure or otherwise tell their resolution?

Without having some plan to obtain reliable resolution figures in all
these and other important use cases, it makes no sense to me to
introduce resolution for just one kind of timestamp, especially since,
as your examples show, that kind of timestamp is the one that needs
resolution information much less than the other ones.

> The obvious fix here would be to use better time-of-day primitives on 
> Microsoft Windows, in particular, primitives that don't use a syncopated 
> clock.

Please don't be obsessed with MS-Windows.  The issue is much more
general and acute, as I tried to explain above.  What is the plan for
resolving the more general issue given the state of our knowledge and
technology, and the variety of sources of time data, each one with its
own resolution?  Is this at all feasible enough to even try?  It makes
no sense to me to start if we cannot solve this at least in most
cases.

> I meant that we can stick with what we currently do for file timestamps, 
> which is to generate Emacs Lisp timestamps with a greater resolution 
> than what the file timestamps actually have. This loses information 
> about the resolution, but does not lose information about the time values.

I don't understand this.  Two out of 3 examples you gave deal with
file timestamps, and you explained that we do risk losing information
there.  Given this importance of file timestamps, without having some
plan of solving that problem, the rest is actually a moot point,
because even if we consider the (contrived) example with 'date'
important for some reason, the result will be incorrect there for a
couple of milliseconds, and thereafter will become correct, and who's
to know, or care, that it was incorrect for those 2 milliseconds?

IOW, as long as we are talking about timestamps from the same source,
in this case from the system clock, the problem is so infinitesimally
small that it's almost academic, not a practical one.  So it makes
little sense to me to make changes just for that one class of
timestamps without a clear plan for the other classes.

Re: Time resolution in Emacs argument optional ones

[Paul Eggert]

On 4/22/22 11:52, Corwin Brust wrote:
> Since this seems to be in reply to Eli's question about use-cases, can
> you elaborate on some of the specific use-cases where 1/64th of a
> second is insufficient?

When Emacs needs to deal with more than 64 spaced events per second, or 
with external time sources that deal with more than 64 such events per 
second, a resolution of 1/64 second is inadequate.

In my previous message I gave the example of comparing the current time 
in 1/64-second resolution to file timestamps in (say) 100 ns resolution, 
where Emacs can compare timestamps incorrectly.

Another example would be comparing internally-generated timestamps to 
Internet RFC 3339 or ISO 8601 timestamps retrieved from the outside 
world. LANs synchronized via NTP can routinely synchronize clocks to 
less than 10 ms, and processes running on a single node can synchronize 
even more closely. But if Emacs reports the current time to within only 
1/64 s, its timestamps introduce more clock skew than its underlying 
platforms do, making it harder to build distributed apps with 
high-quality timestamps.

Of course most apps don't need high-quality timestamps. However, I don't 
see why Emacs would insist on generating low-quality timestamps if 
higher-quality timestamps are readily available.

Re: Time resolution in Emacs argument optional ones

[Paul Eggert]

On 4/22/22 12:35, Eli Zaretskii wrote:

> to do such comparisons correctly we need to
> have each time object state its resolution, otherwise you will never
> know how to compare them.

Yes, quite true.


> That resolution cannot be possibly determined by
> measuring the system clock resolution, because doing that will at best
> provide the figure for one class of time objects.

Also quite right. Emacs needs to determine the timestamp for each class 
of objects as best it can.


> the current implementation of measuring that resolution
> fails miserably on MS-Windows, producing a very incorrect value.)

It's a correct value for what it's measuring, which is (roughly) the 
number of digits needed to express each timestamp precisely. It's not 
correct for a different measure, namely the timestamp resolution, which 
it sounds like is what you'd prefer. As I mentioned in 
<https://bugs.gnu.org/54764#111> it shouldn't be hard to support this 
other measure, and I can look into doing that.


> What is the plan for other kinds of time objects?  For example, how
> would we determine the resolution of file times

We could do what Coreutils does, in commands like 'cp -u' that sample 
timestamps in each file system to determine resolution. This approach 
works well enough in practice. There are Gnulib library functions to 
deal with this sort of thing; it's not a new problem.


> as your examples show, that kind of timestamp is the one that needs
> resolution information much less than the other ones.

I don't see how the examples show that.


> Please don't be obsessed with MS-Windows.

There's no obsession here on my part. MS-Windows appears to be the only 
platform where current-time generates such low-resolution timestamps, or 
generates syncopated timestamps, and that's why discussion has naturally 
centered on that platform.


> examples you gave deal with
> file timestamps, and you explained that we do risk losing information
> there.

Yes, and Emacs currently loses all information about file timestamp 
resolution, on all platforms.  It would be better for Emacs to report as 
much of that information as it easily can. This could be done by using 
the Gnulib library functions suggested above. Reporting this information 
is better than losing it.


> even if we consider the (contrived) example with 'date'
> important for some reason, the result will be incorrect there for a
> couple of milliseconds, and thereafter will become correct, and who's
> to know, or care, that it was incorrect for those 2 milliseconds?

Yes, most users don't notice timestamps being a little off. And these 
users surely won't care whether we improve the timestamps to be more 
accurate. But that doesn't mean that it's a bad idea to improve 
timestamp quality.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Fri, 22 Apr 2022 14:26:16 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>, manikulin@gmail.com,
>  Emacs developers <emacs-devel@gnu.org>
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 4/22/22 11:52, Corwin Brust wrote:
> > Since this seems to be in reply to Eli's question about use-cases, can
> > you elaborate on some of the specific use-cases where 1/64th of a
> > second is insufficient?
> 
> When Emacs needs to deal with more than 64 spaced events per second, or 
> with external time sources that deal with more than 64 such events per 
> second, a resolution of 1/64 second is inadequate.
> 
> In my previous message I gave the example of comparing the current time 
> in 1/64-second resolution to file timestamps in (say) 100 ns resolution, 
> where Emacs can compare timestamps incorrectly.
> 
> Another example would be comparing internally-generated timestamps to 
> Internet RFC 3339 or ISO 8601 timestamps retrieved from the outside 
> world. LANs synchronized via NTP can routinely synchronize clocks to 
> less than 10 ms, and processes running on a single node can synchronize 
> even more closely. But if Emacs reports the current time to within only 
> 1/64 s, its timestamps introduce more clock skew than its underlying 
> platforms do, making it harder to build distributed apps with 
> high-quality timestamps.

You will excuse me, but I find these examples artificial when Emacs
Lisp programs are concerned.  There's no reason to expect Emacs to be
able to support such applications on all platforms.  Not to mention
that synchronizing clocks to millisecond accuracy on non-RT systems is
in most cases futile, because the OS doesn't provide reliable timings
to that accuracy anyway.

> Of course most apps don't need high-quality timestamps. However, I don't 
> see why Emacs would insist on generating low-quality timestamps if 
> higher-quality timestamps are readily available.

We don't _insist_ on providing low-accuracy timestamps, but we should
definitely be certainly concerned about adding such non-trivial
complexity to Emacs for the benefit of extremely rare (if any) use
cases.  So far I have yet to see an example of an important use case
that would justify that, let alone a plan for reasonably practical
ways of providing the time resolution figures for each source of time
information (see my other message).

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Fri, 22 Apr 2022 14:52:47 -0700
> Cc: manikulin@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> > That resolution cannot be possibly determined by
> > measuring the system clock resolution, because doing that will at best
> > provide the figure for one class of time objects.
> 
> Also quite right. Emacs needs to determine the timestamp for each class 
> of objects as best it can.

The question is: how feasible is that?  See below; my current
conclusion is that it is not feasible in general.

> > What is the plan for other kinds of time objects?  For example, how
> > would we determine the resolution of file times
> 
> We could do what Coreutils does, in commands like 'cp -u' that sample 
> timestamps in each file system to determine resolution. This approach 
> works well enough in practice. There are Gnulib library functions to 
> deal with this sort of thing; it's not a new problem.

Doesn't that slow down applications?  How many time stamps should we
need to sample to determine a good estimation?  The sampling should be
separate for each filesystem, but do we have information about the
file's filesystem in all cases?  What about the cases of different
accuracies for different times (creation, modification, access)?

These questions need to have practical reasonable answers before we
decide this is tractable.

I also question the need for Emacs to do what few other applications
do (GNU Coreutils are an exception rather than the rule here, AFAIK),
given the tremendous complexities this will require from us.  For
example, 'cp' specifically deals with copying many files, but that's
not what Emacs does in most applications where it cares about
comparing file timestamps with some other class of timestamps.  So
where 'cp' can sample file timestamps of the files it is copying,
Emacs will be unable to do so, without costly traversing directories
it has no business of looking at, and for which it may not have access
rights in the first place.

> > as your examples show, that kind of timestamp is the one that needs
> > resolution information much less than the other ones.
> 
> I don't see how the examples show that.

The 'date' example is IMNSHO artificial and not important, except for
people whose special interest is in time APIs.

> > Please don't be obsessed with MS-Windows.
> 
> There's no obsession here on my part. MS-Windows appears to be the only 
> platform where current-time generates such low-resolution timestamps, or 
> generates syncopated timestamps, and that's why discussion has naturally 
> centered on that platform.

That's what I'm saying: don't center the discussion on Windows.  The
problem is much more general and broad, see above and below.

> > examples you gave deal with
> > file timestamps, and you explained that we do risk losing information
> > there.
> 
> Yes, and Emacs currently loses all information about file timestamp 
> resolution, on all platforms.  It would be better for Emacs to report as 
> much of that information as it easily can. This could be done by using 
> the Gnulib library functions suggested above. Reporting this information 
> is better than losing it.

Please point to those function so they could be studied, or describe
what they do here.  I don't see how this could be reasonably solved in
general, even for file timestamps.

And what about other sources of time data?  For example, timestamps
from other systems (like in your NTP example), timestamps that are
reported by Tramp functions, etc.?  How do we reliably estimate the
accuracy of those?

Once again, without a comprehensive plan that is capable of covering
all the sources of time information we routinely deal with in Emacs, I
don't think we should make any significant changes in our time-API
infrastructure nor in the structure of our time representation
objects.

> > even if we consider the (contrived) example with 'date'
> > important for some reason, the result will be incorrect there for a
> > couple of milliseconds, and thereafter will become correct, and who's
> > to know, or care, that it was incorrect for those 2 milliseconds?
> 
> Yes, most users don't notice timestamps being a little off. And these 
> users surely won't care whether we improve the timestamps to be more 
> accurate. But that doesn't mean that it's a bad idea to improve 
> timestamp quality.

It isn't a bad idea, but if the complexity it introduces into the way
Emacs handles time information is too high and the benefits are too
low, then we shouldn't add such complexities to Emacs, because the
benefits won't justify the costs.

So far I'm not convinced it is feasible to solve these issues in Emacs
without introducing unreasonable complexity, and even that will only
provide a partial solution.  Once we understand which part(s) of this
can be solved and at what cost, we could perhaps come up with
simplified solutions that avoid most of the costs, such as using a
static database of timestamp accuracies for important classes of
timestamps.  But we aren't there yet, and it is so far premature to
discuss these specifics.

Re: Time resolution in Emacs

[Paul Eggert]

On 4/22/22 23:27, Eli Zaretskii wrote:

> There's no reason to expect Emacs to be
> able to support such applications on all platforms.

Quite right. Many people don't synchronize their clocks at all, and on 
their systems Emacs can't generate accurate timestamps.

However, that doesn't mean Emacs should be sloppy about timestamps. If 
an OS has two low-level time primitives A and B, and A's resolution is 
100 ms while B's is 1 ms, surely Emacs should prefer B. Using B helps 
apps that can use 1-ms timestamps in a well-synchronized environment, 
and it doesn't hurt apps that don't care about timestamp resolution.


>  Not to mention
> that synchronizing clocks to millisecond accuracy on non-RT systems is
> in most cases futile, because the OS doesn't provide reliable timings
> to that accuracy anyway.

Even if one limits one's attention to NTP on non-realtime systems it's 
not hard to get agreement on a LAN to better than 10 ms, counting OS 
jitter. And better-than-10-ms accuracy is growing in popularity, due to 
applications that need good clocks and get them via PTP or other means. 
I regularly use networks that are synchronized better than 10 ms, and I 
wouldn't pooh-pooh the need for this sort of thing in apps that 
coordinate with others.


> We don't _insist_ on providing low-accuracy timestamps, but we should
> definitely be certainly concerned about adding such non-trivial
> complexity to Emacs

It sounds like we have miscommunicated, as this comment seems to 
disagree with your email of a couple of days ago that said "There's no 
problem whatsoever to provide high-resolution time stamps on MS-Windows".

If the question is whether to use a less-accurate method A or 
more-accurate method B to obtain timestamps, where Emacs continues its 
current practice of converting the timestamps to 1-nanosecond resolution 
internally before the user sees the timestamps, then I don't see why 
Emacs should prefer A to B.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Sat, 23 Apr 2022 17:56:01 -0700
> Cc: corwin@bru.st, manikulin@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 4/22/22 23:27, Eli Zaretskii wrote:
> 
> > There's no reason to expect Emacs to be
> > able to support such applications on all platforms.
> 
> Quite right. Many people don't synchronize their clocks at all, and on 
> their systems Emacs can't generate accurate timestamps.
> 
> However, that doesn't mean Emacs should be sloppy about timestamps. If 
> an OS has two low-level time primitives A and B, and A's resolution is 
> 100 ms while B's is 1 ms, surely Emacs should prefer B. Using B helps 
> apps that can use 1-ms timestamps in a well-synchronized environment, 
> and it doesn't hurt apps that don't care about timestamp resolution.

Everything else being equal, yes.  But if using A requires complex
code that will require a lot of maintenance, and if using the results
of A will affect most or all of the Emacs applications that use
timestamps, then we might as well conclude that using A does "hurt" is
enough to reject it, and make do with a lower time accuracy.

And it's not like the current situation is unbearable: Emacs does work
with millisecond time intervals, and works reasonably well.  We are
nowhere near the 1-sec resolution of your argument above, we are waaay
better than that.  You are actually arguing for moving from
millisecond resolution to nanosecond resolution, and I'm asking where
are the Emacs applications that will benefit from such a high
resolution.  So far the examples you provided are marginal at best,
perhaps even irrelevant to what we do or want to do in Emacs.

> >  Not to mention
> > that synchronizing clocks to millisecond accuracy on non-RT systems is
> > in most cases futile, because the OS doesn't provide reliable timings
> > to that accuracy anyway.
> 
> Even if one limits one's attention to NTP on non-realtime systems it's 
> not hard to get agreement on a LAN to better than 10 ms, counting OS 
> jitter. And better-than-10-ms accuracy is growing in popularity, due to 
> applications that need good clocks and get them via PTP or other means. 
> I regularly use networks that are synchronized better than 10 ms, and I 
> wouldn't pooh-pooh the need for this sort of thing in apps that 
> coordinate with others.

That misses the point.  The point is that on a non-RT system there's
precious little one can reliably do with time intervals shorter than
10 ms.

> > We don't _insist_ on providing low-accuracy timestamps, but we should
> > definitely be certainly concerned about adding such non-trivial
> > complexity to Emacs
> 
> It sounds like we have miscommunicated, as this comment seems to 
> disagree with your email of a couple of days ago that said "There's no 
> problem whatsoever to provide high-resolution time stamps on MS-Windows".

You are again focused on low-level details of MS-Windows
implementation of the relevant primitives, whereas I'm talking about
much more general issues that affect all of Emacs on any platform.

My point is that your current plan, such as it is, for providing the
time accuracy for every source of time information we care about means
adding non-trivial complexity to Emacs, on all platforms, and I have
yet to see any evidence that what you propose is capable of solving
any large part of the problems we could face wrt comparing timestamps
from different sources using reasonably practical techniques.  So my
current conclusion is that, if anything, we should look for much
simpler solutions, which might move us closer to the goal at much
lower costs.  For example:

  . provide optional arguments to time-comparison APIs that allow to
    pass the comparison accuracy, leaving the specification of that
    accuracy to the caller;
  . provide some simple functions that return estimated accuracy of a
    time source, using a static database of known timestamp accuracies

I see no reason to go any further, and I definitely don't see any
justification for making the time accuracy be a mandatory part of any
time specification.  The needs for such accurate methods in the Emacs
world are very rare, not to say nonexistent, so we should not force on
everyone something that will only be needed by a very few.

> If the question is whether to use a less-accurate method A or 
> more-accurate method B to obtain timestamps, where Emacs continues its 
> current practice of converting the timestamps to 1-nanosecond resolution 
> internally before the user sees the timestamps, then I don't see why 
> Emacs should prefer A to B.

I do see why: the costs involved in implementing B.  Costs in coding,
testing, documenting, and maintaining that for years to come.  I tried
to explain that above and in my previous messages here.

Re: Time resolution in Emacs

[Max Nikulin]

Am I right that first patch "Support (encode-time (list s m h D M Y))" 
in the series we are currently discussing
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54764;filename=0001-Support-encode-time-list-s-m-h-D-M-Y.patch;msg=48;att=1
(that is the goal of the bug#54764) can be safely applied independently 
of other changes related to time resolution? Does somebody object?

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Sun, 24 Apr 2022 18:47:04 +0700
> Cc: emacs-devel@gnu.org
> From: Max Nikulin <manikulin@gmail.com>
> 
> Am I right that first patch "Support (encode-time (list s m h D M Y))" 
> in the series we are currently discussing
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54764;filename=0001-Support-encode-time-list-s-m-h-D-M-Y.patch;msg=48;att=1
> (that is the goal of the bug#54764) can be safely applied independently 
> of other changes related to time resolution? Does somebody object?

This thread is not about that bug, so please ask this question there.

Re: Time resolution in Emacs

[Paul Eggert]

On 4/23/22 23:10, Eli Zaretskii wrote:

> You are actually arguing for moving from
> millisecond resolution to nanosecond resolution

No, I am not actually arguing for that. Not at all. Again, there is 
miscommunication.

Let's drop the topic of timestamp resolution on MS-Windows. You say 
15.625 ms resolution suffices on that platform and you're an MS-Windows 
expert where I am not, and that's good enough.

Time resolution in Emacs

[Paul Eggert]

On 4/22/22 23:51, Eli Zaretskii wrote:

>> There are Gnulib library functions to
>> deal with this sort of thing; it's not a new problem.
> 
> Doesn't that slow down applications?

Not significantly, no. file-attributes already has the file's filesystem 
ID and timestamps so Emacs can easily cache that (it's a small cache 
with only one entry per filesystem). There's no need to traipse through 
the filesystem looking at other files. This is what coreutils does and 
it works well in practice. It's not expensive and not that complicated.

Your suggestion of maintaining a static table for known filesystem types 
is a good one; we could do that to improve common cases. I would like to 
take a look into doing this. If successful, it should improve coreutils 
and other GNU apps even if Emacs makes no changes in this area. In the 
meantime I'll withdraw the proposed changes that would cause Emacs to 
communicate OS timestamp resolution to the user; that sort of thing can 
wait until after I've had that look (assuming I ever find the time :-).


> what about other sources of time data?  For example, timestamps
> from other systems (like in your NTP example), timestamps that are
> reported by Tramp functions, etc.?  How do we reliably estimate the
> accuracy of those?

For example, if Emacs imports from the network a timestamp with three 
digits after the decimal point, it should convert it to an internal 
timestamp with millisecond resolution. Conversely when sending a textual 
timestamp, Emacs should generate only as many trailing digits as needed. 
This sort of thing is not perfect (nothing to do with timestamps is :-), 
but it is better than losing whatever resolution information is available.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Mon, 25 Apr 2022 08:32:36 -0700
> Cc: corwin@bru.st, manikulin@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 4/23/22 23:10, Eli Zaretskii wrote:
> 
> Let's drop the topic of timestamp resolution on MS-Windows. You say 
> 15.625 ms resolution suffices on that platform and you're an MS-Windows 
> expert where I am not, and that's good enough.

I was not talking about MS-Windows.  I'm saying that I see no reason
to do anything about timestamp resolution in Emacs _on_any_platform_.
We are okay as we are, on all platforms.

Now, if you want to argue for _any_ changes in this direction, on
_any_ of the supported platforms, please describe the changes you had
in mind and the rationale for each such change in a manner that is as
system-independent as possible.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Mon, 25 Apr 2022 08:34:04 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: manikulin@gmail.com, emacs-devel@gnu.org
> 
> On 4/22/22 23:51, Eli Zaretskii wrote:
> 
> >> There are Gnulib library functions to
> >> deal with this sort of thing; it's not a new problem.
> > 
> > Doesn't that slow down applications?
> 
> Not significantly, no. file-attributes already has the file's filesystem 
> ID and timestamps so Emacs can easily cache that (it's a small cache 
> with only one entry per filesystem). There's no need to traipse through 
> the filesystem looking at other files. This is what coreutils does and 
> it works well in practice. It's not expensive and not that complicated.

Sorry, I don't understand.  Suppose the given Emacs session called
file-attributes on only 1 file: how do we know the resolution of the
file timestamps from that single file?  The fact that a timestamp has
N last digits zero says nothing at all about the resolution.

> Your suggestion of maintaining a static table for known filesystem types 
> is a good one; we could do that to improve common cases. I would like to 
> take a look into doing this.

Thanks, I think for starters we don't need anything else, like dynamic
determination of the actual resolution.

> If successful, it should improve coreutils 
> and other GNU apps even if Emacs makes no changes in this area. In the 
> meantime I'll withdraw the proposed changes that would cause Emacs to 
> communicate OS timestamp resolution to the user; that sort of thing can 
> wait until after I've had that look (assuming I ever find the time :-).

Yes, it can definitely wait.  I'd like first to see if my proposal is
good enough, and if not, to hear really good reasons why not.

> For example, if Emacs imports from the network a timestamp with three 
> digits after the decimal point, it should convert it to an internal 
> timestamp with millisecond resolution.

I'm not sure this is TRT.  What do you do if you get a timestamp whose
fractional part looks like .123000 -- do you consider this to be
millisecond resolution or microsecond resolution?  If the latter, we'd
get in trouble as result of trivial arithmetics on timestamps; if the
former, you risk losing resolution for now good reason, just because
the time stamp was too "round".

> Conversely when sending a textual timestamp, Emacs should generate
> only as many trailing digits as needed.

It's very hard to keep only that many significant digits when working
with fractional numbers.

Re: Time resolution in Emacs

[Paul Eggert]

On 4/25/22 09:10, Eli Zaretskii wrote:
> What do you do if you get a timestamp whose
> fractional part looks like .123000 -- do you consider this to be
> millisecond resolution or microsecond resolution?

The latter. That's standard practice, pretty much everywhere.


> we'd
> get in trouble as result of trivial arithmetics on timestamps

Whatever trouble we get into, will be less trouble than we get with 
arbitrarily converting to a resolution that differs from that of the 
incoming timestamp.

More generally, we can't do a perfect job with timestamp resolution, but 
that's OK. There's never perfection in timestamps anyway. What we can 
do, is do a better job than what we're currently doing.


> It's very hard to keep only that many significant digits when working
> with fractional numbers.

It's not hard to send a textual timestamp. Emacs does this sort of thing 
routinely when printing floats, and fractions are easier than floats. 
And Emacs code that computes timestamps internally can continue to use 
whatever resolution it is currently using; it doesn't need to arbitrary 
limit that resolution.

Re: Time resolution in Emacs

[Max Nikulin]

On 25/04/2022 22:34, Paul Eggert wrote:
> On 4/22/22 23:51, Eli Zaretskii wrote:
> 
>>> There are Gnulib library functions to
>>> deal with this sort of thing; it's not a new problem.
>>
>> Doesn't that slow down applications?
> 
> Not significantly, no. file-attributes already has the file's filesystem 
> ID and timestamps so Emacs can easily cache that (it's a small cache 
> with only one entry per filesystem). There's no need to traipse through 
> the filesystem looking at other files. This is what coreutils does and 
> it works well in practice. It's not expensive and not that complicated.
> 
> Your suggestion of maintaining a static table for known filesystem types 
> is a good one; we could do that to improve common cases. I would like to 
> take a look into doing this.

Notice that on Linux while file metadata are in cache, timestamps may 
have high time resolution. When later fetched from disk they may have 
coarse resolution, e.g. whole seconds.

Generally it is fragile to rely on file modification time. That is why 
build systems relying of file hashes appeared.

As to 16ms resolution, at least for `benchmark-run' I would like to have 
finer granularity.

On the other hand, when higher timer frequency is required, from my 
point of view, it may be better to run such application in a separate 
process.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Mon, 25 Apr 2022 09:38:00 -0700
> Cc: manikulin@gmail.com, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> 
> On 4/25/22 09:10, Eli Zaretskii wrote:
> > It's very hard to keep only that many significant digits when working
> > with fractional numbers.
> 
> It's not hard to send a textual timestamp.

Yes, but the moment you do any arithmetics on that on the other end,
you start losing precision and gaining random digits.  We should keep
that in mind and not treat the results too seriously.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Mon, 25 Apr 2022 23:54:58 +0700
> Cc: emacs-devel@gnu.org
> From: Max Nikulin <manikulin@gmail.com>
> 
> As to 16ms resolution, at least for `benchmark-run' I would like to have 
> finer granularity.

Do you really run benchmarks that end in a matter of milliseconds?

In any case, measuring a time interval with high precision doesn't
necessarily need a fine-granular wall clock.

Re: Time resolution in Emacs

[Paul Eggert]

On 4/25/22 09:54, Max Nikulin wrote:

> on Linux while file metadata are in cache, timestamps may 
> have high time resolution. When later fetched from disk they may have 
> coarse resolution, e.g. whole seconds.

Yes, that is a known bug in older Linux kernels. As I understand it, the 
bug was fixed many years ago. (It broke GCC builds circa 2004[1] so 
there was a good deal of motivation to fix it.) I expect this problem is 
no longer relevant; if I'm wrong, Eli's suggestion would help to work 
around it.

[1]: 
https://linux.kernel.narkive.com/XLtQbTzQ/linux-2-6-nanosecond-time-stamp-weirdness-breaks-gcc-build

Re: Time resolution in Emacs

[Max Nikulin]

On 26/04/2022 02:27, Paul Eggert wrote:
> On 4/25/22 09:54, Max Nikulin wrote:
> 
>> on Linux while file metadata are in cache, timestamps may have high 
>> time resolution. When later fetched from disk they may have coarse 
>> resolution, e.g. whole seconds.
> 
> Yes, that is a known bug in older Linux kernels. As I understand it, the 
> bug was fixed many years ago. (It broke GCC builds circa 2004[1] so 
> there was a good deal of motivation to fix it.) I expect this problem is 
> no longer relevant; if I'm wrong, Eli's suggestion would help to work 
> around it.
> 
> [1]: 
> https://linux.kernel.narkive.com/XLtQbTzQ/linux-2-6-nanosecond-time-stamp-weirdness-breaks-gcc-build 

Thank you. I do not remember why I was aware of such peculiarity and 
considered it as a feature. I have not expected that it was fixed.

Accidentally I have found the following proposal to add fsinfo() system 
call to the Linux kernel. It should expose timestamp resolution, but 
likely the patches was not merged. I have not read the discussion though.

https://lwn.net/Articles/827934/
David Howells <dhowells-AT-redhat.com> VFS: Filesystem information [ver 
#21] Mon, 03 Aug 2020 14:36:21 +0100

On 26/04/2022 00:02, Eli Zaretskii wrote:
 >> Date: Mon, 25 Apr 2022 23:54:58 +0700
 >> From: Max Nikulin
 >
>> As to 16ms resolution, at least for `benchmark-run' I would like to have 
>> finer granularity.
> 
> Do you really run benchmarks that end in a matter of milliseconds?

It may be enough for quick estimations during comparison of methods 
having order of magnitude difference in running time.

Sometimes it is necessary to measure duration of particular calls 
without repetition.

> In any case, measuring a time interval with high precision doesn't
> necessarily need a fine-granular wall clock.

My experience is that it is even less probable to find monotonic clock 
or performance counters implemented when some legacy method is used for 
wall time.

Re: Time resolution in Emacs

[Eli Zaretskii]

> Date: Fri, 29 Apr 2022 22:19:10 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: Max Nikulin <manikulin@gmail.com>
> 
> >> As to 16ms resolution, at least for `benchmark-run' I would like to have 
> >> finer granularity.
> > 
> > Do you really run benchmarks that end in a matter of milliseconds?
> 
> It may be enough for quick estimations during comparison of methods 
> having order of magnitude difference in running time.

Ut usually produces skewed or random results, IME.

> Sometimes it is necessary to measure duration of particular calls 
> without repetition.

In which case one must arrange for repetitions that don't have the
effect of repetitions.

> > In any case, measuring a time interval with high precision doesn't
> > necessarily need a fine-granular wall clock.
> 
> My experience is that it is even less probable to find monotonic clock 
> or performance counters implemented when some legacy method is used for 
> wall time.

My experience is different.