current-time and GMP

current-time and GMP

[Stefan Monnier]

Of the functions that use "handmade" bignums, there's also current-time.
Since this one tends to grow over time in precision than in magnitude,
I'm wondering if people have an idea what to do with it:
- keep it as is (that's the easy way out)
- use GMP's mpf or mpq numbers (that seems like a lot of work for this
  only use, tho I guess we could then use those numbers in Calc as
  well).
- use a handmade arbitrary-precision float made up of a bignum plus
  a "scaling factor" (halfway solution: less work, but less gain).
WDYT?


        Stefan

Re: current-time and GMP

[Paul Eggert]

Stefan Monnier wrote:

> - keep it as is (that's the easy way out)

It really should be fixed. Suggestion below.

> - use GMP's mpf or mpq numbers (that seems like a lot of work for this
>    only use, tho I guess we could then use those numbers in Calc as
>    well).

Since the underlying nominal precision is typically nanoseconds, mpf is 
unsatisfactory since it's inexact, and mpq's canonicalization would lose useful 
information about timestamp resolution. Also, as you mention, both approaches 
are overkill for timestamps.

> - use a handmade arbitrary-precision float made up of a bignum plus
>    a "scaling factor" (halfway solution: less work, but less gain).

Here's a variant of that idea: add a function that yields the current time as an 
integer count, and a constant that yields the system timestamp resolution. So, 
for example, if (current-time) yields (23430 22558 300289 683000), representing 
2018-08-29 08:23:58.300289683 UTC (i.e., 1535531038.300289683 seconds after the 
Epoch), we add a function (current-clock) yielding 1535531038300289683 and a 
constant clock-resolution yielding 1000000000. (As luck would have it, these are 
both fixnums on 64-bit hosts.)

Furthermore, we extend existing timestamp functions to accept the form (A), 
where A is an integer, to stand for a timestamp with count A using 
clock-resolution ticks per second since the epoch. Thus, for example, the 
following two calls would be equivalent:

(format-time-string "%Y-%m-%d %H:%M:%S.%N" '(1535531038300289683) t)
(format-time-string "%Y-%m-%d %H:%M:%S.%N" '(23430 22558 300289 683000) t)

and would both yield "2018-08-29 08:23:58.300289683". This would be an 
upward-compatible extension to current behavior for these functions.

Finally, let's deprecate the long-obsolete timestamp form (A . B) where A and B 
are integers, so that future Emacs versions can support a new format where B is 
the timestamp resolution. By "deprecate" I mean Emacs issues a runtime warning 
if the obsolete form is used. As the old timestamp form has obsolete for two 
decades and has been documented to be obsolete for over a dozen years, it's OK 
to deprecate it now.

Re: current-time and GMP

[Stefan Monnier]

>> - use GMP's mpf or mpq numbers (that seems like a lot of work for this
>>    only use, tho I guess we could then use those numbers in Calc as
>>    well).
> Since the underlying nominal precision is typically nanoseconds, mpf is
> unsatisfactory since it's inexact,

Indeed, the problem with it is that mpf works with binary rather decimal
exponents so it can't represent exactly the 10^-9 of nanoseconds.

> and mpq's canonicalization would lose useful information about
>  timestamp resolution.

Not sure to what extend knowledge about the resolution is important here.

> Also, as you mention, both approaches are overkill for timestamps.

That's right.

> Furthermore, we extend existing timestamp functions to accept the form (A),
> where A is an integer, to stand for a timestamp with count A using
> clock-resolution ticks per second since the epoch.

I guess in practice it might work fine, but I'm uncomfortable with the
idea of using a representation that depends on `clock-resolution`, so if
you print a timestamp with one Emacs and read it with another they may
disagree (in a big way) about what it means (if one Emacs uses
nanoseconds while the other uses picoseconds resolution).

> Finally, let's deprecate the long-obsolete timestamp form (A . B)
> where A and B are integers,

Sounds good.

> so that future Emacs versions can support a new
> format where B is the timestamp resolution.

Right, I think (INTEGER-TIMESTAMP . RESOLUTION) would be a nice
representation.  We can additionally (and temporarily) signal an
error/warning if INTEGER-TIMESTAMP (and/or RESOLUTION) is "too small"
(indicating a very likely use of the deprecated (HOGH . LOW)
representation).


        Stefan

Re: current-time and GMP

[Helmut Eller]

On Wed, Aug 29 2018, Stefan Monnier wrote:

>> so that future Emacs versions can support a new
>> format where B is the timestamp resolution.
>
> Right, I think (INTEGER-TIMESTAMP . RESOLUTION) would be a nice
> representation.  We can additionally (and temporarily) signal an
> error/warning if INTEGER-TIMESTAMP (and/or RESOLUTION) is "too small"
> (indicating a very likely use of the deprecated (HOGH . LOW)
> representation).

I think it would better to finally use an opaque type.  Say a struct
that prints like #<timestamp 2018-08-29 23:05:25.325602+02> along with a
constructor like (ticks-to-timestamp TICKS TICKS-PER-SECOND TIMEZONE).

Helmut

Re: current-time and GMP

[Paul Eggert]

On 8/29/18 12:16 PM, Stefan Monnier wrote:
> Right, I think (INTEGER-TIMESTAMP . RESOLUTION) would be a nice
> representation.  We can additionally (and temporarily) signal an
> error/warning if INTEGER-TIMESTAMP (and/or RESOLUTION) is "too small"
> (indicating a very likely use of the deprecated (HOGH . LOW)
> representation).

OK, I submitted a proposed patch to that effect as Bug#32902. I tested 
it on 32- and 64-bit platforms, as well as in a "future" mode where 
current-time etc. return the new (INTEGER-TIMESTAMP . RESOLUTION) 
format. It works well for me, though of course there may be glitches I 
missed. Comments welcome.

Re: current-time and GMP

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 8/29/18 12:16 PM, Stefan Monnier wrote:
>> Right, I think (INTEGER-TIMESTAMP . RESOLUTION) would be a nice
>> representation.  We can additionally (and temporarily) signal an
>> error/warning if INTEGER-TIMESTAMP (and/or RESOLUTION) is "too small"
>> (indicating a very likely use of the deprecated (HOGH . LOW)
>> representation).

> OK, I submitted a proposed patch to that effect as Bug#32902. I tested it on 32- and 64-bit platforms, as well as in a "future" mode where current-time etc. return the new (INTEGER-TIMESTAMP . RESOLUTION)
> format. It works well for me, though of course there may be glitches I missed. Comments welcome.

I began seeing entries like "obsolete timestamp with cdr 53421" in *Messages* soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.  Everything seems fine otherwise.  There are a few similar reports, e.g., https://github.com/Malabarba/paradox/issues/154 although unlike me, they say they do not see the messages with `emacs -Q`.

Build details
-------------
GNU Emacs 27.0.50 (build 1, amd64-portbld-freebsd11.2, GTK+ Version 3.22.30)
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: 11.2-RELEASE-p4

Commit: d2a07b9

Configured using:
 'configure --disable-build-details --localstatedir=/var
 --with-gameuser=games:games --without-libsystemd --without-mini-gmp
 --with-wide-int=no --with-x --enable-acl --without-cairo --without-dbus
 --without-gconf --with-gif --with-gnutls --without-gsettings
 --with-x-toolkit=gtk3 --with-jpeg --with-json
 --with-file-notification=kqueue --with-lcms2 --with-m17n-flt
 --with-imagemagick --with-mailutils --with-modules --with-sound=oss
 --with-libotf --with-png --with-toolkit-scroll-bars --with-rsvg
 --with-threads --with-tiff --with-xft --with-xim --with-xml2 --with-xpm
 --without-xwidgets --x-libraries=/usr/local/lib
 --x-includes=/usr/local/include --prefix=/usr/local
 --mandir=/usr/local/man --disable-silent-rules
 --infodir=/usr/local/share/emacs/info/
 --build=amd64-portbld-freebsd11.2 'CFLAGS=-O2 -pipe -fstack-protector
 -isystem /usr/local/include -fno-strict-aliasing ' 'CPPFLAGS=-isystem
 /usr/local/include' 'LDFLAGS= -fstack-protector -L/usr/local/lib ''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GLIB NOTIFY ACL GNUTLS
LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11
XDBE XIM MODULES THREADS JSON LCMS2 GMP

Important settings:
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8-unix

--
Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: current-time and GMP

[Kaushal Modi]

On Mon, Oct 22, 2018 at 11:08 AM Joseph Mingrone <jrm@ftfl.ca> wrote:
>
> I began seeing entries like "obsolete timestamp with cdr 53421" in *Messages* soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.  Everything seems fine otherwise.  There are a few similar reports, e.g., https://github.com/Malabarba/paradox/issues/154 although unlike me, they say they do not see the messages with `emacs -Q`.

I confirm. Looks like the spinner[0] package (code[1]) used by paradox
is causing this:

=====
(wrong-type-argument number-or-marker-p 1.5402211759930708e+21)
  timer-next-integral-multiple-of-time((23501 59639 893070 824000) 0.1)
  spinner--start-timer(#s(spinner :frames ["  " "▌ " "█ " "▐▌" " █" "
▐"] :counter 0 :fps 10 :timer [t nil nil nil nil nil nil nil nil]
:active-p t :buffer #<buffer *Packages*> :delay 0))
  spinner-start(#s(spinner :frames ["  " "▌ " "█ " "▐▌" " █" " ▐"]
:counter 0 :fps 10 :timer [t nil nil nil nil nil nil nil nil]
:active-p t :buffer #<buffer *Packages*> :delay 0))
  paradox--start-spinner()
  paradox--menu-execute-1(nil)
  paradox-menu-execute(nil)
  funcall-interactively(paradox-menu-execute nil)
  call-interactively(paradox-menu-execute nil nil)
  command-execute(paradox-menu-execute)
=====

[0]: https://elpa.gnu.org/packages/spinner.html
[1]: https://github.com/Malabarba/spinner.el/blob/master/spinner.el

Re: current-time and GMP

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Mon, 22 Oct 2018 11:15:17 -0400
> Cc: Paul Eggert <eggert@cs.ucla.edu>, Stefan Monnier <monnier@iro.umontreal.ca>,
> 	Emacs developers <emacs-devel@gnu.org>
> 
> On Mon, Oct 22, 2018 at 11:08 AM Joseph Mingrone <jrm@ftfl.ca> wrote:
> >
> > I began seeing entries like "obsolete timestamp with cdr 53421" in *Messages* soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.  Everything seems fine otherwise.  There are a few similar reports, e.g., https://github.com/Malabarba/paradox/issues/154 although unlike me, they say they do not see the messages with `emacs -Q`.
> 
> I confirm. Looks like the spinner[0] package (code[1]) used by paradox
> is causing this:
> 
> =====
> (wrong-type-argument number-or-marker-p 1.5402211759930708e+21)
>   timer-next-integral-multiple-of-time((23501 59639 893070 824000) 0.1)

Please try again, I think I fixed this 2 days ago.

Re: current-time and GMP

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Date: Mon, 22 Oct 2018 12:07:31 -0300
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> I began seeing entries like "obsolete timestamp with cdr 53421" in *Messages* soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.  Everything seems fine otherwise.  There are a few similar reports, e.g., https://github.com/Malabarba/paradox/issues/154 although unlike me, they say they do not see the messages with `emacs -Q`.
> 
> Build details
> -------------
> GNU Emacs 27.0.50 (build 1, amd64-portbld-freebsd11.2, GTK+ Version 3.22.30)
> Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
> System Description: 11.2-RELEASE-p4
> 
> Commit: d2a07b9

Please try with the latest master branch.

Re: current-time and GMP

[Kaushal Modi]

On Mon, Oct 22, 2018 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> Please try again, I think I fixed this 2 days ago.
> Please try with the latest master branch.

Hah! I am on:

Emacs version: GNU Emacs 27.0.50 (build 28, x86_64-pc-linux-gnu, GTK+
Version 2.24.23)
 of 2018-10-19, built using commit fc3f93705543408b868feb7b93b8d77ab1c6ae53.

I see that this got fixed in
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=efb214622a0f4e077c09e721d134552dfe76ef70.

Rebuilding now ..

Re: current-time and GMP

[Kaushal Modi]

On Mon, Oct 22, 2018 at 11:57 AM Kaushal Modi <kaushal.modi@gmail.com> wrote:
>
> On Mon, Oct 22, 2018 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > Please try again, I think I fixed this 2 days ago.
> > Please try with the latest master branch.
>
> Hah! I am on:
>
> Emacs version: GNU Emacs 27.0.50 (build 28, x86_64-pc-linux-gnu, GTK+
> Version 2.24.23)
>  of 2018-10-19, built using commit fc3f93705543408b868feb7b93b8d77ab1c6ae53.
>
> I see that this got fixed in
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=efb214622a0f4e077c09e721d134552dfe76ef70.
>
> Rebuilding now ..

@Eli: I confirm the fix. Many thanks!

Re: current-time and GMP

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> On Mon, Oct 22, 2018 at 11:57 AM Kaushal Modi <kaushal.modi@gmail.com> wrote:

>> On Mon, Oct 22, 2018 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:

>> > Please try again, I think I fixed this 2 days ago.
>> > Please try with the latest master branch.

>> Hah! I am on:

>> Emacs version: GNU Emacs 27.0.50 (build 28, x86_64-pc-linux-gnu, GTK+
>> Version 2.24.23)
>>  of 2018-10-19, built using commit fc3f93705543408b868feb7b93b8d77ab1c6ae53.

>> I see that this got fixed in
>> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=efb214622a0f4e077c09e721d134552dfe76ef70.

>> Rebuilding now ..

> @Eli: I confirm the fix. Many thanks!

After building 969b561, I still see the message with `list-packages' under `emacs -Q'.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: current-time and GMP

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Paul Eggert <eggert@cs.ucla.edu>,  Stefan Monnier <monnier@iro.umontreal.ca>,  Emacs developers <emacs-devel@gnu.org>
> Date: Mon, 22 Oct 2018 13:24:04 -0300
> 
> >> > Please try again, I think I fixed this 2 days ago.
> >> > Please try with the latest master branch.
> 
> >> Hah! I am on:
> 
> >> Emacs version: GNU Emacs 27.0.50 (build 28, x86_64-pc-linux-gnu, GTK+
> >> Version 2.24.23)
> >>  of 2018-10-19, built using commit fc3f93705543408b868feb7b93b8d77ab1c6ae53.
> 
> >> I see that this got fixed in
> >> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=efb214622a0f4e077c09e721d134552dfe76ef70.
> 
> >> Rebuilding now ..
> 
> > @Eli: I confirm the fix. Many thanks!
> 
> After building 969b561, I still see the message with `list-packages' under `emacs -Q'.

Then yours is a different problem, and I'm afraid I cannot reproduce
it here with the latest master.  Could it be that you have customized
your packages or the repositories from which you produce the list of
packages?  Or did something else that modifies the behavior of "emacs -Q"?

Re: current-time and GMP

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Mingrone <jrm@ftfl.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Paul Eggert <eggert@cs.ucla.edu>,  Stefan Monnier <monnier@iro.umontreal.ca>,  Emacs developers <emacs-devel@gnu.org>
>> Date: Mon, 22 Oct 2018 13:24:04 -0300

>> >> > Please try again, I think I fixed this 2 days ago.
>> >> > Please try with the latest master branch.

>> >> Hah! I am on:

>> >> Emacs version: GNU Emacs 27.0.50 (build 28, x86_64-pc-linux-gnu, GTK+
>> >> Version 2.24.23)
>> >>  of 2018-10-19, built using commit fc3f93705543408b868feb7b93b8d77ab1c6ae53.

>> >> I see that this got fixed in
>> >> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=efb214622a0f4e077c09e721d134552dfe76ef70.

>> >> Rebuilding now ..

>> > @Eli: I confirm the fix. Many thanks!

>> After building 969b561, I still see the message with `list-packages' under `emacs -Q'.

> Then yours is a different problem, and I'm afraid I cannot reproduce
> it here with the latest master.  Could it be that you have customized
> your packages or the repositories from which you produce the list of
> packages?  Or did something else that modifies the behavior of "emacs -Q"?

I don't think so.

--
Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: current-time and GMP

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

On 10/22/18 8:07 AM, Joseph Mingrone wrote:
> I began seeing entries like "obsolete timestamp with cdr 53421" in*Messages*  soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.

Thanks, I reproduced the bug with that recipe and installed the 
attached, which fixed things for me.


[-- Attachment #2: 0001-Fix-epg-bug-with-TICKS-.-HZ-timestamp.txt --]
[-- Type: text/plain, Size: 1193 bytes --]

From 5bb8d5aa20febabbfb5fa20ea124eed632d447e4 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 22 Oct 2018 10:54:45 -0700
Subject: [PATCH] Fix epg bug with (TICKS . HZ) timestamp
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Joseph Mingrone in:
https://lists.gnu.org/r/emacs-devel/2018-10/msg00380.html
* lisp/epg.el (epg--time-from-seconds): Just use a seconds count;
don’t generate an obsolete-format timestamp.
---
 lisp/epg.el | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/lisp/epg.el b/lisp/epg.el
index 8f26cd34ee..9d9bc9051d 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -946,10 +946,7 @@ epg--status-NO_SECKEY
    (cons (cons 'no-seckey string)
 	 (epg-context-result-for context 'error))))
 
-(defun epg--time-from-seconds (seconds)
-  (let ((number-seconds (string-to-number (concat seconds ".0"))))
-    (cons (floor (/ number-seconds 65536))
-	  (floor (mod number-seconds 65536)))))
+(defalias 'epg--time-from-seconds #'string-to-number)
 
 (defun epg--status-ERRSIG (context string)
   (if (string-match "\\`\\([^ ]+\\) \\([0-9]+\\) \\([0-9]+\\) \
-- 
2.17.2

Re: current-time and GMP

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/22/18 8:07 AM, Joseph Mingrone wrote:
>> I began seeing entries like "obsolete timestamp with cdr 53421" in*Messages*  soon after this was committed.  It can be reproduced with `emacs -Q` and `list-packages'.

> Thanks, I reproduced the bug with that recipe and installed the attached, which fixed things for me.

That fixed did the trick here as well.  Thanks.

--
Joseph

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: current-time and GMP

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> - keep it as is

Just to play devil's advocate: Does it need to change? Is there any Emacs Lisp
code you think of that whose utility could benefit from greater precision than
what we have now?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: current-time and GMP

[Stefan Monnier]

> SM> - keep it as is
> Just to play devil's advocate: Does it need to change?

It's definitely not high priority.

> Is there any Emacs Lisp code you think of that whose utility could
> benefit from greater precision than what we have now?

The time-manipulation code is just pretty ugly because of the need to
manipulate those 3-4 element list, basically performing bignum-style
operations by hand (see (F)time_add, (F)time_substract, .... in
editfns.c).

Admittedly, it'll probably take a while before we move to something else
than picoseconds as to basis, so I don't think it's a serious
maintenance hassle.  It's just that the code could be more elegant.


        Stefan

Re: current-time and GMP

[T.V Raman]

In addition to the issues you raise, the time- related functions are
hard to discover/remember because of how they're named etc --- one
possibility would be to leave them as is and write a new library
alongside it that handled it better.  
-- 

Re: current-time and GMP

[John Wiegley]

>>>>> "TVR" == T V Raman <raman@google.com> writes:

TVR> In addition to the issues you raise, the time- related functions are hard
TVR> to discover/remember because of how they're named etc --- one possibility
TVR> would be to leave them as is and write a new library alongside it that
TVR> handled it better.

I would have no problem supporting this.  time-ng.el!

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: current-time and GMP

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

SM> The time-manipulation code is just pretty ugly because of the need to
SM> manipulate those 3-4 element list, basically performing bignum-style
SM> operations by hand (see (F)time_add, (F)time_substract, .... in
SM> editfns.c).

True, but all of this is behind an interface that people will probably keep
using for a decade at least, to stay compatible with older Emacsen. As long as
it's hidden behind an API, I'm not as concerned if the (basically existential)
type of time values is ugly.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2