Represent NTP's origin time

Represent NTP's origin time

[Stefan Monnier]

NTP represents time stamps by counting seconds from January 1 1900, but
my Emacs doesn't seem to be able to represent this:

    (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
    => (-32745 59520)
    (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
    => (error "Specified time is not representable")

Do we have some "standard" workaround?


        Stefan

Re: Represent NTP's origin time

[Andreas Schwab]

On Apr 15 2021, Stefan Monnier wrote:

> NTP represents time stamps by counting seconds from January 1 1900, but
> my Emacs doesn't seem to be able to represent this:
>
>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>     => (-32745 59520)
>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>     => (error "Specified time is not representable")
>
> Do we have some "standard" workaround?

Worksforme.

ELISP> (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
(-33707 33152)

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Represent NTP's origin time

[Gregory Heytings]

>> NTP represents time stamps by counting seconds from January 1 1900, but 
>> my Emacs doesn't seem to be able to represent this:
>>
>>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>>     => (-32745 59520)
>>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>>     => (error "Specified time is not representable")
>>
>> Do we have some "standard" workaround?
>
> Worksforme.
>
> ELISP> (encode-time (parse-time-string "1900-01-01T00:00-00:00")) 
> (-33707 33152)
>

Works for me, too:

(encode-time (parse-time-string "9999-01-01T00:00-00:00"))
(3866131 3584)

(encode-time (parse-time-string "0000-01-01T00:00-00:00"))
(-948597 33792)

(encode-time (parse-time-string "-9999-01-01T00:00-00:00"))
(-5763323 19328)

Re: Represent NTP's origin time

[Ulrich Mueller]

>>>>> On Thu, 15 Apr 2021, Stefan Monnier wrote:

> NTP represents time stamps by counting seconds from January 1 1900, but
> my Emacs doesn't seem to be able to represent this:

>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>     => (-32745 59520)
>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>     => (error "Specified time is not representable")

> Do we have some "standard" workaround?

Overflow in the underlying mktime(3) or mktime_z(3)? Are there still
systems around where time_t has only 32 bits?

Re: Represent NTP's origin time

[Eli Zaretskii]

> From: Ulrich Mueller <ulm@gentoo.org>
> Date: Thu, 15 Apr 2021 10:27:27 +0200
> Cc: emacs-devel@gnu.org
> 
> Are there still systems around where time_t has only 32 bits?

Yes, of course.

Re: Represent NTP's origin time

[Stefan Monnier]

>> NTP represents time stamps by counting seconds from January 1 1900, but
>> my Emacs doesn't seem to be able to represent this:
>
>>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>>     => (-32745 59520)
>>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>>     => (error "Specified time is not representable")
>
>> Do we have some "standard" workaround?
>
> Overflow in the underlying mktime(3) or mktime_z(3)? Are there still
> systems around where time_t has only 32 bits?

AFAIK all GNU/Linux i386 installations do.  Nowadays amd64 is more
common, but most of my machines are still running an i386 image and I'm
pretty sure I'm not the only one.

Also all armhf GNU/Linux systems do, and machines for which this is the
main/sole option are still being produced.

So, getting back to the question: Do we have some "standard" workaround?
I don't actually need to represent times from the early 20th century,
but I need to compute NTP timestamps.  I can come up with my own
workaround (e.g. hardcoding as a "magic constant" the number of seconds
between NTP's origin time and some arbitrary other origin time of my
choosing which 32bit systems can represent), but I was hoping we already
have something like that somewhere.


        Stefan

Re: Represent NTP's origin time

[Andreas Schwab]

On Apr 15 2021, Stefan Monnier wrote:

> AFAIK all GNU/Linux i386 installations do.

They will either get 64-bit time_t or die out in 17 years.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Represent NTP's origin time

[Stefan Monnier]

Andreas Schwab [2021-04-15 14:56:09] wrote:
> On Apr 15 2021, Stefan Monnier wrote:
>> AFAIK all GNU/Linux i386 installations do.
> They will either get 64-bit time_t or die out in 17 years.

Yes, I'm not worried about the future, but I'm sadly stuck in
the present.


        Stefan

Re: Represent NTP's origin time

[Ulrich Mueller]

>>>>> On Thu, 15 Apr 2021, Stefan Monnier wrote:

>> Overflow in the underlying mktime(3) or mktime_z(3)? Are there still
>> systems around where time_t has only 32 bits?

> AFAIK all GNU/Linux i386 installations do.

It should be 64 bit time_t since Linux 5.6:
https://linux.slashdot.org/story/20/02/15/0247201/linux-is-ready-for-the-end-of-time

Re: Represent NTP's origin time

[Stefan Monnier]

> It should be 64 bit time_t since Linux 5.6:
> https://linux.slashdot.org/story/20/02/15/0247201/linux-is-ready-for-the-end-of-time

FWIW, the original report was sent from a machine running a 5.10 kernel.
I strongly suspect that the libc part is crucial as well.

In any case, my question is not about which systems support it and how
common/rare they are or will be but just how to deal with the situation.


        Stefan

Re: Represent NTP's origin time

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]


>
> So, getting back to the question: Do we have some "standard" workaround? 
> I don't actually need to represent times from the early 20th century, 
> but I need to compute NTP timestamps.  I can come up with my own 
> workaround (e.g. hardcoding as a "magic constant" the number of seconds 
> between NTP's origin time and some arbitrary other origin time of my 
> choosing which 32bit systems can represent), but I was hoping we already 
> have something like that somewhere.
>

Apparently there is no standard workaround in Emacs.  So on a 32bit 
system, I would use the constant 2208988800 (25567 days times 86400 
seconds), to represent the difference in seconds between 1900-01-01 and 
1970-01-01 (epoch).  This constant is given in the very first NTP RFC [1], 
and doing this should be safe: the current NTP format (32 bits for the 
integer part) will work until Feb 7, 2036, that is, before 32-bit counting 
after epoch will stop working on Jan 19, 2038.

[1] https://tools.ietf.org/html/rfc868

Re: Represent NTP's origin time

[Stefan Monnier]

> Apparently there is no standard workaround in Emacs.  So on a 32bit system,
> I would use the constant 2208988800 (25567 days times 86400 seconds), to
> represent the difference in seconds between 1900-01-01 and 1970-01-01
> (epoch).

Thanks, I'll go with that.


        Stefan

Re: Represent NTP's origin time

[Achim Gratz]

Stefan Monnier writes:
> NTP represents time stamps by counting seconds from January 1 1900, but
> my Emacs doesn't seem to be able to represent this:
>
>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>     => (-32745 59520)
>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>     => (error "Specified time is not representable")
>
> Do we have some "standard" workaround?

Get off your 32bit system already?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Represent NTP's origin time

[tomas]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On Thu, Apr 15, 2021 at 09:12:50PM +0200, Achim Gratz wrote:
> Stefan Monnier writes:
> > NTP represents time stamps by counting seconds from January 1 1900, but
> > my Emacs doesn't seem to be able to represent this:
> >
> >     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
> >     => (-32745 59520)
> >     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
> >     => (error "Specified time is not representable")
> >
> > Do we have some "standard" workaround?
> 
> Get off your 32bit system already?

The maintainer's perspective isn't like that. Fortunately.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Represent NTP's origin time

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Get off your 32bit system already?

Let's be kind to our users, even when we can't do what they ask for.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Represent NTP's origin time

[tomas]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Fri, Apr 16, 2021 at 11:53:07PM -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>   > Get off your 32bit system already?
> 
> Let's be kind to our users, even when we can't do what they ask for.

Now thanks for that lesson. While I was snarky to the snarky,
you showed that even that is unnecessary :-)

Cheers
 - tomas

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Represent NTP's origin time

[Achim Gratz]

Achim Gratz writes:
> Stefan Monnier writes:
>> NTP represents time stamps by counting seconds from January 1 1900, but
>> my Emacs doesn't seem to be able to represent this:
>>
>>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
>>     => (-32745 59520)
>>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
>>     => (error "Specified time is not representable")
>>
>> Do we have some "standard" workaround?
>
> Get off your 32bit system already?

There have been responses that called this answer "unkind" or "snarky".
It wasn't meant to be, I apologise for the impression that was likely
created by being too brief.  I stand by what I said, though: The
standard workaround is to do whatever you were trying to do on a system
with a 64bit time_t instead.

Indeed that very problem is what drove me off my own 32bit installation
a few years ago and other than the time spent to figure out how to
convert the system without a full re-installation (which probably would
have taken less time, all things considered) I have no regrets about
doing that.

One of the workarounds (or really limp-alongs) that I've used before
that was using the difference of 2,208,988,800 seconds between UNIX and
NTP epoch (as mentioned in another answer already) that is documented in
many places, for instance here:

https://www.eecis.udel.edu/~mills/y2k.html

That works for the majority of cases (in particular all "real" NTPv3
time stamps ever produced) on all systems that I've worked on.  It
doesn't change the fact that dates before the UNIX epoch can not be
represented with an unsigned 32bit time_t (does Emacs still run on any
systems that do this?) and a signed one peters out at
1901-12-13T20:45:52+00:00 when looking back from the epoch.  Outside the
"NTP prime era" you'd have to keep track of the era/epoch seperately and
that's just the start of your troubles then.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Represent NTP's origin time

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Sat, Apr 17, 2021 at 03:44:28PM +0200, Achim Gratz wrote:
> Achim Gratz writes:
> > Stefan Monnier writes:
> >> NTP represents time stamps by counting seconds from January 1 1900, but
> >> my Emacs doesn't seem to be able to represent this:
> >>
> >>     (encode-time (parse-time-string "1902-01-01T00:00-00:00"))
> >>     => (-32745 59520)
> >>     (encode-time (parse-time-string "1900-01-01T00:00-00:00"))
> >>     => (error "Specified time is not representable")
> >>
> >> Do we have some "standard" workaround?
> >
> > Get off your 32bit system already?
> 
> There have been responses that called this answer "unkind" or "snarky".

Only mildly so, from my side.

> It wasn't meant to be, I apologise for the impression that was likely
> created by being too brief [...]

I do understand the issues, but see below.

I replied (or so I thought) "in kind". A better reply would have
been: Emacs supports 32 bit platforms. Unless ending 32 bit
support is in the near-future, it'd be a good idea to think about
how to support a wider (possibly 64 bit) time_t even on a 32 bit
Emacs.

The *BSDs and Linux (beyond kernel 5.6ish) do support 64 bit time_t
in their 32 bit variants: perhaps this is an orientation.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Represent NTP's origin time

[Stefan Monnier]

> I replied (or so I thought) "in kind". A better reply would have
> been: Emacs supports 32 bit platforms. Unless ending 32 bit
> support is in the near-future, it'd be a good idea to think about
> how to support a wider (possibly 64 bit) time_t even on a 32 bit
> Emacs.

AFAIK this should mostly "just happen" for those 32bit systems that have
a 64bit time_t.

BTW, the funny part of this discussion is that the problem has existed
"for ever".  It just so happens that there is now a bunch of
"easy" OS-level solutions.  The magical 2208988800 constant is the
"standard solution" that I was looking for (I knew it had to exist
because it had not OS-level solution for at least 30 years).


        Stefan

Re: Represent NTP's origin time

[Achim Gratz]

tomas@tuxteam.de writes:
> I replied (or so I thought) "in kind". A better reply would have
> been: Emacs supports 32 bit platforms. Unless ending 32 bit
> support is in the near-future, it'd be a good idea to think about
> how to support a wider (possibly 64 bit) time_t even on a 32 bit
> Emacs.

The support for 64bit time_t on 32bit platforms is quite old by now
actually, but it hasn't seen as much uptake as initially envisioned for
whatever reasons.  So in effect those platforms have announced their EOL
and you should move off them.

> The *BSDs and Linux (beyond kernel 5.6ish) do support 64 bit time_t
> in their 32 bit variants: perhaps this is an orientation.

I have no 32bit system left that can run a modern enough Linux kernel
and doesn't also support 64bit, so once I update those will not be 32bit
anymore.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Represent NTP's origin time

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

On Sat, Apr 17, 2021 at 04:55:14PM +0200, Achim Gratz wrote:
> tomas@tuxteam.de writes:
> > I replied (or so I thought) "in kind". A better reply would have
> > been: Emacs supports 32 bit platforms. Unless ending 32 bit
> > support is in the near-future, it'd be a good idea to think about
> > how to support a wider (possibly 64 bit) time_t even on a 32 bit
> > Emacs.
> 
> The support for 64bit time_t on 32bit platforms is quite old by now
> actually, but it hasn't seen as much uptake as initially envisioned for
> whatever reasons.  So in effect those platforms have announced their EOL
> and you should move off them.
> 
> > The *BSDs and Linux (beyond kernel 5.6ish) do support 64 bit time_t
> > in their 32 bit variants: perhaps this is an orientation.
> 
> I have no 32bit system left that can run a modern enough Linux kernel
> and doesn't also support 64bit, so once I update those will not be 32bit
> anymore.

Again. The fact that *you* haven't doesn't mean that others are
in the same boat. A maintainer's job involves exactly that kind
of deliberation :)

(I'd bet that Stefan's hardware is still 32 bit :)

So basically it reduces to whether the Emacs project wants to
support 32 bit systems (currently, the answer seems to be "yes")
and what to do about time_t when yes.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Represent NTP's origin time

[Achim Gratz]

tomas@tuxteam.de writes:
> Again. The fact that *you* haven't doesn't mean that others are
> in the same boat. A maintainer's job involves exactly that kind
> of deliberation :)

You are tilting at windmills.

> (I'd bet that Stefan's hardware is still 32 bit :)

The hardware is probably not, but it wasn't explicitly mentioned.  The
hardware isn't the issue here anyway.  The implementation of time_t is,
as was clear from the original posting and later confirmed to be the
case.

> So basically it reduces to whether the Emacs project wants to
> support 32 bit systems (currently, the answer seems to be "yes")
> and what to do about time_t when yes.

No, that was not the question Stefan originally asked, but one that you
brought into the discussion.  To be clear, I have not and I will not
offer an opinion on that particular topic in this thread.

The question of Stefan was how to represent the NTP epoch (or "origin
time") and the answer was and still is you can't do that on a system
that has 32bit time_t of any kind.  The question of whether you'd want
to support Emacs on such systems is immaterial to both the question and
the answer.

The workaround that was mentioned multiple times now works for
conversion between NTP to UNIX representation (both ways) without the
need to have an explicit representation of the NTP epoch, with the
disadvantage that it only works where those two representations overlap
(i.e you lose about two years on either end).  But that is the answer to
an extrapolation of the original question which Stefan revealed later as
the actual problem he wanted to solve.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

[OFFTOPIC] Re: Represent NTP's origin time

[Stefan Monnier]

>> The support for 64bit time_t on 32bit platforms is quite old by now
>> actually, but it hasn't seen as much uptake as initially envisioned for
>> whatever reasons.  So in effect those platforms have announced their EOL
>> and you should move off them.

IIUC the issue is that the ripple effects of the change are non-trivial:
lots and lots of APIs include a time_t somewhere (where recompilation
leads to an incompatible ABI).  So changing time_t ends up requiring
recompiling (or, from the users's PoV upgrading) lots and lots of
packages, plus a lot of work to figure out which packages need
recompiling and which don't.  So the alternative of making a completely
new port is not much worse (implies more recompilation, admittedly, but
completely avoids the need to figure out which packages need
recompilation).

[ IIUC for i386 the plan for Debian is indeed to hope that the port will
  die before the 32bit limit becomes really unbearable.  Not sure what
  will happen with the ports for smaller systems where the move from
  32bit to 64bit is much younger (e.g. armhf).  ]

>> > The *BSDs and Linux (beyond kernel 5.6ish) do support 64 bit time_t
>> > in their 32 bit variants: perhaps this is an orientation.
>> I have no 32bit system left that can run a modern enough Linux kernel
>> and doesn't also support 64bit, so once I update those will not be 32bit
>> anymore.

FWIW, I use an amd64 kernel with a i386 Debian install on most of my machines.

> (I'd bet that Stefan's hardware is still 32 bit :)

I do still own several machines with a 32bit-only CPU:
- two armhf SBCs (both using the Allwinner A20 CPU) which I use as
  always-on servers.
- my old Thinkpad X30, which I use as "PDF to VGA adapter" when I teach.
- an old Thinkpad X60 which was supposed to replace the X30 but which I end up
  not using because the X30 still works well enough.
- my DSL modem/router (running a `mips` port of OpenWRT; a rare big-endian machine ;-)
- my smartphone
- my tablet

Admittedly, I don't use Debian on the last 3.

I also own 3 more machines which had a 32bit-only CPU (Core Duo)
originally but in which I retrofitted Core 2 Duos.

I tried using an amd64 Debian install on those machines that are limited
to 3GB of RAM and found that the increased memory pressure was
sufficiently noticeable that I decided to stick to i386.

> So basically it reduces to whether the Emacs project wants to
> support 32 bit systems (currently, the answer seems to be "yes")

Quite.

> and what to do about time_t when yes.

I don't think there's any need for Emacs to make any special effort in
this respect.


        Stefan

Re: [OFFTOPIC] Re: Represent NTP's origin time

[tomas]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Sat, Apr 17, 2021 at 12:54:59PM -0400, Stefan Monnier wrote:

[...]

> - my old Thinkpad X30, which I use as "PDF to VGA adapter" when I teach.

I think this is the tidbit I was remembering :)

[...]

> > and what to do about time_t when yes.
> 
> I don't think there's any need for Emacs to make any special effort in
> this respect.

I see. So it's rather about specific workarounds like this one.

Thanks
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [OFFTOPIC] Re: Represent NTP's origin time

[Stefan Monnier]

>> - my old Thinkpad X30, which I use as "PDF to VGA adapter" when I teach.
> I think this is the tidbit I was remembering :)

This is also the part where Emacs is relevant, since I typically use
`doc-view-mode` to display those PDFs.


        Stefan