Making decoded-times and calendar dates compatible?

Making decoded-times and calendar dates compatible?

[Richard Lawrence]

Hi everyone,

I've been drafting a sub-library to handle iCalendar recurrence rules
over the last week, and have run into an issue that I would like some
feedback here on: the representations used by make-decoded-time and
related functions are not compatible with those of calendar.el.
I'd like to know if there's any appetite for making these
representations compatible in Emacs, so that functions like e.g.
calendar-day-of-week, calendar-nth-named-day, calendar-date-equal, and
so on could also work with decoded times. 

The reason this would be nice: handling recurrence rules requires doing
a lot of calendar arithmetic. Decoded times have one important
arithmetic function (decoded-time-add) but many more useful arithmetic
functions, like the ones named above, are part of calendar.el. Most of
the iCalendar properties that accept date-time values also accept plain
calendar dates, so I've had to write a lot of boilerplate/wrapper code
to convert between the two types when necessary, and to do type-based
dispatch on values which can be of either type.

Both calendar.el and decoded-time functions represent date(-time) values
as lists. But the calendar assumes dates are in the format (MONTH DAY YEAR), 
whereas decoded times have the format (SEC MIN HOUR DAY MON YEAR DOW DST TZ);
the month, day and year are at different indices in these lists.
If we changed them to use a format like, say,
  (YEAR MONTH DAY)
and 
  (YEAR MONTH DAY DOW HOUR MINUTE SECOND DST TZ)
respectively, changing the relevant accessors, then calendar arithmetic
functions could also work effortlessly with [the date part of] decoded
times, which would really simplify working with both types of values
from other code (including mine, but I assume also in Org, Gnus, diary,
third-party packages, ...).

I see that there was a discussion a few years ago about changing the
representation of decoded-time values, though, and it seems that there
wasn't much appetite even for that because of the possibility of
breaking user code which makes assumptions about the underlying
representation; discussion starts here:

https://lists.gnu.org/archive/html/emacs-devel/2019-08/msg00129.html

So I expect there won't be much appetite for this proposal either, but I
thought I would ask, because (IMHO) the benefits of the change might be
worth it.

Best,
Richard

Re: Making decoded-times and calendar dates compatible?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > If we changed them to use a format like, say,
  >   (YEAR MONTH DAY)
  > and 
  >   (YEAR MONTH DAY DOW HOUR MINUTE SECOND DST TZ)
  > respectively, changing the relevant accessors, then calendar arithmetic
  > functions could also work effortlessly with [the date part of] decoded
  > times,

In principle it sounds like a good idea, but I think that the
incompatibility might be a big pain to fix.  Doesn't some user code
have to operate on those formats?

I wonder also if calendar.el was designed to be compatible with
something in Unix that existed before GNU Emacs.  But I wasn't the
one who wrote it, so I wouldn't know.

One possible way to make the incompatible change less of a pain to
cope with would be to use a list like

  (calendar YEAR MONTH DAY)

in caledar.el.  The presence of the synbol `calendar' would say "this
date uses the new format", thus avoiding ambiguity of the datum.
There would still need to be a lot of change, but at least it would be
easier to be sure you found all the places that had to be changed.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Making decoded-times and calendar dates compatible?

[Richard Lawrence]

Hi Richard and all,

Richard Stallman <rms@gnu.org> writes:

> I tried reading the code of calendar.el to see what data structures it
> uses, but I could not find documentation for that.  (The file is
> enormous.)
>
> Can you tell me where to find that documentation?  Perhaps
> wuotq a few lines so I can search for them.

The only documentation I'm aware of (besides the Calendar/Diary section
in the Emacs manual, which doesn't deal with the programming interface)
is in the docstrings of the library. The docstrings of the functions

calendar-extract-year
calendar-extract-month
calendar-extract-day

are where the (MONTH DAY YEAR) format is mentioned. I *believe* these
functions are used pretty consistently as accessors in the calendar
code, so in theory changing the representation would only require
changing these functions, but honestly I have no idea at this point how
much of a pain it would be.

This is the "standard", Gregorian format for calendar dates, but it is
not the only format that the calendar works with. Internally it also
uses an "absolute" format which is an integer number of days since
December 31, 1BC (see e.g. `calendar-absolute-from-gregorian'), and
converts between different calendar scales by converting to and from the
absolute format (see e.g. `calendar-iso-from-absolute').

Similarly, the only documentation I'm aware of for the decoded-time
format is in the docstrings of the functions `decode-time' and
`parse-time-string', and in the symbol documentation for `decoded-time',
which is declared as a cl-struct with :type list in simple.el. According
to a comment there, the point of this declaration is to provide
accessors for the various slots in the list, e.g. `decoded-time-month'.

>   > If we changed them to use a format like, say,
>   >   (YEAR MONTH DAY)
>   > and 
>   >   (YEAR MONTH DAY DOW HOUR MINUTE SECOND DST TZ)
>   > respectively, changing the relevant accessors, then calendar arithmetic
>   > functions could also work effortlessly with [the date part of] decoded
>   > times,
>
> In principle it sounds like a good idea, but I think that the
> incompatibility might be a big pain to fix.  Doesn't some user code
> have to operate on those formats?

It could very well be a big pain to fix. I'm not sure how much user code
has to work with these formats. I know I've written some code to work
with them once or twice.

Personally, I think it's reasonable to change the underlying format as
long as the documented accessors (calendar-extract-*, decoded-time-*)
continue to work as expected. User code which uses these functions will
then continue to work. User code which instead makes assumptions about
the underlying format is not respecting an abstraction boundary, and
it's reasonable to break this code. But I realize that my personal view
may be more liberal than the general view of the Emacs community on this
point; that's fine (and I'm surely grateful for it in other cases!).

> I wonder also if calendar.el was designed to be compatible with
> something in Unix that existed before GNU Emacs.  But I wasn't the
> one who wrote it, so I wouldn't know.

I don't know either.

> One possible way to make the incompatible change less of a pain to
> cope with would be to use a list like
>
>   (calendar YEAR MONTH DAY)
>
> in caledar.el.  The presence of the synbol `calendar' would say "this
> date uses the new format", thus avoiding ambiguity of the datum.
> There would still need to be a lot of change, but at least it would be
> easier to be sure you found all the places that had to be changed.

If we were going to do that, it might make sense to just declare
calendar dates as cl-structs, since this provides the tag automatically,
and comes along with some other benefits.

Best,
Richard

Re: Making decoded-times and calendar dates compatible?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > The only documentation I'm aware of (besides the Calendar/Diary section
  > in the Emacs manual, which doesn't deal with the programming interface)
  > is in the docstrings of the library. The docstrings of the functions

  > calendar-extract-year
  > calendar-extract-month
  > calendar-extract-day

  > are where the (MONTH DAY YEAR) format is mentioned.

Would someoe like to add some comments at the start of the file
documenging this data structure?

  > This is the "standard", Gregorian format for calendar dates, but it is
  > not the only format that the calendar works with. Internally it also
  > uses an "absolute" format which is an integer number of days since
  > December 31, 1BC (see e.g. `calendar-absolute-from-gregorian'), and
  > converts between different calendar scales by converting to and from the
  > absolute format (see e.g. `calendar-iso-from-absolute').

It would be helpful to describe that representation too, in those comments.

  > Similarly, the only documentation I'm aware of for the decoded-time
  > format is in the docstrings of the functions `decode-time' and
  > `parse-time-string', and in the symbol documentation for `decoded-time',
  > which is declared as a cl-struct with :type list in simple.el. According
  > to a comment there, the point of this declaration is to provide
  > accessors for the various slots in the list, e.g. `decoded-time-month'.

This IS described in the Emacs Lisp Reference Manual, in section Time of Day.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

[PATCH] Re: Making decoded-times and calendar dates compatible?

[Richard Lawrence]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Richard Stallman <rms@gnu.org> writes:
>   The docstrings of the functions
>
>   > calendar-extract-year
>   > calendar-extract-month
>   > calendar-extract-day
>
>   > are where the (MONTH DAY YEAR) format is mentioned.
>
> Would someoe like to add some comments at the start of the file
> documenging this data structure?

Attached is a patch which adds such a comment, based on my previous
message; let me know if I should open a bug report for this.

For now I take it that it's better not to touch the calendar code? I
don't think there is any constructor function for dates, or at least, I
see that throughout the calendar code, there are many uses of "(list
month day year)" to construct dates, so there might be a lot of code to
fix if we change the representation, and it might be hard to be sure
we've made all the necessary changes. Anyway, this is more work than I
want to do at the moment, though I'm willing to take a stab at it
eventually if others here think it would be worthwhile.

Best,
Richard


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Document-representation-of-dates-in-calendar.el.patch --]
[-- Type: text/patch, Size: 1350 bytes --]

From 4bf17cf59f7f8cc29ca79b03649037416aab1901 Mon Sep 17 00:00:00 2001
From: Richard Lawrence <rwl@recursewithless.net>
Date: Mon, 16 Dec 2024 20:46:34 +0100
Subject: [PATCH] Document representation of dates in calendar.el

* lisp/calendar/calendar.el: Add a comment in file header
explaining how dates are represented.
---
 lisp/calendar/calendar.el | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lisp/calendar/calendar.el b/lisp/calendar/calendar.el
index 60d8fdd6aee..36d64a2b11c 100644
--- a/lisp/calendar/calendar.el
+++ b/lisp/calendar/calendar.el
@@ -90,6 +90,16 @@
 ;; <https://doi.org/10.1002/spe.4380230404>
 ;; <http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.42.6421&rep=rep1&type=pdf>
 
+;; A note on how dates are represented:
+
+;; The standard format for a (Gregorian) calendar date in this file is a
+;; list of integers (MONTH DAY YEAR) -- see the functions
+;; `calendar-extract-year', `calendar-extract-month', and
+;; `calendar-extract-day'.  Internally it also uses an "absolute" format
+;; which is an integer number of days since December 31, 1BC (see
+;; e.g. `calendar-absolute-from-gregorian'), and converts between
+;; different calendar scales by converting to and from the absolute
+;; format (see e.g. `calendar-iso-from-absolute' in cal-iso.el).
 
 ;; A note on free variables:
 
-- 
2.39.5

Re: [PATCH] Re: Making decoded-times and calendar dates compatible?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It says what needs to be said.  A little more whitespace
woudl make it easier to read.

I encourage everyone to add comments to help people understand various
modules after you have figured them out.  For instance, sometimes it is
helpful to expain the vaious sections of a file and what jobs they do.
Sometimes it is helpful to describe the main entry points
in the comments at the top.  Sometimes it is useful to explaib
some of the file's data structures and what they mean.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [PATCH] Re: Making decoded-times and calendar dates compatible?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I
  > see that throughout the calendar code, there are many uses of "(list
  > month day year)" to construct dates, so there might be a lot of code to
  > fix if we change the representation,

It might be helpful to replace `list' there with a new function
`calendar-make-date' just for clarity.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [PATCH] Re: Making decoded-times and calendar dates compatible?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 16 Dec 2024 23:39:56 -0500
> 
> I encourage everyone to add comments to help people understand various
> modules after you have figured them out.  For instance, sometimes it is
> helpful to expain the vaious sections of a file and what jobs they do.
> Sometimes it is helpful to describe the main entry points
> in the comments at the top.  Sometimes it is useful to explaib
> some of the file's data structures and what they mean.

110% agreement.  Changes to add such commentary or improve the
existing one will always be welcome.

Re: [PATCH] Re: Making decoded-times and calendar dates compatible?

[Richard Lawrence]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 16 Dec 2024 23:39:56 -0500
>> 
>> I encourage everyone to add comments to help people understand various
>> modules after you have figured them out.  For instance, sometimes it is
>> helpful to expain the vaious sections of a file and what jobs they do.
>> Sometimes it is helpful to describe the main entry points
>> in the comments at the top.  Sometimes it is useful to explaib
>> some of the file's data structures and what they mean.
>
> 110% agreement.  Changes to add such commentary or improve the
> existing one will always be welcome.

OK I've sent a revised patch with such a comment to Bug#74965.

Best,
Richard

Re: [PATCH] Re: Making decoded-times and calendar dates compatible?

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Cc: emacs-devel@gnu.org
> Date: Mon, 16 Dec 2024 21:05:30 +0100
> 
> Richard Stallman <rms@gnu.org> writes:
> >   The docstrings of the functions
> >
> >   > calendar-extract-year
> >   > calendar-extract-month
> >   > calendar-extract-day
> >
> >   > are where the (MONTH DAY YEAR) format is mentioned.
> >
> > Would someoe like to add some comments at the start of the file
> > documenging this data structure?
> 
> Attached is a patch which adds such a comment, based on my previous
> message; let me know if I should open a bug report for this.

In general, yes.  But we can make exceptions.

I've now installed your change on the emacs-30 release branch.

> For now I take it that it's better not to touch the calendar code? I
> don't think there is any constructor function for dates, or at least, I
> see that throughout the calendar code, there are many uses of "(list
> month day year)" to construct dates, so there might be a lot of code to
> fix if we change the representation, and it might be hard to be sure
> we've made all the necessary changes. Anyway, this is more work than I
> want to do at the moment, though I'm willing to take a stab at it
> eventually if others here think it would be worthwhile.

I think you are right.