Improving Emacs' iCalendar support

Improving Emacs' iCalendar support

[Richard Lawrence]

Dear emacs-devel,

Hello! I've been a happy Emacs user for almost twenty years now(!), but
this is my first time subscribing to this list, so please help me follow
the local conventions.

I would like to start a discussion about improving Emacs' iCalendar
support, beyond what is already available in icalendar.el. I personally
would like to see Emacs gain a more full-fledged RFC5545 implementation
that is primarily designed as a library for other applications to use.

This is an itch that's been bugging me for a while, and so for the past
couple of weeks I've been working on scratching it, and I now have a
reasonable chunk of work to share: I've drafted a new implementation of
the iCalendar grammar, and a major mode which uses this grammar to
provide syntax highlighting. I wrote up what I've done and why here:

https://recursewithless.net/emacs/icalendar-parser-and-mode.org

That's a literate Org mode file containing the code and my commentary.
If you just want to read the code itself, see:

https://recursewithless.net/emacs/icalendar/icalendar-parser.el
https://recursewithless.net/emacs/icalendar/icalendar-mode.el

I could release this work as a package, but as I describe in more
detail in the write-up, I think there's a good case that an improved
iCalendar library belongs in Emacs' core. There are currently at least
*three* partial iCalendar implementations in Emacs (icalendar.el,
gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
particular major mode (diary, Gnus, and Org). I think it would be good
to consolidate this work in one place and generalize it so that all
three of these applications, as well as third party packages, can
benefit.

So, some questions for the list:

1) Is there interest in getting this code, and/or a future version of
such a library, into Emacs?

And if so, 
2) Would anyone here be willing to mentor me/collaborate with me on it?
3) What should the library's API look like? What would be most useful?

Many thanks for your thoughts and feedback!

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Date: Fri, 18 Oct 2024 11:01:10 +0200
> 
> So, some questions for the list:
> 
> 1) Is there interest in getting this code, and/or a future version of
> such a library, into Emacs?

We are definitely interested in developing icalendar.el to cover the
missing functionalities and support the recent standards, guidelines
and accepted practices.

> And if so, 
> 2) Would anyone here be willing to mentor me/collaborate with me on it?

I hope someone will step forward, but the very least we can promise is
that any questions you ask here will be answered.

> 3) What should the library's API look like? What would be most useful?

I think this is up to you as the expert.

Thanks!

Re: Improving Emacs' iCalendar support

[Sebastián Monía]

Hello Richard!

Richard Lawrence <rwl@recursewithless.net> writes:
> Dear emacs-devel,
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el

Will take a look, thank you for sharing!

> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.

I am only familiar with icalendar.el, which I consumed in this CalDAV
sync package: https://git.sr.ht/~sebasmonia/cdsync.el
I haven't used it much lately, but while testing it I found a few cases
not supported by icalendar.el At some point I wanted to try my hand at
fixing some bugs reported in the ical -> diary conversion.

> So, some questions for the list:
> 2) Would anyone here be willing to mentor me/collaborate with me on
> it?

I would be happy to help.

> 3) What should the library's API look like? What would be most useful?

Like Eli said in another reply, you are probably the best person to
define this now - until the library is merged, the API can change, so
don't stress too much over that _yet_ :)

Regards,
Seb

-- 
Sebastián Monía
https://site.sebasmonia.com/

Re: Improving Emacs' iCalendar support

[Adam Porter]

Hi Richard,

I don't have much to add to what's been said, but my first thought is 
that a robust iCal library would probably be most useful for importing 
and exporting between the iCal format, obviously.  So I'd suggest that 
its API should help with transforming iCal data into other formats, 
probably by converting it into a simple Lisp data structure that other 
libraries could work with.  And then, it should also provide a function 
to transform that data structure back into an iCal file.

Also, it would be good if it were designed to be extensible in case the 
standards receive updates, or in case vendor-specific functionality is 
needed by users.

So I'd suggest looking at the existing Emacs/Org applications that use 
iCal, and designing your API so that such applications would find it 
easy to use (not that it should directly imitate what already exists, 
unless such APIs are already ideal).

Finally, I'd suggest that you consider using structs as the primary 
internal data structure, because they provide setter and accessor 
functions, which are helpful to developers (e.g. they help catch some 
mistakes at compilation time).  If you do, I've found it helpful to 
define an "etc" slot in various structs to which an alist or plist can 
be attached, which provides the ability for the structs to carry 
additional data that wasn't foreseen during its design.

--Adam

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> 1) Is there interest in getting this code, and/or a future version of
>> such a library, into Emacs?
>
> We are definitely interested in developing icalendar.el to cover the
> missing functionalities and support the recent standards, guidelines
> and accepted practices.

Great!

>> And if so, 
>> 2) Would anyone here be willing to mentor me/collaborate with me on it?
>
> I hope someone will step forward, but the very least we can promise is
> that any questions you ask here will be answered.

Will do. I'm glad a few other people have already expressed interest!

Another question: what is the best place for this work to happen? In a
separate repo for now? In a feature branch in the Emacs repo? Should I
send what I've already got as a patch to this list?

I believe I signed FSF paperwork about ten years ago but am no longer
100% sure that I completed the process (there was some uncertainty
because of my employer back then, UC Berkeley). Does someone here know
how I can check my status?

Thanks!
Richard

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Sebastián,

Sebastián Monía <sebastian@sebasmonia.com> writes:

> I am only familiar with icalendar.el, which I consumed in this CalDAV
> sync package: https://git.sr.ht/~sebasmonia/cdsync.el

This looks great! A working CalDAV implementation would be a very useful
addition to the library I'm envisioning.

> I would be happy to help.

Wonderful! Since you're on sr.ht I posted the work I've done so far in a
repository there:

https://git.sr.ht/~wyleyr/icalendar-el

Perhaps you could take a look and let me know what you'd be interested
in working on. 

Thanks!
Richard

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Adam,

Adam Porter <adam@alphapapa.net> writes:

> I don't have much to add to what's been said, but my first thought is 
> that a robust iCal library would probably be most useful for importing 
> and exporting between the iCal format, obviously.  So I'd suggest that 
> its API should help with transforming iCal data into other formats, 
> probably by converting it into a simple Lisp data structure that other 
> libraries could work with.  And then, it should also provide a function 
> to transform that data structure back into an iCal file.
>
> Also, it would be good if it were designed to be extensible in case the 
> standards receive updates, or in case vendor-specific functionality is 
> needed by users.

Yep, that's exactly the plan! Thanks for confirming that this is a good
idea.

> So I'd suggest looking at the existing Emacs/Org applications that use 
> iCal, and designing your API so that such applications would find it 
> easy to use (not that it should directly imitate what already exists, 
> unless such APIs are already ideal).

Yes, I was hoping especially to hear from Org/Gnus/diary developers if
they have any thoughts about what would make such a library easy (or
-ier) for them to use. Perhaps I should ask on their respective
lists...does Gnus have a separate mailing list? I will also grep through
the code to see if anything else in Emacs is using the icalendar--*
internal functions.

> Finally, I'd suggest that you consider using structs as the primary 
> internal data structure, because they provide setter and accessor 
> functions, which are helpful to developers (e.g. they help catch some 
> mistakes at compilation time).  If you do, I've found it helpful to 
> define an "etc" slot in various structs to which an alist or plist can 
> be attached, which provides the ability for the structs to carry 
> additional data that wasn't foreseen during its design.

Thanks, that's useful advice. I've been toying with the idea of using
EIEIO classes to represent at least the "component"-level data
structures (events, to-dos, journals, timezones, whole calendars), which
if I understand correctly use structs under the hood. But I don't really
have any experience with EIEIO and I'm not sure if the added complexity
of the object system will buy much over using plain structs. Do you have
any thoughts about this?

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 19 Oct 2024 10:22:13 +0200
> 
> I believe I signed FSF paperwork about ten years ago but am no longer
> 100% sure that I completed the process (there was some uncertainty
> because of my employer back then, UC Berkeley). Does someone here know
> how I can check my status?

I don't see any assignment from you on file, so I think you will need
to do the paperwork again, unless you did it under some different name
and/or email address.

Re: Improving Emacs' iCalendar support

[Adam Porter]

Hi Richard,

On 10/19/24 03:39, Richard Lawrence wrote:

> Thanks, that's useful advice. I've been toying with the idea of using
> EIEIO classes to represent at least the "component"-level data
> structures (events, to-dos, journals, timezones, whole calendars), which
> if I understand correctly use structs under the hood. But I don't really
> have any experience with EIEIO and I'm not sure if the added complexity
> of the object system will buy much over using plain structs. Do you have
> any thoughts about this?

I'd recommend sticking with CL-STRUCTs unless and until you find a real 
need for EIEIO's additional functionality.

My point of experience is having used EIEIO for my work on the 
now-obsolete matrix-client.el package: It worked fine, but I found that 
I never used more than what CL-STRUCT's API offers; so in the successor, 
Ement.el, I've just used structs, and it's worked out very well.

If you're still not sure which to use, you might talk with Jonas 
Bernoulli (aka tarsius, the Magit maintainer), as he's written more 
EIEIO-related code than I have, and his perspective could be valuable.

--Adam

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Lawrence <rwl@recursewithless.net>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 19 Oct 2024 10:22:13 +0200
>> 
>> I believe I signed FSF paperwork about ten years ago but am no longer
>> 100% sure that I completed the process (there was some uncertainty
>> because of my employer back then, UC Berkeley). Does someone here know
>> how I can check my status?
>
> I don't see any assignment from you on file, so I think you will need
> to do the paperwork again, unless you did it under some different name
> and/or email address.

It would have been the same name but a different email address (probably
wyley.r@gmail.com, possibly richard.lawrence@berkeley.edu). Also, FWIW,
I did the paperwork through the Org Mode maintainers back then, so if
that's a separate list, you could check there.

In any case, if you send me the paperwork, I'm happy to do it again.

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Cc: emacs-devel@gnu.org
> Date: Sun, 20 Oct 2024 07:08:57 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Richard Lawrence <rwl@recursewithless.net>
> >> Cc: emacs-devel@gnu.org
> >> Date: Sat, 19 Oct 2024 10:22:13 +0200
> >> 
> >> I believe I signed FSF paperwork about ten years ago but am no longer
> >> 100% sure that I completed the process (there was some uncertainty
> >> because of my employer back then, UC Berkeley). Does someone here know
> >> how I can check my status?
> >
> > I don't see any assignment from you on file, so I think you will need
> > to do the paperwork again, unless you did it under some different name
> > and/or email address.
> 
> It would have been the same name but a different email address (probably
> wyley.r@gmail.com, possibly richard.lawrence@berkeley.edu). Also, FWIW,
> I did the paperwork through the Org Mode maintainers back then, so if
> that's a separate list, you could check there.

I don't see anything like that.  I think Org assignments are handled
the same as Emacs assignments and should be recorded on the same file.

> In any case, if you send me the paperwork, I'm happy to do it again.

Form sent off-list.

Thanks.

Re: Improving Emacs' iCalendar support

[Björn Bidar]

Good Morning from Finland,

Richard Lawrence <rwl@recursewithless.net> writes:

> Dear emacs-devel,
>
> Hello! I've been a happy Emacs user for almost twenty years now(!), but
> this is my first time subscribing to this list, so please help me follow
> the local conventions.
>
> I would like to start a discussion about improving Emacs' iCalendar
> support, beyond what is already available in icalendar.el. I personally
> would like to see Emacs gain a more full-fledged RFC5545 implementation
> that is primarily designed as a library for other applications to use.
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el
>
> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.

From my dealings will the latter two I noticed that currently these
separatism make it harder to support all features and maybe in some way
easier.

For example Gnus does deal with the responding to these invitations
while org focuses on the exporting. Not all features are support with
any of these making some things such as recurring events quite dangerous
to deal with if you interact with others.

It would be great to have these features supported better. In think
some limitations in for example org-mode come also into play here as
last time I looked it can't deal with timezones.

Some parts of the current approach limit the amount of features or make
it harder as all of these move on.
For example Gnus uses org-capture to convert an icalendar
event to org instead of some form of native API to synchronize the
invite with-org.

Any advancement in regards of icalendar hopefully also helps the
other related formats such as vtodo and vcard.

Maybe also cc the org-mode mailing-list.

Another consumer of such efforts would be also (indirectly) org-caldav.
When it comes to the -dav part of this than also bbdb.

Have a good day,

Björn

Re: Improving Emacs' iCalendar support

[Visuwesh]

[வெள்ளி அக்டோபர் 18, 2024] Richard Lawrence wrote:

> Dear emacs-devel,
>
> Hello! I've been a happy Emacs user for almost twenty years now(!), but
> this is my first time subscribing to this list, so please help me follow
> the local conventions.
>
> I would like to start a discussion about improving Emacs' iCalendar
> support, beyond what is already available in icalendar.el. I personally
> would like to see Emacs gain a more full-fledged RFC5545 implementation
> that is primarily designed as a library for other applications to use.
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el
>
> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.
>
> So, some questions for the list:
>
> 1) Is there interest in getting this code, and/or a future version of
> such a library, into Emacs?

I don't use much of the icalendar stuff except this one function that I
use to extract holidays from govt. supplied ical file:

    (defun vz/get-indian-holidays ()
      "Parse the icalendar file provided by the Indian government.
    This returns a list that can be used in calendar's holiday."
      (unless (file-exists-p (expand-file-name "~/lib/indianical/"))
        (make-directory (expand-file-name "~/lib/indianical/") t))
      (let* ((year (decoded-time-year (decode-time)))
             (file (expand-file-name (format "~/lib/indianical/%d" year))))
        (unless (file-exists-p file)
          (url-copy-file
           (format "http://www.india.gov.in/calendar/%d/export.ics" year)
           file))
        (with-temp-buffer
          (require 'icalendar)
          (insert-file-contents file)
          ;; (set-buffer-file-coding-system 'unix)
          (goto-char (point-min))
          (seq-keep
           (lambda (x)
             (when-let* ((description (icalendar--get-event-property x 'DESCRIPTION))
                         ((seq-find (lambda (h) (let (case-fold-search)
                                                    (string-match-p h description)))
                                      vz/indian-holidays)))
               (let ((time (icalendar--decode-isodatetime (icalendar--get-event-property x 'DTSTART))))
                 (list 'holiday-fixed (decoded-time-month time) (decoded-time-day time)
                       (string-trim-right description " +([GR])")))))
           (icalendar--all-events (icalendar--read-element nil nil))))))

It has been at least a couple years since I wrote this function so I
don't remember the hurdles I faced when I wrote it very well but it
perplexed me that there were no "public" functions that were geared
towards parsing an ical file.  In retrospect, it would have been less
confusing if these functions were "public" functions.  AFAIR, and this
very well might not be true, the structure returned
icalendar--read-element is not really documented anywhere which further
added to the confusion.

That said, this function has been rock solid ever since I wrote it.
Going by the ical files I have, it is working just fine since 2021.