Improving Emacs' iCalendar support

Improving Emacs' iCalendar support

[Richard Lawrence]

Dear emacs-devel,

Hello! I've been a happy Emacs user for almost twenty years now(!), but
this is my first time subscribing to this list, so please help me follow
the local conventions.

I would like to start a discussion about improving Emacs' iCalendar
support, beyond what is already available in icalendar.el. I personally
would like to see Emacs gain a more full-fledged RFC5545 implementation
that is primarily designed as a library for other applications to use.

This is an itch that's been bugging me for a while, and so for the past
couple of weeks I've been working on scratching it, and I now have a
reasonable chunk of work to share: I've drafted a new implementation of
the iCalendar grammar, and a major mode which uses this grammar to
provide syntax highlighting. I wrote up what I've done and why here:

https://recursewithless.net/emacs/icalendar-parser-and-mode.org

That's a literate Org mode file containing the code and my commentary.
If you just want to read the code itself, see:

https://recursewithless.net/emacs/icalendar/icalendar-parser.el
https://recursewithless.net/emacs/icalendar/icalendar-mode.el

I could release this work as a package, but as I describe in more
detail in the write-up, I think there's a good case that an improved
iCalendar library belongs in Emacs' core. There are currently at least
*three* partial iCalendar implementations in Emacs (icalendar.el,
gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
particular major mode (diary, Gnus, and Org). I think it would be good
to consolidate this work in one place and generalize it so that all
three of these applications, as well as third party packages, can
benefit.

So, some questions for the list:

1) Is there interest in getting this code, and/or a future version of
such a library, into Emacs?

And if so, 
2) Would anyone here be willing to mentor me/collaborate with me on it?
3) What should the library's API look like? What would be most useful?

Many thanks for your thoughts and feedback!

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Date: Fri, 18 Oct 2024 11:01:10 +0200
> 
> So, some questions for the list:
> 
> 1) Is there interest in getting this code, and/or a future version of
> such a library, into Emacs?

We are definitely interested in developing icalendar.el to cover the
missing functionalities and support the recent standards, guidelines
and accepted practices.

> And if so, 
> 2) Would anyone here be willing to mentor me/collaborate with me on it?

I hope someone will step forward, but the very least we can promise is
that any questions you ask here will be answered.

> 3) What should the library's API look like? What would be most useful?

I think this is up to you as the expert.

Thanks!

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> writes:

>> 1) Is there interest in getting this code, and/or a future version of
>> such a library, into Emacs?
>
> We are definitely interested in developing icalendar.el to cover the
> missing functionalities and support the recent standards, guidelines
> and accepted practices.

Great!

>> And if so, 
>> 2) Would anyone here be willing to mentor me/collaborate with me on it?
>
> I hope someone will step forward, but the very least we can promise is
> that any questions you ask here will be answered.

Will do. I'm glad a few other people have already expressed interest!

Another question: what is the best place for this work to happen? In a
separate repo for now? In a feature branch in the Emacs repo? Should I
send what I've already got as a patch to this list?

I believe I signed FSF paperwork about ten years ago but am no longer
100% sure that I completed the process (there was some uncertainty
because of my employer back then, UC Berkeley). Does someone here know
how I can check my status?

Thanks!
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 19 Oct 2024 10:22:13 +0200
> 
> I believe I signed FSF paperwork about ten years ago but am no longer
> 100% sure that I completed the process (there was some uncertainty
> because of my employer back then, UC Berkeley). Does someone here know
> how I can check my status?

I don't see any assignment from you on file, so I think you will need
to do the paperwork again, unless you did it under some different name
and/or email address.

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Lawrence <rwl@recursewithless.net>
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 19 Oct 2024 10:22:13 +0200
>> 
>> I believe I signed FSF paperwork about ten years ago but am no longer
>> 100% sure that I completed the process (there was some uncertainty
>> because of my employer back then, UC Berkeley). Does someone here know
>> how I can check my status?
>
> I don't see any assignment from you on file, so I think you will need
> to do the paperwork again, unless you did it under some different name
> and/or email address.

It would have been the same name but a different email address (probably
wyley.r@gmail.com, possibly richard.lawrence@berkeley.edu). Also, FWIW,
I did the paperwork through the Org Mode maintainers back then, so if
that's a separate list, you could check there.

In any case, if you send me the paperwork, I'm happy to do it again.

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Cc: emacs-devel@gnu.org
> Date: Sun, 20 Oct 2024 07:08:57 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Richard Lawrence <rwl@recursewithless.net>
> >> Cc: emacs-devel@gnu.org
> >> Date: Sat, 19 Oct 2024 10:22:13 +0200
> >> 
> >> I believe I signed FSF paperwork about ten years ago but am no longer
> >> 100% sure that I completed the process (there was some uncertainty
> >> because of my employer back then, UC Berkeley). Does someone here know
> >> how I can check my status?
> >
> > I don't see any assignment from you on file, so I think you will need
> > to do the paperwork again, unless you did it under some different name
> > and/or email address.
> 
> It would have been the same name but a different email address (probably
> wyley.r@gmail.com, possibly richard.lawrence@berkeley.edu). Also, FWIW,
> I did the paperwork through the Org Mode maintainers back then, so if
> that's a separate list, you could check there.

I don't see anything like that.  I think Org assignments are handled
the same as Emacs assignments and should be recorded on the same file.

> In any case, if you send me the paperwork, I'm happy to do it again.

Form sent off-list.

Thanks.

Re: Improving Emacs' iCalendar support

[Sebastián Monía]

Hello Richard!

Richard Lawrence <rwl@recursewithless.net> writes:
> Dear emacs-devel,
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el

Will take a look, thank you for sharing!

> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.

I am only familiar with icalendar.el, which I consumed in this CalDAV
sync package: https://git.sr.ht/~sebasmonia/cdsync.el
I haven't used it much lately, but while testing it I found a few cases
not supported by icalendar.el At some point I wanted to try my hand at
fixing some bugs reported in the ical -> diary conversion.

> So, some questions for the list:
> 2) Would anyone here be willing to mentor me/collaborate with me on
> it?

I would be happy to help.

> 3) What should the library's API look like? What would be most useful?

Like Eli said in another reply, you are probably the best person to
define this now - until the library is merged, the API can change, so
don't stress too much over that _yet_ :)

Regards,
Seb

-- 
Sebastián Monía
https://site.sebasmonia.com/

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Sebastián,

Sebastián Monía <sebastian@sebasmonia.com> writes:

> I am only familiar with icalendar.el, which I consumed in this CalDAV
> sync package: https://git.sr.ht/~sebasmonia/cdsync.el

This looks great! A working CalDAV implementation would be a very useful
addition to the library I'm envisioning.

> I would be happy to help.

Wonderful! Since you're on sr.ht I posted the work I've done so far in a
repository there:

https://git.sr.ht/~wyleyr/icalendar-el

Perhaps you could take a look and let me know what you'd be interested
in working on. 

Thanks!
Richard

Re: Improving Emacs' iCalendar support

[Adam Porter]

Hi Richard,

I don't have much to add to what's been said, but my first thought is 
that a robust iCal library would probably be most useful for importing 
and exporting between the iCal format, obviously.  So I'd suggest that 
its API should help with transforming iCal data into other formats, 
probably by converting it into a simple Lisp data structure that other 
libraries could work with.  And then, it should also provide a function 
to transform that data structure back into an iCal file.

Also, it would be good if it were designed to be extensible in case the 
standards receive updates, or in case vendor-specific functionality is 
needed by users.

So I'd suggest looking at the existing Emacs/Org applications that use 
iCal, and designing your API so that such applications would find it 
easy to use (not that it should directly imitate what already exists, 
unless such APIs are already ideal).

Finally, I'd suggest that you consider using structs as the primary 
internal data structure, because they provide setter and accessor 
functions, which are helpful to developers (e.g. they help catch some 
mistakes at compilation time).  If you do, I've found it helpful to 
define an "etc" slot in various structs to which an alist or plist can 
be attached, which provides the ability for the structs to carry 
additional data that wasn't foreseen during its design.

--Adam

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Adam,

Adam Porter <adam@alphapapa.net> writes:

> I don't have much to add to what's been said, but my first thought is 
> that a robust iCal library would probably be most useful for importing 
> and exporting between the iCal format, obviously.  So I'd suggest that 
> its API should help with transforming iCal data into other formats, 
> probably by converting it into a simple Lisp data structure that other 
> libraries could work with.  And then, it should also provide a function 
> to transform that data structure back into an iCal file.
>
> Also, it would be good if it were designed to be extensible in case the 
> standards receive updates, or in case vendor-specific functionality is 
> needed by users.

Yep, that's exactly the plan! Thanks for confirming that this is a good
idea.

> So I'd suggest looking at the existing Emacs/Org applications that use 
> iCal, and designing your API so that such applications would find it 
> easy to use (not that it should directly imitate what already exists, 
> unless such APIs are already ideal).

Yes, I was hoping especially to hear from Org/Gnus/diary developers if
they have any thoughts about what would make such a library easy (or
-ier) for them to use. Perhaps I should ask on their respective
lists...does Gnus have a separate mailing list? I will also grep through
the code to see if anything else in Emacs is using the icalendar--*
internal functions.

> Finally, I'd suggest that you consider using structs as the primary 
> internal data structure, because they provide setter and accessor 
> functions, which are helpful to developers (e.g. they help catch some 
> mistakes at compilation time).  If you do, I've found it helpful to 
> define an "etc" slot in various structs to which an alist or plist can 
> be attached, which provides the ability for the structs to carry 
> additional data that wasn't foreseen during its design.

Thanks, that's useful advice. I've been toying with the idea of using
EIEIO classes to represent at least the "component"-level data
structures (events, to-dos, journals, timezones, whole calendars), which
if I understand correctly use structs under the hood. But I don't really
have any experience with EIEIO and I'm not sure if the added complexity
of the object system will buy much over using plain structs. Do you have
any thoughts about this?

Best,
Richard

Re: Improving Emacs' iCalendar support

[Adam Porter]

Hi Richard,

On 10/19/24 03:39, Richard Lawrence wrote:

> Thanks, that's useful advice. I've been toying with the idea of using
> EIEIO classes to represent at least the "component"-level data
> structures (events, to-dos, journals, timezones, whole calendars), which
> if I understand correctly use structs under the hood. But I don't really
> have any experience with EIEIO and I'm not sure if the added complexity
> of the object system will buy much over using plain structs. Do you have
> any thoughts about this?

I'd recommend sticking with CL-STRUCTs unless and until you find a real 
need for EIEIO's additional functionality.

My point of experience is having used EIEIO for my work on the 
now-obsolete matrix-client.el package: It worked fine, but I found that 
I never used more than what CL-STRUCT's API offers; so in the successor, 
Ement.el, I've just used structs, and it's worked out very well.

If you're still not sure which to use, you might talk with Jonas 
Bernoulli (aka tarsius, the Magit maintainer), as he's written more 
EIEIO-related code than I have, and his perspective could be valuable.

--Adam

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Richard Lawrence <rwl@recursewithless.net> writes:

>> So I'd suggest looking at the existing Emacs/Org applications that use 
>> iCal, and designing your API so that such applications would find it 
>> easy to use (not that it should directly imitate what already exists, 
>> unless such APIs are already ideal).

Some kind of icalendar parser could be useful.

In Org mode, we currently do icalendar export by hand-writing iCalendar
entries as strings and need to be careful about things like line endings
and special symbols.

What could be done instead is some kind of Elisp API similar to
`xml-parse-region' and `xml-print', so that things like escaping
symbols, exact literals, newlines, etc are handled behind the scenes, and
we could instead just define an AST and "print" it into a valid
iCalendar file.

So, my suggestion would be something akin xml.el and maybe (optionally)
https://github.com/ndwarshuis/org-ml - to manipulate the AST.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Björn Bidar]

Ihor Radchenko <yantar92@posteo.net> writes:

> What could be done instead is some kind of Elisp API similar to
> `xml-parse-region' and `xml-print', so that things like escaping
> symbols, exact literals, newlines, etc are handled behind the scenes, and
> we could instead just define an AST and "print" it into a valid
> iCalendar file.

Is the VCard and Vtodo file format similar enough to icalendar in the base form
that for these base line handling of these formats could be done
together for these?

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Is the VCard and Vtodo file format similar enough to icalendar in the base form
> that for these base line handling of these formats could be done
> together for these?

Not sure, but quick googling shows that icalendar is the successor of
vcard: https://stackoverflow.com/questions/1310420/difference-between-icalendar-ics-and-the-vcalendar-vcs

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Björn,

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Is the VCard and Vtodo file format similar enough to icalendar in the base form
> that for these base line handling of these formats could be done
> together for these?

I haven't looked at the RFC, but a quick look at the example on
Wikipedia tells me that vCard should be very similar and probably can
use the same parsing infrastructure that I'm building for iCalendar now.

However, I am unable to find any information about a "Vtodo file
format". There is a VTODO component within the iCalendar standard -- is
that what you mean, or do you have something else in mind?

Best,
Richard

Re: Improving Emacs' iCalendar support

[Björn Bidar]

Richard Lawrence <rwl@recursewithless.net> writes:

> However, I am unable to find any information about a "Vtodo file
> format". There is a VTODO component within the iCalendar standard -- is
> that what you mean, or do you have something else in mind?

Yes that's what I meant, maybe I mixed something up or separated them
even thou they part of the same standard.

In general the basic parsing of all these V-format is very similar.
There are more of these such as vMessage[1] although that one is unrelated
to this.

[1] https://en.wikipedia.org/wiki/VMessage

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Some kind of icalendar parser could be useful.

Great to know you'd be interested in this!

> In Org mode, we currently do icalendar export by hand-writing iCalendar
> entries as strings and need to be careful about things like line endings
> and special symbols.

Yes. 

> What could be done instead is some kind of Elisp API similar to
> `xml-parse-region' and `xml-print', so that things like escaping
> symbols, exact literals, newlines, etc are handled behind the scenes, and
> we could instead just define an AST and "print" it into a valid
> iCalendar file.
>
> So, my suggestion would be something akin xml.el and maybe (optionally)
> https://github.com/ndwarshuis/org-ml - to manipulate the AST.

Thank you, this is a very useful suggestion. I wasn't aware that xml.el
had an API like this -- I will take a look.

At the moment, I am drafting a set of data structures that the parser
will produce, and the printer will consume. Following Adam's suggestion,
I've been basing these on cl-structs (except for simple atomic values). So
eventually you should be able to write code that's something like this:

(icalendar-make-vevent
  ;; properties:
  (list (icalendar-make-property "DTSTART"
           (icalendar-datetime-from-timestamp some-lisp-timestamp))
        (icalendar-make-property "DTEND"
           (icalendar-datetime-from-timestamp some-other-timestamp)))
  ;; subcomponents:
  (list (icalendar-make-valarm ...))
  ...)
           

and get a struct back that you could then send directly to a function
like 'icalendar-print-component' or similar. Does that seem like a
reasonable approach, or is it already too cumbersome? (Would you prefer
e.g. to just be able to write the appropriate data structure as a
backquoted (p)list?)

Best,
Richard

Re: Improving Emacs' iCalendar support

[Ferdinand Pieper]

(Resending just to list due to messed up spf entry; sorry for the trouble)

Hi Richard,

this effort to consolidate the iCalendar support is fantastic!

Emacs is really lacking a central parser for iCalendar and a common internal representation of iCalendar items.
I have ran over this several times, but was always very quick to shy away from working on a central overhaul of icalendar.el. I resorted to small extensions of gnus-icalendar.el to bring event (request) creation to gnus, which still is very much in a prototype state [1].

While I started extending the gnus-icalendar-event EIEIO class to support property parameters, it ended up being very cumbersome to create new event objects with all parameters correctly set.

Richard Lawrence <rwl@recursewithless.net> writes:

> (icalendar-make-vevent
>   ;; properties:
>   (list (icalendar-make-property "DTSTART"
>            (icalendar-datetime-from-timestamp some-lisp-timestamp))
>         (icalendar-make-property "DTEND"
>            (icalendar-datetime-from-timestamp some-other-timestamp)))
>   ;; subcomponents:
>   (list (icalendar-make-valarm ...))
>   ...)
>            
>
> and get a struct back that you could then send directly to a function
> like 'icalendar-print-component' or similar. Does that seem like a
> reasonable approach, or is it already too cumbersome? (Would you prefer
> e.g. to just be able to write the appropriate data structure as a
> backquoted (p)list?)

I think structs could be a good fit for the internal representation. Creating these from scratch should still be kept fairly straightforward for ease of use.
There could also be a translation function that creates the appropriate struct from a very basic list representation like this:

--8<---------------cut here---------------start------------->8---
(VCALENDAR
 ((PRODID nil "-//...")
  (VERSION nil "2.0"))
 (VEVENT
  (;; ...
   (DTSTART ((VALUE "DATE")) "20241201")
   (DTEND ((VALUE "DATE")) "20241231"))))
--8<---------------cut here---------------end--------------->8---


Best,
Ferdinand

[1]  https://git.nand.xyz/emacs/gnus-icalendar-request/ or mirrored at 
https://github.com/fpiper/gnus-icalendar-request

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Ferdinand,

Ferdinand Pieper <ferdi@nand.xyz> writes:

> Emacs is really lacking a central parser for iCalendar and a common
> internal representation of iCalendar items. I have ran over this
> several times, but was always very quick to shy away from working on a
> central overhaul of icalendar.el. I resorted to small extensions of
> gnus-icalendar.el to bring event (request) creation to gnus, which
> still is very much in a prototype state [1].
>
> While I started extending the gnus-icalendar-event EIEIO class to
> support property parameters, it ended up being very cumbersome to
> create new event objects with all parameters correctly set.

This is good feedback, thank you. Can you say more about what exactly
was cumbersome about it? (Was it the need to call various constructors,
instead of just consing up lists? or the need to interact with objects
via setters/getters? or lack of validation of parameter lists? or...?)

> I think structs could be a good fit for the internal representation.
> Creating these from scratch should still be kept fairly
> straightforward for ease of use. There could also be a translation
> function that creates the appropriate struct from a very basic list
> representation like this:
>
> --8<---------------cut here---------------start------------->8---
> (VCALENDAR
>  ((PRODID nil "-//...")
>   (VERSION nil "2.0"))
>  (VEVENT
>   (;; ...
>    (DTSTART ((VALUE "DATE")) "20241201")
>    (DTEND ((VALUE "DATE")) "20241231"))))
> --8<---------------cut here---------------end--------------->8---

Yes, as I'm sure you're aware, icalendar.el creates a list
representation like this. I've been thinking we might need such a
function anyway for the sake of backward compatibility. And the more I
think about it, the more I realize that a list representation with a few
accessors should be just fine for the "container" types (components,
properties, attribute lists).

The disadvantage of a representation like the one above is there is no
structure yet in the *value* types: e.g. "20241201" is just a string,
not a date. This is fine for export (assuming you've already validated
the data...) but when reading the data, one needs to parse these values
too, and this is what I've been using structs for so far.

So do you think it would still be too cumbersome to have a
representation like

> ... (DTSTART ((VALUE "DATE")) (icalendar-read-date "20241201"))

or even just

> ... (DTSTART ((VALUE "DATE")) ('icalendar-date . "20241201"))

where in the latter, 'icalendar-date is a cl type symbol and some magic
type dispatch will call the associated read function on the string at
an appropriate time?

Best,
Richard

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Richard Lawrence <rwl@recursewithless.net> writes:

>> While I started extending the gnus-icalendar-event EIEIO class to
>> support property parameters, it ended up being very cumbersome to
>> create new event objects with all parameters correctly set.

As an alternative option, you may consider using org-element-ast API -
it is a generic data structure and API that Org mode uses to handle
parsed AST. It is servility optimized for performance and does not
really depend on other parts of Org mode.

The top level commentary should have all the details.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Ihor,

Ihor Radchenko <yantar92@posteo.net> writes:

> Richard Lawrence <rwl@recursewithless.net> writes:
>
>>> While I started extending the gnus-icalendar-event EIEIO class to
>>> support property parameters, it ended up being very cumbersome to
>>> create new event objects with all parameters correctly set.
>
> As an alternative option, you may consider using org-element-ast API -
> it is a generic data structure and API that Org mode uses to handle
> parsed AST. It is servility optimized for performance and does not
> really depend on other parts of Org mode.

Thanks for pointing me to this! I took a look at the file commentary,
and I've played around with org-element before. I do like org-element's
plist-based format...it makes things simple and readable as an outside
tinkerer.

Can you tell me a bit more about the performance benefits? Do these
mostly just stem from the use of an array for :standard-properties?
(You get this for free with cl-structs, too.) Or is it more about when
and how parsing happens? (This is something I feel like I've never
grasped in org-element...in the past I've been confused because the
properties I'd see when calling e.g. org-element-at-point would be
different than the ones available in some other context.)

Thanks!

Best,
Richard

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Richard Lawrence <rwl@recursewithless.net> writes:

> Can you tell me a bit more about the performance benefits? Do these
> mostly just stem from the use of an array for :standard-properties?

:standard-properties and lazy evaluation.

> (You get this for free with cl-structs, too.)

Not exactly for free. There are extra type checks in cl-structs, and they
are slower to create. I did the benchmarking and decided to go with less
sophisticated data structure supported by inlined accessors/setters.

Also, as a technical detail that is irrelevant to your case, I had to
preserve backwards compatibility with the existing design to the highest
extent possible.

> ... Or is it more about when
> and how parsing happens? (This is something I feel like I've never
> grasped in org-element...in the past I've been confused because the
> properties I'd see when calling e.g. org-element-at-point would be
> different than the ones available in some other context.)

org-element-ast has little to do with parsing details. However, it does
provide features we need during parsing: (1) storing arbitrary
properties in addition to "standard"; (2) supporting "plain string" as a
special variant of AST leaf; (3) supporting anonymous nodes - simple
list of AST nodes; (4) lazy evaluation of properties - org-element uses
the APIs from org-element-ast to defer large parts of parsing to when
the relevant part of the AST node is actually requested by a caller.

For org-element-at-point, you may be confused because Org parser can
work in several modes, parsing down to certain
"granularity". org-element-at-point usually limits itself to parsing
down to paragraph level (in particular, it does not parse markup
inside headline :title leaving it as a simple plain text leaf), while
what you see during export is the most complete parsing mode, reaching
down to markup (in headlines, :title then contains a collection of Org
"objects" - the markup is parsed).  This has nothing to do with the AST
layout.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Björn Bidar]

Good Morning from Finland,

Richard Lawrence <rwl@recursewithless.net> writes:

> Dear emacs-devel,
>
> Hello! I've been a happy Emacs user for almost twenty years now(!), but
> this is my first time subscribing to this list, so please help me follow
> the local conventions.
>
> I would like to start a discussion about improving Emacs' iCalendar
> support, beyond what is already available in icalendar.el. I personally
> would like to see Emacs gain a more full-fledged RFC5545 implementation
> that is primarily designed as a library for other applications to use.
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el
>
> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.

From my dealings will the latter two I noticed that currently these
separatism make it harder to support all features and maybe in some way
easier.

For example Gnus does deal with the responding to these invitations
while org focuses on the exporting. Not all features are support with
any of these making some things such as recurring events quite dangerous
to deal with if you interact with others.

It would be great to have these features supported better. In think
some limitations in for example org-mode come also into play here as
last time I looked it can't deal with timezones.

Some parts of the current approach limit the amount of features or make
it harder as all of these move on.
For example Gnus uses org-capture to convert an icalendar
event to org instead of some form of native API to synchronize the
invite with-org.

Any advancement in regards of icalendar hopefully also helps the
other related formats such as vtodo and vcard.

Maybe also cc the org-mode mailing-list.

Another consumer of such efforts would be also (indirectly) org-caldav.
When it comes to the -dav part of this than also bbdb.

Have a good day,

Björn

Re: Improving Emacs' iCalendar support

[Visuwesh]

[வெள்ளி அக்டோபர் 18, 2024] Richard Lawrence wrote:

> Dear emacs-devel,
>
> Hello! I've been a happy Emacs user for almost twenty years now(!), but
> this is my first time subscribing to this list, so please help me follow
> the local conventions.
>
> I would like to start a discussion about improving Emacs' iCalendar
> support, beyond what is already available in icalendar.el. I personally
> would like to see Emacs gain a more full-fledged RFC5545 implementation
> that is primarily designed as a library for other applications to use.
>
> This is an itch that's been bugging me for a while, and so for the past
> couple of weeks I've been working on scratching it, and I now have a
> reasonable chunk of work to share: I've drafted a new implementation of
> the iCalendar grammar, and a major mode which uses this grammar to
> provide syntax highlighting. I wrote up what I've done and why here:
>
> https://recursewithless.net/emacs/icalendar-parser-and-mode.org
>
> That's a literate Org mode file containing the code and my commentary.
> If you just want to read the code itself, see:
>
> https://recursewithless.net/emacs/icalendar/icalendar-parser.el
> https://recursewithless.net/emacs/icalendar/icalendar-mode.el
>
> I could release this work as a package, but as I describe in more
> detail in the write-up, I think there's a good case that an improved
> iCalendar library belongs in Emacs' core. There are currently at least
> *three* partial iCalendar implementations in Emacs (icalendar.el,
> gnus-icalendar.el, and ox-icalendar.el), which are each focused on a
> particular major mode (diary, Gnus, and Org). I think it would be good
> to consolidate this work in one place and generalize it so that all
> three of these applications, as well as third party packages, can
> benefit.
>
> So, some questions for the list:
>
> 1) Is there interest in getting this code, and/or a future version of
> such a library, into Emacs?

I don't use much of the icalendar stuff except this one function that I
use to extract holidays from govt. supplied ical file:

    (defun vz/get-indian-holidays ()
      "Parse the icalendar file provided by the Indian government.
    This returns a list that can be used in calendar's holiday."
      (unless (file-exists-p (expand-file-name "~/lib/indianical/"))
        (make-directory (expand-file-name "~/lib/indianical/") t))
      (let* ((year (decoded-time-year (decode-time)))
             (file (expand-file-name (format "~/lib/indianical/%d" year))))
        (unless (file-exists-p file)
          (url-copy-file
           (format "http://www.india.gov.in/calendar/%d/export.ics" year)
           file))
        (with-temp-buffer
          (require 'icalendar)
          (insert-file-contents file)
          ;; (set-buffer-file-coding-system 'unix)
          (goto-char (point-min))
          (seq-keep
           (lambda (x)
             (when-let* ((description (icalendar--get-event-property x 'DESCRIPTION))
                         ((seq-find (lambda (h) (let (case-fold-search)
                                                    (string-match-p h description)))
                                      vz/indian-holidays)))
               (let ((time (icalendar--decode-isodatetime (icalendar--get-event-property x 'DTSTART))))
                 (list 'holiday-fixed (decoded-time-month time) (decoded-time-day time)
                       (string-trim-right description " +([GR])")))))
           (icalendar--all-events (icalendar--read-element nil nil))))))

It has been at least a couple years since I wrote this function so I
don't remember the hurdles I faced when I wrote it very well but it
perplexed me that there were no "public" functions that were geared
towards parsing an ical file.  In retrospect, it would have been less
confusing if these functions were "public" functions.  AFAIR, and this
very well might not be true, the structure returned
icalendar--read-element is not really documented anywhere which further
added to the confusion.

That said, this function has been rock solid ever since I wrote it.
Going by the ical files I have, it is working just fine since 2021.

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Hi Visuwesh,

Visuwesh <visuweshm@gmail.com> writes:

> It has been at least a couple years since I wrote this function so I
> don't remember the hurdles I faced when I wrote it very well but it
> perplexed me that there were no "public" functions that were geared
> towards parsing an ical file.  In retrospect, it would have been less
> confusing if these functions were "public" functions.  AFAIR, and this
> very well might not be true, the structure returned
> icalendar--read-element is not really documented anywhere which further
> added to the confusion.

Yes, I had the same feeling. I spent a while looking through
icalendar.el about six months ago, hoping I could find an appropriate
point to hook into it/extend it. But the private function names and lack
of documentation of the data structures eventually caused me to give up.
I couldn't figure out a good way to use it or patch it to do what I
wanted, and it seemed like there were a lot of things it didn't
implement anyway...thus this current project ;)

> That said, this function has been rock solid ever since I wrote it.
> Going by the ical files I have, it is working just fine since 2021.

Are you saying it would be important to you to preserve the "private"
API of icalendar.el? or just that you're glad this API is still working? 

Best,
Richard

Re: Improving Emacs' iCalendar support

[Visuwesh]

[புதன் அக்டோபர் 23, 2024] Richard Lawrence wrote:

> Hi Visuwesh,
>
> Visuwesh <visuweshm@gmail.com> writes:
>
>> It has been at least a couple years since I wrote this function so I
>> don't remember the hurdles I faced when I wrote it very well but it
>> perplexed me that there were no "public" functions that were geared
>> towards parsing an ical file.  In retrospect, it would have been less
>> confusing if these functions were "public" functions.  AFAIR, and this
>> very well might not be true, the structure returned
>> icalendar--read-element is not really documented anywhere which further
>> added to the confusion.
>
> Yes, I had the same feeling. I spent a while looking through
> icalendar.el about six months ago, hoping I could find an appropriate
> point to hook into it/extend it. But the private function names and lack
> of documentation of the data structures eventually caused me to give up.
> I couldn't figure out a good way to use it or patch it to do what I
> wanted, and it seemed like there were a lot of things it didn't
> implement anyway...thus this current project ;)

Understood.  pp is the reason why that function exists.  It look a bit
of trial and error to write it.

>> That said, this function has been rock solid ever since I wrote it.
>> Going by the ical files I have, it is working just fine since 2021.
>
> Are you saying it would be important to you to preserve the "private"
> API of icalendar.el? or just that you're glad this API is still working? 

The latter.  It would be nice if the former could be achieved too but
that is not a hard requirement.  It wouldn't be too much of a hurdle if
the new library is well documented.

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> It would be great to have these features supported better. In think
> some limitations in for example org-mode come also into play here as
> last time I looked it can't deal with timezones.

Org can deal with timezones in a limited way, during export.
More generic support is indeed not available yet.
The future plan for time zone support has been discussed in
https://list.orgmode.org/orgmode/CAL9aZkusXaNVCu7+nO-ic8SHhwVdOiXYtDyF=Gz7h+NoLaOZXQ@mail.gmail.com/

> For example Gnus uses org-capture to convert an icalendar
> event to org instead of some form of native API to synchronize the
> invite with-org.

Sorry, but I do not understand how org-capture has anything to do with
conversion. Capture is just for capturing user text to be saved into Org
file, based on a provided template. It is basically an extended
remember.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Björn Bidar]

Ihor Radchenko <yantar92@posteo.net> writes:

>> For example Gnus uses org-capture to convert an icalendar
>> event to org instead of some form of native API to synchronize the
>> invite with-org.
>
> Sorry, but I do not understand how org-capture has anything to do with
> conversion. Capture is just for capturing user text to be saved into Org
> file, based on a provided template. It is basically an extended
> remember.el.

Gnus uses org-capture instead of somekind ical->org conversion when
importing ical invites which limits the amount and quality of conversion to org it can
do in my opinion.

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

>> Sorry, but I do not understand how org-capture has anything to do with
>> conversion. Capture is just for capturing user text to be saved into Org
>> file, based on a provided template. It is basically an extended
>> remember.el.
>
> Gnus uses org-capture instead of somekind ical->org conversion when
> importing ical invites which limits the amount and quality of conversion to org it can
> do in my opinion.

Then, it does not look like anything can be done on Org side.

I have no idea about Gnus internals, but abusing org-capture for
conversion seems like a strange design.

Maybe better CC Gnus maintainers?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Richard Lawrence]

Dear emacs-devel,

Richard Lawrence <rwl@recursewithless.net> writes:

> I would like to start a discussion about improving Emacs' iCalendar
> support, beyond what is already available in icalendar.el. I personally
> would like to see Emacs gain a more full-fledged RFC5545 implementation
> that is primarily designed as a library for other applications to use.

Time for an update about this: I've been working steadily on the parser,
and I reached an important milestone this week: the parser can now parse
and reproduce all the examples in RFC5545, as well as my own calendar
data! There is still a lot of work to do, but this seems like a good
moment to ask again for feedback from anyone who's interested.

The repository where I've been doing this work is here:
https://sr.ht/~wyleyr/icalendar-el/

There are some instructions in the README for how to get started playing
around with the parser, for those interested in trying it out.

Based on the feedback I got earlier, I've made a few important design
decisions at this stage:

1) The basic form of the syntax tree is based on the format of the Org
parse tree in org-element-ast.el (though for now I have not implemented
any of the performance optimizations). Syntax nodes are a four-element
list of a type symbol, a metadata structure, a value, and a list of
children. Type symbols are used to store metadata about all the
iCalendar grammar categories and to do type-based dispatch in the
parser.

2) As far as possible, I have represented iCalendar data using "native"
data types in Emacs: dates are represented as calendar.el dates,
date-times as decoded-times, text as strings or buffer regions, etc.
This should hopefully make it as easy as possible to use iCalendar data
and parse trees from elsewhere in Emacs. In cases where no data type
already exists, the data types I defined are usually simple lists with
accessors defined for different fields.

(My initial attempts used cl-structs for the data types, following Adam
Porter's suggestion, but I quickly found myself writing tons of
boilerplate to convert between these structs and the "native" formats,
so I abandoned it. I am using a struct for syntactic metadata stored by
the parser, though.)

3) I have focused a lot on making good documentation, since I think
Emacs' documentation system is one of its best features, and the lack of
documentation in icalendar.el is one of the things that makes it
difficult to work with. All the symbols in the library should have good
documentation available via M-x describe-symbol -- please let me know if
you find anything confusing or broken in these docs.

Many thanks in advance for your thoughts!

Best,
Richard

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Richard Lawrence <rwl@recursewithless.net>
> Date: Sat, 23 Nov 2024 09:44:47 +0100
> 
> 1) The basic form of the syntax tree is based on the format of the Org
> parse tree in org-element-ast.el (though for now I have not implemented
> any of the performance optimizations). Syntax nodes are a four-element
> list of a type symbol, a metadata structure, a value, and a list of
> children. Type symbols are used to store metadata about all the
> iCalendar grammar categories and to do type-based dispatch in the
> parser.

Is it possible to use peg.el instead?  Since iCalendar is not tightly
coupled to Org, it would be best not to load Org for supporting
iCalendar features.

Thanks.

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> ... it would be best not to load Org for supporting
> iCalendar features.

AFAIU, it is a single library from Org that should be loaded. Nothing
more. It is no different than implementing data structure in some other way.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Richard Lawrence <rwl@recursewithless.net>, emacs-devel@gnu.org
> Date: Sun, 24 Nov 2024 10:21:44 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > ... it would be best not to load Org for supporting
> > iCalendar features.
> 
> AFAIU, it is a single library from Org that should be loaded. Nothing
> more. It is no different than implementing data structure in some other way.

Then why is org-element-ast.el part of Org, and not a general-purpose
library in lisp/emacs-lisp/, for example?

Re: Improving Emacs' iCalendar support

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Then why is org-element-ast.el part of Org, and not a general-purpose
> library in lisp/emacs-lisp/, for example?

I planned to upstream it eventually. It is a relatively new library,
part of the effort to modularize Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Improving Emacs' iCalendar support

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rwl@recursewithless.net, emacs-devel@gnu.org
> Date: Sun, 24 Nov 2024 10:58:23 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then why is org-element-ast.el part of Org, and not a general-purpose
> > library in lisp/emacs-lisp/, for example?
> 
> I planned to upstream it eventually. It is a relatively new library,
> part of the effort to modularize Org.

OK, then let's move it and rename it soon, so that this work on
iCalendar could be based on the new name.  (I guess renaming will also
need to change the prefix of the symbols defined by the package?)

Upstreaming org-element-ast (was: Improving Emacs' iCalendar support)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I planned to upstream it eventually. It is a relatively new library,
>> part of the effort to modularize Org.
>
> OK, then let's move it and rename it soon, so that this work on
> iCalendar could be based on the new name.  (I guess renaming will also
> need to change the prefix of the symbols defined by the package?)

The major issue with renaming is how to do it and yet preserve the
ability to use the library from Org distributed via ELPA - in older
Emacs versions. I suspect that the right way to go will be creating a
separate ELPA package in addition to moving and renaming the library.

Also, there are certain things in the library that are specific to
Org:

1. org-element--standard-properties constant
2. commentary that is mostly talking about Org

They do not prevent usage outside Org mode, but should probably be
changed before upstreaming.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Upstreaming org-element-ast (was: Improving Emacs' iCalendar support)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rwl@recursewithless.net, emacs-devel@gnu.org
> Date: Sun, 24 Nov 2024 11:21:38 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I planned to upstream it eventually. It is a relatively new library,
> >> part of the effort to modularize Org.
> >
> > OK, then let's move it and rename it soon, so that this work on
> > iCalendar could be based on the new name.  (I guess renaming will also
> > need to change the prefix of the symbols defined by the package?)
> 
> The major issue with renaming is how to do it and yet preserve the
> ability to use the library from Org distributed via ELPA - in older
> Emacs versions. I suspect that the right way to go will be creating a
> separate ELPA package in addition to moving and renaming the library.

Something like that, yes.  An Org-specific compatibility package is
also a possibility.

> Also, there are certain things in the library that are specific to
> Org:
> 
> 1. org-element--standard-properties constant
> 2. commentary that is mostly talking about Org
> 
> They do not prevent usage outside Org mode, but should probably be
> changed before upstreaming.

Right.

My point is that the sooner we do this (quite unpleasant) job, the
smaller it is, and the smaller are the effects on other packages which
use it.

Thanks.