org-mode functional programming library

org-mode functional programming library

[Dwarshuis, Nathan J]

Hello,

I recently authored an package called "om.el" which is a functional org-mode API akin to dash.el primarily using org-element. Briefly, it provides a library of (mostly) pure functions that manipulate the parse tree generated by org-element.el, and uses this to either edit or query the buffer with all the advantages of functional programming (eg lack of side effects, referential transparency, easier testing, etc). The github repo for om.el is here: https://github.com/ndwarshuis/om.el.

I'm posting to the mailing list a) for general feedback on this package and b) because I am wondering if this would be a good package to include with org-mode itself rather than in another repository such as MELPA. The code for om.el is tightly integrated with org-element.el and it might make sense for development between these to be closely intertwined.

There is also an open submission for this to MELPA and the discussion is here: https://github.com/melpa/melpa/pull/6623.

Thank you,
Nathan Dwarshuis

Re: org-mode functional programming library

[Tim Visher]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

On Mon, Jan 20, 2020 at 6:48 PM Dwarshuis, Nathan J <ndwar@yavin4.ch> wrote:

> I recently authored an package called "om.el" which is a functional
> org-mode API akin to dash.el primarily using org-element. Briefly, it
> provides a library of (mostly) pure functions that manipulate the parse
> tree generated by org-element.el, and uses this to either edit or query the
> buffer with all the advantages of functional programming (eg lack of side
> effects, referential transparency, easier testing, etc). The github repo
> for om.el is here: https://github.com/ndwarshuis/om.el.
>
> I'm posting to the mailing list a) for general feedback on this package
> and b) because I am wondering if this would be a good package to include
> with org-mode itself rather than in another repository such as MELPA. The
> code for om.el is tightly integrated with org-element.el and it might make
> sense for development between these to be closely intertwined.
>
> There is also an open submission for this to MELPA and the discussion is
> here: https://github.com/melpa/melpa/pull/6623.
>

Oooo this looks cool. I have a simmering project to have an Evernote like
system built on top of org mode and a set of directories where I've done a
lot of work like this. I'll have to see if I can work this into that and if
I can I'll be sure to open up issues/try to contribute patches etc.

[-- Attachment #2: Type: text/html, Size: 1879 bytes --]

Re: org-mode functional programming library

[Nicolas Goaziou]

Hello,

"Dwarshuis, Nathan J" <ndwar@yavin4.ch> writes:

> I recently authored an package called "om.el" which is a functional
> org-mode API akin to dash.el primarily using org-element. Briefly, it
> provides a library of (mostly) pure functions that manipulate the
> parse tree generated by org-element.el, and uses this to either edit
> or query the buffer with all the advantages of functional programming
> (eg lack of side effects, referential transparency, easier testing,
> etc). The github repo for om.el is here:
> https://github.com/ndwarshuis/om.el.
>
> I'm posting to the mailing list a) for general feedback on this
> package and b) because I am wondering if this would be a good package
> to include with org-mode itself rather than in another repository such
> as MELPA. The code for om.el is tightly integrated with org-element.el
> and it might make sense for development between these to be closely
> intertwined.

Thank you for this thorough work.

Note that code going into Org proper cannot rely on external libraries,
e.g., "s.el" or "dash.el". So it may make sense to integrate it, but not
in its current form. Is it possible to write it without these libraries,
and without re-inventing the wheel? Note that, at some point, Org will
support "seq.el", i.e., when we drop support for Emacs 24.

Skimming through your code, I read a lot of griefs against Element
library (inconvenient, unfortunate, buggy, inconsistent... I stopped
there). I agree on most points, of course. Though, there are a few cases
where you seem to miss the point. Also, the way you handle plain lists
is not how it is done in Org. Anyway, it could be beneficial for both
Org and your library to discuss the points above and improve the parser.
WDYT?

I didn't check, but how do you alter the buffer when applying changes to
the parse tree? Is it optimized, e.g., only changed lines are replaced?
Or are you deleting the whole thing and replacing it with the
interpreted stuff? Note that Org has hardly ever access to the full
parse-tree, because parsing a whole buffer is too slow. So, because of
these limitations, I wonder if your library can be used efficiently to
alter the buffer.

Regards,

-- 
Nicolas Goaziou

Re: org-mode functional programming library

[Dwarshuis, Nathan J]

Hello Nicolas, thank you for the reply

> Note that code going into Org proper cannot rely on external libraries,
> e.g., "s.el" or "dash.el". So it may make sense to integrate it, but not
> in its current form. Is it possible to write it without these libraries,
> and without re-inventing the wheel? Note that, at some point, Org will
> support "seq.el", i.e., when we drop support for Emacs 24.

It would be possible but would obviously require some work but would be doable. I will say that there's a macro I am hoping to add to dash that implements &rest and &key simultaneously (impossible with 'cl-defun') so if this package doesn't depend on dash that might be reinventing the wheel (assuming it gets in).

> Skimming through your code, I read a lot of griefs against Element
> library (inconvenient, unfortunate, buggy, inconsistent... I stopped
> there). I agree on most points, of course. Though, there are a few cases
> where you seem to miss the point. Also, the way you handle plain lists
> is not how it is done in Org. Anyway, it could be beneficial for both
> Org and your library to discuss the points above and improve the parser.
> WDYT?

My apologies if this came across as rude; I can be quite blunt in my comments. Anyways, I would definitely like to discuss further how we can improve both. Do you mind elaborating on how plain-lists are handled differently between Org and this library. Also, just so we are on the same page, where do I seem to miss the point?

> I didn't check, but how do you alter the buffer when applying changes to
> the parse tree? Is it optimized, e.g., only changed lines are replaced?
> Or are you deleting the whole thing and replacing it with the
> interpreted stuff? Note that Org has hardly ever access to the full
> parse-tree, because parsing a whole buffer is too slow. So, because of
> these limitations, I wonder if your library can be used efficiently to
> alter the buffer.

This is actually a place where I would like feedback. The function you are probably looking for is 'om-update' (and related) which parses all/part of the buffer to a parse tree, transforms it in some way, then writes it back. There is also 'om-insert' which just inserts a parse tree created from scratch.

I wouldn't call it optimized, but I designed the update functions to parse as little as possible to only get the relevant parse tree. Eg if you only care about one headline, there is the function 'om-update-headline' which parses only within the boundaries of that headline. That being said, the limitations of this are that it doesn't get the :parent property (unless you actually parse the parent as well) and it replaces the entire parse tree (as opposed to only replacing the lines that are changed) which automatically deletes the overlays that allow folding. To deal with the latter I have 'om-fold' and 'om-unfold' but these aren't the most convenient and it would be much easier all-around if the write-back operation only replaced what changed (currently on my todo list).

I also wonder if there is a way to control what is being parsed, as some elements can be fully represented with only a subset of their properties.


________________________________
From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
Sent: 2020-02-01T08:53:43-0500
To: Dwarshuis, Nathan J
Subject: org-mode functional programming library
Hello,

"Dwarshuis, Nathan J" <ndwar@yavin4.ch> writes:

> I recently authored an package called "om.el" which is a functional
> org-mode API akin to dash.el primarily using org-element. Briefly, it
> provides a library of (mostly) pure functions that manipulate the
> parse tree generated by org-element.el, and uses this to either edit
> or query the buffer with all the advantages of functional programming
> (eg lack of side effects, referential transparency, easier testing,
> etc). The github repo for om.el is here:
> https://github.com/ndwarshuis/om.el.
>
> I'm posting to the mailing list a) for general feedback on this
> package and b) because I am wondering if this would be a good package
> to include with org-mode itself rather than in another repository such
> as MELPA. The code for om.el is tightly integrated with org-element.el
> and it might make sense for development between these to be closely
> intertwined.

Thank you for this thorough work.

Note that code going into Org proper cannot rely on external libraries,
e.g., "s.el" or "dash.el". So it may make sense to integrate it, but not
in its current form. Is it possible to write it without these libraries,
and without re-inventing the wheel? Note that, at some point, Org will
support "seq.el", i.e., when we drop support for Emacs 24.

Skimming through your code, I read a lot of griefs against Element
library (inconvenient, unfortunate, buggy, inconsistent... I stopped
there). I agree on most points, of course. Though, there are a few cases
where you seem to miss the point. Also, the way you handle plain lists
is not how it is done in Org. Anyway, it could be beneficial for both
Org and your library to discuss the points above and improve the parser.
WDYT?

I didn't check, but how do you alter the buffer when applying changes to
the parse tree? Is it optimized, e.g., only changed lines are replaced?
Or are you deleting the whole thing and replacing it with the
interpreted stuff? Note that Org has hardly ever access to the full
parse-tree, because parsing a whole buffer is too slow. So, because of
these limitations, I wonder if your library can be used efficiently to
alter the buffer.

Regards,

Re: org-mode functional programming library

[Adam Porter]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Note that, at some point, Org will support "seq.el", i.e., when we
> drop support for Emacs 24.

Just a small FYI about seq.el, for those who may not be aware: while
it's a very useful library, it can be quite slow since it uses generics.
For example, here are some benchmarks comparing seq-intersection with
other functions that intersect lists:

https://gist.github.com/alphapapa/36b117c178dec677258f372c3a01d8b5

Note the last benchmark listed, which shows that cl-intersection is
about 2x as fast as seq-intersection.  As well, dash.el's -intersection
is about 17x faster than seq-intersection.

So while seq.el will undoubtedly be useful in Org, it should be used
carefully with regard to performance.  Type-specific functions will
generally be much faster.  And as long as dash.el can't be used in Org
proper, a custom implementation may be called for at times.

Re: org-mode functional programming library

[Bastien]

Hello Nathan,

this looks very interesting and promising, thanks for this work.

I hope you can work with Nicolas to spot possible enhancements to
your own library or to org-element.el, if that's relevant.

I will reference om.el (or org-ml.el?) on Worg, but if you want
to do it yourself, please go ahead:

https://orgmode.org/worg/org-contribute.html

Thanks!

-- 
 Bastien

Re: org-mode functional programming library

[Dwarshuis, Nathan J]

Hi Bastien,

Thank you for the feedback. I don't mind writing something up myself. Under what section should this be submitted ("contributed packages" or "additions under development")?

Thank you,
Nate

________________________________
From: Bastien <bzg@gnu.org>
Sent: 2020-02-11T03:13:48-0500
To: Dwarshuis, Nathan J
Subject: org-mode functional programming library
Hello Nathan,

this looks very interesting and promising, thanks for this work.

I hope you can work with Nicolas to spot possible enhancements to
your own library or to org-element.el, if that's relevant.

I will reference om.el (or org-ml.el?) on Worg, but if you want
to do it yourself, please go ahead:

https://orgmode.org/worg/org-contribute.html

Thanks!

Re: org-mode functional programming library

[Bastien]

Hi Nate,

"Dwarshuis, Nathan J" <ndwar@yavin4.ch> writes:

> Thank you for the feedback. I don't mind writing something up
> myself. Under what section should this be submitted ("contributed
> packages" or "additions under development")?

I would say *both* to give it as much visibility as possible.

Also, beware that I plan to make a big revamping of Worg's contents,
precisely in order to make it easier to answer such questions.

Thanks!

-- 
 Bastien

Re: org-mode functional programming library

[Dwarshuis, Nathan J]

Hi Bastien,

It has been a while but I renamed this library (now org-ml instead of om.el) and updated it to version 2.0.0 with new features and bug fixes.

I would like to add something on worg about it as previously discussed but I don't see how to add myself as a user and upload my public key (looking here: https://code.orgmode.org/explore/users). Do I need to be added by someone else?

Thank you,
Nate

________________________________
From: Bastien <bzg@gnu.org>
Sent: 2020-02-12T12:35:51-0500
To: Dwarshuis, Nathan J
Subject: org-mode functional programming library
Hi Nate,

"Dwarshuis, Nathan J" <ndwar@yavin4.ch> writes:

> Thank you for the feedback. I don't mind writing something up
> myself. Under what section should this be submitted ("contributed
> packages" or "additions under development")?

I would say *both* to give it as much visibility as possible.

Also, beware that I plan to make a big revamping of Worg's contents,
precisely in order to make it easier to answer such questions.

Thanks!

Re: org-mode functional programming library

[Bastien]

Hi,

"Dwarshuis, Nathan J" <ndwar@yavin4.ch> writes:

> It has been a while but I renamed this library (now org-ml instead of
> om.el) and updated it to version 2.0.0 with new features and bug
> fixes.

Thanks for the update.

> I would like to add something on worg about it as previously discussed
> but I don't see how to add myself as a user and upload my public key
> (looking here: https://code.orgmode.org/explore/users). Do I need to
> be added by someone else?

Yes, please send me the username you want in private and I will add it
and give you further instructions.

Thanks in advance!

-- 
 Bastien