Org-Mode as DSL

Org-Mode as DSL

[Mauro Mandracchia]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

Hi everybody 👋
This is my first message to the community, let me first state, I love
*org-mode.*Thanks to the authors and the contributors!
I use *org-mode* everyday, but I don't want to bore you with my stories
about how *org-mode* facilitates my life, just ask if you are interested.

I believe that *org-mode* is the most powerful toolset that can leverage
documentation and code.
I also love literate programming and I've found nothing like org-mode for
it.

I think that org-mode adoption is mainly found in academic and personal
context and rarely in commercial organizations.
Too often I see myself being forced to use Markdown for project
documentation and miss all the features that org-mode could have offered to
the entire project and not just the documentation.

The main reason why I can't use org-mode at work is that my
colleagues don't all use
*emacs.*
What I miss is an DSL for Org-Mode, like HTML does, and that other editors
or IDE would implement his functions, a bit like the browsers do.

If org-mode could be embeddable in different runtimes I could imagine HTTP
services implemented for *org-mode*, and I think of tools like Jupyter
<https://jupyter.org/> or Observable <https://observablehq.com/> could be
built on the org-mode standards.
I have many other implementation details and use cases that I would like to
share, but can be a bit too much for introducing the topic.

Is it a crazy idea if Org-Mode could become a DSL rather than a mode for
emacs?

Best Regards,
Mauro.

[-- Attachment #2: Type: text/html, Size: 1802 bytes --]

Re: Org-Mode as DSL

[Tom Gillespie]

Hi Mauro,
    Welcome! This is definitely not a crazy idea (or at least if it is
then you are in good company).
Org already has wide support as a markup language, and tools like
https://github.com/tecosaur/org-pandoc-import exist that leverage
this,
but the syntax is not standardized and as a result the elisp
implementation is the defacto standard.
Depending on what editors your colleagues use and the extent to which
features of Org they need to
use there might be an org-mode plugin that meets their needs (editing
Org as markup yes, using Org babel, no).
See the In Other Editors section of https://orgmode.org/install.html#org70be9a1.

Beyond its use as a markup language, there are efforts toward making
the more powerful parts of Org portable so that runtimes
beyond Emacs can correctly interpret and enable the dynamic nature of Org.

See for example this thread on formalizing org syntax
https://lists.gnu.org/archive/html/emacs-orgmode/2020-10/msg00344.html.
Parsers for org already exist in a variety of languages, but they vary
in their correctness and their support for org features.
In part this is because some of those features (such as footnotes)
look like they are easy to implement, but support syntactic variants
that parsers do not implement correctly or at all (e.g. the org-ruby
parser). Formalizing what it means to be a compliant parser could
help in those cases. There is also a need to specify some additional
underlying semantics about how to correctly construct trees
from the parsed elements and how to associate keywords to elements
etc. I am writing up an executable grammar for Org using Racket
and #lang brag that I will share in the mentioned thread.

Beyond surface syntax and basic semantics I see two key areas that
need attention in order for other full org implementations to be
viable.

The first obstacle is the fact that the semantics of org babel source
blocks are inconsistent and that it is hard to know what features a
particular ob-lang supports.
See https://lists.gnu.org/archive/html/emacs-orgmode/2020-10/msg00244.html
for a discussion of some of the challenges here.
I have been drafting an email on babel regularization for quite a
while, and it will show up on this list in the near future.

The second obstacle that I see is the fact that there are many places
where Org is deeply integrated with Emacs Lisp.
For example it is possible to include raw Emacs Lisp expressions in
certain areas of an Org file that will be evaluated
at a certain time (the time is not the same in all cases).
Supporting this functionality is something that other implementations
must be able to do
if we do not want Org files to be split into those that use elisp and
those that do not (somewhat like the current sorry state of markdown).
I have a couple of thoughts about how this could be addressed, but in
the absence of a portable implementation of elisp there would
be a divide between Org files that use elisp closures outside of src
blocks and those that do not.
In the event that other languages could someday be used in the top
level of an org file, it would be easier for the
reference elisp implementation of Org mode to run such files by
leveraging the ob-lang implementation for those languages.

A third area (of less immediate concern to me) that might need
standardization is the export backends.
Should different implementations of Org produce the same latex output?
Probably. How much trouble
would it be to specify? Probably quite a bit. So it might be easier to
just run Emacs behind the scenes
in those cases and not worry about trying to standardize.

Best!

Tom

Re: Org-Mode as DSL

[Eric S Fraga]

On Thursday, 29 Oct 2020 at 22:12, Mauro Mandracchia wrote:
> Is it a crazy idea if Org-Mode could become a DSL rather than a mode for
> emacs?

It's not a crazy idea but one which misses one of the best features of
org mode: it is part of Emacs.  For me, a markup language is not so
exciting: we have plenty of them.  What makes org mode powerful is that
it is infinitely customizable by being part of Emacs.  I can add code,
change existing code, advice code, etc.  The org I use is not the org
others use, as a result.

But I'm sure many would love to have org markup understood in many more
tools.

My own approach is to ensure I have Emacs with me wherever I go.  Hence
my obsession with palmtop computers... ;-)

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: Org-Mode as DSL

[TEC]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]


Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> It's not a crazy idea but one which misses one of the best 
> features of
> org mode: it is part of Emacs.  For me, a markup language is not 
> so
> exciting: we have plenty of them.  What makes org mode powerful 
> is that
> it is infinitely customizable by being part of Emacs.  I can add 
> code,
> change existing code, advice code, etc.  The org I use is not 
> the org
> others use, as a result.

I have wondered whether it might be viable to create a LSP client 
by
putting Emacs /inside/ the LSP client. Checking ~ls -lh
/usr/bin/emacs-nox~ I am told that my terminal-only Emacs client 
is 5.3
MiB. This seems nice and small.

Hence, any and all concerns about feature parity etc. are 
completely
resolved. One 'just' needs to implement the bindings and piping 
(as
opposed to the whole shebang).

Maybe this is the real 'crazy idea' in this thread :P
Hopefully it's of some interest :)

All the best,
Timothy.


[-- Attachment #2: fun_emacs_lsp_sketch.png --]
[-- Type: image/png, Size: 73171 bytes --]

Re: Org-Mode as DSL

[Lejon]

TEC <tecosaur@gmail.com> writes:

>
> Hence, any and all concerns about feature parity etc. are completely
> resolved. One 'just' needs to implement the bindings and piping (as
> opposed to the whole shebang).
>

Something that I am missing when hacking on org-mode is some form of api
reference. Things like 'org-map-entries' or 'org-agenda-get-deadlines'
can be very useful for when you want to extend org's functionality, or
use org's functionality in a different way, but can be hard to find.

In order to implement this "Emacs as org-mode lsp server" idea I think org
would need a stable (for some measure of stable) api, which I think
would benefit org anyways. I think it would be a good idea at least to
put some some effort into gathering the useful functions and put that
list somewhere in the manual for example.

Regards,

Lejon

Re: Org-Mode as DSL

[Mauro Mandracchia]

[-- Attachment #1: Type: text/plain, Size: 3663 bytes --]

Thanks for the warm welcome and the amount of inputs and details that
everybody so fast provides.
I'm really impressed by this community.

Ok seems that delivering a language specification in EBNF has some
challenges that can hardly be addressed due to the nature of org-mode
design, but I read around that often EBNF is not the silver bullet to solve
a grammar representation although it is a de-facto standard.

So I will try to put some orders in my thoughts, when I look at
functionalities of org-mode I will split them in the following categories:

   - Grammar Specification
   - Metadata
      - TODO
      - Code Block

      - Tangling (or waving for the web people)
   - Properties of the file
      - Templating

      - Block Execution
   - Environment (Global variables such as PATH)
      - Language
      - Context for arguments

      - Publishing Functions
      - Target Format
      - Custom Parser

I share a lot with of your ideas and inputs, here some observations and
options that I noticed:

   - Indeed emacs is not that big and could run as a background process to
   deliver org-mode features to other systems.
   I do something very similar to *TEC* for my CI/CD and my static
   generated website, but I'm not sure I would run emacs as a web service.

   - LSP doesn't mention org-mode as available language, but it does for
   markdown
   https://microsoft.github.io/language-server-protocol/specifications/specification-current/#textDocumentItem
   maybe because org-mode is not a language?

   - I share the sentiment that some api are a bit obscure and most
   important there isn't for instance an official way to say "this is how you
   create extensions in org".

   - I found that multiple people are already converting org-mode into json
   by using org*-map-entries *which indeed is not very publicly shown (I
   miss something like "this is how you create extensions in org").

   - I relate a lot to the issues that *Tom Gillespie* discusses in his
   post (
   https://lists.gnu.org/archive/html/emacs-orgmode/2020-10/msg00244.html)
   about tangling and executions and I loved to see your efforts in trying to
   solving them.

   - Since gccemacs elisp could compile to C (not sure this is relevant,
   but can be investigated if there are opportunities to be capable to have
   org-mode embendable).


For my personal needs I think I would pragmatically start toying around
with a org-mode to json like here https://github.com/ludios/org-to-json
And once I gain some confidence on the output JSON, I think I will gain
more confidence about the grammar definition.

I think that converting org-mode to JSON could show org-mode as an Abstract
Syntax Tree, I think from there it should be easier to create a
specification and maybe is what already *orgajs
<https://github.com/orgapp/orgajs>* already did.
I believe creating functions like tangle and execution shouldn't be that
trivial from an AST.

Concluding,
there is overlap between the Interfaces of LSP and org-mode which makes a
lot of sense since both target the interaction for with a document eg:

   - Propose actions, these can be:
      - Execute
      - Tangle
      - Change state TODO
      - Insert date, tag, etc...
   - Go to definition (For linking sections)
   - Support multi language in the same document

This makes me think that creating an org-mode LSP is not that crazy idea
after all, what do you think?
If you would build an LSP for org-mode which setup would you choose? What
do you think would be hard to implement?

Again many thanks 🙏
Mauro.

[-- Attachment #2: Type: text/html, Size: 4244 bytes --]