Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Dear Fellow Orgers,

The recent spike of discussions following Karl's presentation in
Emacsconf 2021 revealed a lot of controversy among Org and Emacs
enthusiasts. Yet, Karl named a number of very real problems surrounding
Org mode usage outside Emacs.

From the narrow perspective of this mailing list, I would like to list
some of the problems and possible solutions to them on our (Org dev)
side.

1. Org mode is almost impossible to separate from Emacs in its full strength
   - Yet, a number of people seems to be interested in using Org mode
     outside Emacs
     + Most notably, mobile users
     + A number of websites, like Github/Gitlab
   - The existing interest gave a rise to a number of third-party
     Org syntax parsers
     + None of the parsers support all the Org features, and even not
      all the grammar!
     + The parsers often do not even try to support all the features.
       They are merely looking at Org as a lightweight markup format.
       
2. Despite user interest, we lack a clear definition of Org grammar with
   examples and concrete guidelines for third-party parser developers

3. Many elements of the grammar are excessive for simple cases not
   involving document export, babel, and other powerful Org mode
   features

4. "Org mode" is an ambiguous word combination for search engines and
   people may not be able to find relevant information.
   - This one is not 100% true from my quick search. Try the following
     links:
     + https://duckduckgo.com/?q=org-mode&ia=web
     + https://duckduckgo.com/?q=org+mode&ia=web
     + https://duckduckgo.com/?q=org+mode+syntax&ia=web
     + https://duckduckgo.com/?q=org+mode+markup&ia=web
     The results are extremely relevant to Org, though orgmode.org
     search result looks slightly confusing (more below).
          
------------------------------------------------------
| My suggestions how we can address the above points |
------------------------------------------------------

1. Despite webengines delivering fairly good results for "Org mode"
   search term, I am a bit concerned about the first search hit, which
   is our flagship "https://orgmode.org" website.

   The website title is "Org mode for Emacs", repelling users who _do
   not want_ to use Org inside Emacs. Maybe we can do better? Something
   with less accent on Emacs like "Org mode: your life in plain text"

   The "abstract" in the search result is also not fully relevant:
   > Org and Org-mode have so many use cases that it is simply not
   > possible to easily document them, let alone show them all off on a
   > single page. As a result, Worg serves as a community wiki and
   > provides a place to document and share information about all aspects
   > of using and working with Org. For example, Worg contains:
   Again, we can make a simple change revealing the paragraph shown the
   at our front page:
   > Org is a highly flexible structured plain text file format,
   > composed of a few simple, yet versatile, structures — constructed
   > to be both simple enough for the novice and powerful enough for the
   > expert. 
   > 
   > Org mode is also a GNU Emacs major mode for keeping notes,
   > authoring documents, computational notebooks, literate programming,
   > maintaining to-do lists, planning projects, and more — in a fast
   > and effective plain text system.

2. Our front pages gives an impression that user must install Org
   I refer to the big image links "Features Install Quickstart Contribute"

   Maybe we can add "Try in browser" linking to our own instance of
   https://organice.200ok.ch/sample

3. We can provide a "source of truth" for Org syntax for third-party
   parser developers. Something easily reachable from the front page:
   "Org-Mode Logo Org Mode
   Features
   Releases
   ...
   --> Add Org support in third-party apps"

   The page should give a nice summary of existing third-party
   libraries, official _technical_ Org syntax, and tools for developers.

   3.1. In particular, I suggest to link
        https://orgmode.org/worg/dev/org-syntax.html (it will be ready
        eventually)
   
   3.2. Also, we may add a simplified Org syntax, as Karl suggested
        (similar to Basic and Extended syntax in
        https://www.markdownguide.org/, but more technical)
   
   3.3. I strongly suggest to add a community test set with example Org
        files. The files should be a source of tests for Org parsers
        with the true parsed representations in sexp format (possibly
        also converted to json).

        The example files can live in a separate repo for easy
        contributions (possibly with Github/Gitlab mirrors is someone is
        willing to maintain those).

        The example files will be used by Org mode itself in our test
        suite and will serve as a benchmark for external parser quality.

   3.4. Finally, we can have a separate page listing recommended
        features for editors interacting with Org files. Something like
        "implementation roadmap" (citing Timothy) for external devs.
        Again, unlike our existing feature page, this should be more
        technical and target developers.

        The features may include (we can add them as needed):
        - Folding / structural editing
        - Table editing / alignment
        - Source block execution
        - Babel
        - Export / publish
        - Setting TODO keywords
        - Agenda / searching in Org files
        - Clocking data
        - Capture
        - ...

WDYT?

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Ihor Radchenko writes:

> The website title is "Org mode for Emacs", repelling users who _do
>    not want_ to use Org inside Emacs. Maybe we can do better? Something
>    with less accent on Emacs like "Org mode: your life in plain text"

I am not at all in favor of separating the 'Org Mode' name from 'GNU Emacs'.
In any case, regardless of my opinion, I see here two problems:

1. "Org mode: your life in plain text". The word "mode" remains orphan:
mode of what? It is clear that it is an Emacs mode, therefore it doesn't
make much sense to hide Emacs and then suggest it.

2. One possibility: "Org: your life in plain text". But here what
remains orphaned is "Org", too generic name. Unless "Org Mode" is a
lexicalized construct, without reference to any emacs mode.

(In any case, I would be extremely saddened if the reference to GNU Emacs
disappears in the title, just to please a minority).

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> I am not at all in favor of separating the 'Org Mode' name from 'GNU
> Emacs'.

To clarify, I do not suggest to remove the linkage between Org mode and
GNU Emacs. Just change the emphasis. I had no intention to remove the
reference to Emacs from search result. It should be still there in the
short description of orgmode.org. See the second part of my suggestion
#1.

> 1. "Org mode: your life in plain text". The word "mode" remains orphan:
> mode of what? It is clear that it is an Emacs mode, therefore it doesn't
> make much sense to hide Emacs and then suggest it.
>
> 2. One possibility: "Org: your life in plain text". But here what
> remains orphaned is "Org", too generic name. Unless "Org Mode" is a
> lexicalized construct, without reference to any emacs mode.

I view "Org Mode" as a "brand name". Something uniquely identifying Org
mode and serving as a search term.

> (In any case, I would be extremely saddened if the reference to GNU Emacs
> disappears in the title, just to please a minority).

Is it your principal position about the title specifically? Do you think
that just referring to Emacs in the website description is not
sufficient?

Note that my suggestion #1 has nothing to do with actual orgmode.org
front page. Just about how it appears in search results.

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Ihor Radchenko writes:

> I view "Org Mode" as a "brand name". Something uniquely identifying Org
> mode and serving as a search term.

Yes, it makes sense.

> Is it your principal position about the title specifically? Do you think
> that just referring to Emacs in the website description is not
> sufficient?

> Note that my suggestion #1 has nothing to do with actual orgmode.org
> front page. Just about how it appears in search results.

Yes, sorry for not explaining myself well: I was also referring to
search results, not the title in the web site...

But the question is: what need is there to remove the reference to Emacs
in the search result? I think the emphasis is necessary. As we say in
spanish, it's like putting the bandage on before the wound. If there are
people who think that Org Mode can 'live' in some way outside of Emacs
(a respectable opinion, but that I do not share, at least 100%), I think
the burden of the work falls on them and not on us, the users of
Emacs-Org-Mode. Anyway it is my simple and subjetive opinion, and it is
not my intention to initiate a controversy with it.

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Heinz Tuechler]

Juan Manuel Macías wrote/hat geschrieben on/am 05.12.2021 12:08:
> Ihor Radchenko writes:
>
>> I view "Org Mode" as a "brand name". Something uniquely identifying Org
>> mode and serving as a search term.
>
> Yes, it makes sense.
>
>> Is it your principal position about the title specifically? Do you think
>> that just referring to Emacs in the website description is not
>> sufficient?
>
>> Note that my suggestion #1 has nothing to do with actual orgmode.org
>> front page. Just about how it appears in search results.
>
> Yes, sorry for not explaining myself well: I was also referring to
> search results, not the title in the web site...
>
> But the question is: what need is there to remove the reference to Emacs
> in the search result? I think the emphasis is necessary. As we say in
> spanish, it's like putting the bandage on before the wound. If there are
> people who think that Org Mode can 'live' in some way outside of Emacs
> (a respectable opinion, but that I do not share, at least 100%), I think
> the burden of the work falls on them and not on us, the users of
> Emacs-Org-Mode. Anyway it is my simple and subjetive opinion, and it is
> not my intention to initiate a controversy with it.
>
> Best regards,
>
> Juan Manuel
>
>

As a humble emacs and org-mode user I second this "simple and subjectiv
opinion".
Therefore I would first mention org as an emacs mode and secondly as a
file format.

best regards,

Heinz

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Yes, sorry for not explaining myself well: I was also referring to
> search results, not the title in the web site...
>
> But the question is: what need is there to remove the reference to Emacs
> in the search result? I think the emphasis is necessary. As we say in
> spanish, it's like putting the bandage on before the wound. If there are
> people who think that Org Mode can 'live' in some way outside of Emacs
> (a respectable opinion, but that I do not share, at least 100%), I think
> the burden of the work falls on them and not on us, the users of
> Emacs-Org-Mode. Anyway it is my simple and subjetive opinion, and it is
> not my intention to initiate a controversy with it.

Ok. Let me explain my thought process.

First of all, there is no burden on users of Org mode in making edits to
orgmode.org. It is a burden on Org contributors.

One of the aims of my proposal is reducing this burden by involving
non-emacs users to provide contributions to Org (e.g. by making more
tests for Org-element parser). To do it, we need to make the
contribution process for non-emacs developers easier. Ideally, without
too much effort on our side.

The idea of involving non-emacs users does have a potential because we
do know that third-party tools are already using Org. The problem is the
disconnect between those tools and Org mode proper.

The sources of the disconnect are (1) lack of technical "blueprints" for
Org that do not require knowing Elisp; (2) lack of discovereability of
Org mode as something that can live outside narrow field of Emacs. In
this branch of our discussion, I am going to talk about the second
point.

People simply do not expect to see a markup language when they encounter
a link with "Org mode for Emacs" title. Someone looking for Org mode
markup to be used in, say, websites will think that "Org mode for Emacs"
is limited to Emacs. Someone just interested in plain text markup will
find no relevance at all.

Title is important. If we care at all about orgmode.org website
appearing in search results, we want the title and the summary to have 2
main properties: (1) Provide search keywords to make it searchable by
potentially interested people; (2) Provide a title that immediately
signal that our website contains the information people are looking for.

Now, we need to understand what kind of people may be looking to
orgmode.org website.

1. Existing emacs users

   If a Emacs user is faced with "Org mode for Emacs", the word "Emacs"
   is indeed recognisable. On the other hand, the word "Org mode" does
   not provide much further info, except that it is a major (or maybe
   minor?) mode for "Org"??
   
   Now, consider "Org mode: your life in plain text".
   For emacs users, "Org mode" is not just a strange phrase, but a very
   clear indication that we are talking about Emacs.
   The "your life in plain text" provides extra information about what
   "Org mode" refers to. Clearly, text documents and "your life in plain
   text" should resonate with every Emacs user's soul.

   We can change the second variant of the title to contain "Emacs", but
   will it add much value? I am not convinced. On the other hand, making
   title too long or too complex _is_ bad. Long titles tend to be
   skipped (there was even formal research on this!)

2. Non-emacs users interested in plain text markup

   These users know nothing about Emacs and "Org mode" has no meaning
   for them as is. So, we do need something more descriptive.
   Adding "Emacs" may be fine, but it will serve no purpose for people
   not familiar with emacs. Just another unknown term making the title
   longer.

3. Non-emacs users interested in GTD/project management, etc
   "Org mode: your life in plain text" is somewhat relevant when people
   are looking to manage "life" (typically true for GTD enthusiasts).

   Though we can probably do better for this category.
   Maybe "Org mode: manage your life and notes in plain text"?
   Though it makes the title less relevant to group #2.

4. Researchers looking for ipython-like environment

   Not covered, except by reading my proposed site summary. I am not
   sure how we can improve title for this audience.

5. ??? (Suggestions are welcome)

Of course, better suggestions for the title are welcome. I just wanted
to make it clear the reasoning I do not like the current title and how
the proposed alternative is better (though not ideal).

Finally, I want to emphasise that our aim for search engines is not
advertising Emacs (we already do it by trapping users inside Org and
making them switch to Emacs by force :evil_laughter:). The aim is
encouraging people to use and contribute to Org mode in useful ways
(even unrelated to writing Elisp or, really, any code at all).

Search result is just an entrance for users to be curious about the new
beast of "Org mode". The website front page is the means to make users
try. And the Org mode itself is the way to make users fall in love with
Org in one way or another (even unrelated to Emacs [at least
initially]).

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Ihor Radchenko writes:
>
>> The website title is "Org mode for Emacs", repelling users who _do
>>    not want_ to use Org inside Emacs. Maybe we can do better? Something
>>    with less accent on Emacs like "Org mode: your life in plain text"
>
> I am not at all in favor of separating the 'Org Mode' name from 'GNU Emacs'.
> In any case, regardless of my opinion, I see here two problems:
>
> 1. "Org mode: your life in plain text". The word "mode" remains orphan:
> mode of what? It is clear that it is an Emacs mode, therefore it doesn't
> make much sense to hide Emacs and then suggest it.
>
> 2. One possibility: "Org: your life in plain text". But here what
> remains orphaned is "Org", too generic name. Unless "Org Mode" is a
> lexicalized construct, without reference to any emacs mode.
>
> (In any case, I would be extremely saddened if the reference to GNU Emacs
> disappears in the title, just to please a minority).
>

There is another big issue which people are not considering.

Org mode is a GNU project and it is part of Emacs. This actually has
some consequences, most notably -

- It is not acceptable for the project to explicitly or implicitly
  recommend or appear to recommend any non-free solutions or provide
  support for non-free software. Products like Github and MS Visual Code
  fall squarely in the unacceptable bucket.

- It would not be possible to reference any 3rd party libraries or
  programs which are not free software in the proposed list of external
  tools

- The suggested org mode in a browser example is unlikely to be
  acceptable to the FSF (or RMS). The FSF is very much against cloud
  based computing services or any web service which uses non-free
  Javascript (which is most of them and one of the many reasons Github
  is discouraged by the FSF).

A number of the ideas proposed are good ideas for org mode generally -
for example, a repository of reference documents which could be used for
testing purposes would be extremely useful for org-mode development and
testing. Likewise, any effort to clarify the syntax and remove any
ambiguities is beneficial for org mode itself. However, the emphasis and
priority needs to remain focused on org mode as a mode for Emacs. The
use of org mode by other external programs is really outside (but
related) to the project.

As a consequence and to eliminate/remove potential conflicts with FSF
philosophy and goals, it may be worth considering spinning off a
separate project. which happens to use the same markup syntax, but is
not a GNU project (though it would be good to still be GPL'd). 

If you want ot get a feel for the sort of issues which could come up
when trying to develop/support 3rd party tools, have a look at the
recent thread on creating an LSP server for emacs-lisp. While I
personally disagree with most of the fears raised by some contributors
to that thread and disagree with RMS's view that such a server would not
be in the best interests of Emacs, the thread does give you a sample of
the sort of issues which could come up with efforts to support or
encourage 3rd party libraries for org markup, much of which could be
avoided if the work is clearly not part of, related to or supported by
the main org-mode project. 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Juan Manuel Macías <maciaschain@posteo.net> writes:
>
>> Yes, sorry for not explaining myself well: I was also referring to
>> search results, not the title in the web site...
>>
>> But the question is: what need is there to remove the reference to Emacs
>> in the search result? I think the emphasis is necessary. As we say in
>> spanish, it's like putting the bandage on before the wound. If there are
>> people who think that Org Mode can 'live' in some way outside of Emacs
>> (a respectable opinion, but that I do not share, at least 100%), I think
>> the burden of the work falls on them and not on us, the users of
>> Emacs-Org-Mode. Anyway it is my simple and subjetive opinion, and it is
>> not my intention to initiate a controversy with it.
>
> Ok. Let me explain my thought process.
>
> First of all, there is no burden on users of Org mode in making edits to
> orgmode.org. It is a burden on Org contributors.
>
> One of the aims of my proposal is reducing this burden by involving
> non-emacs users to provide contributions to Org (e.g. by making more
> tests for Org-element parser). To do it, we need to make the
> contribution process for non-emacs developers easier. Ideally, without
> too much effort on our side.
>
> The idea of involving non-emacs users does have a potential because we
> do know that third-party tools are already using Org. The problem is the
> disconnect between those tools and Org mode proper.
>
> The sources of the disconnect are (1) lack of technical "blueprints" for
> Org that do not require knowing Elisp; (2) lack of discovereability of
> Org mode as something that can live outside narrow field of Emacs. In
> this branch of our discussion, I am going to talk about the second
> point.
>
> People simply do not expect to see a markup language when they encounter
> a link with "Org mode for Emacs" title. Someone looking for Org mode
> markup to be used in, say, websites will think that "Org mode for Emacs"
> is limited to Emacs. Someone just interested in plain text markup will
> find no relevance at all.
>
> Title is important. If we care at all about orgmode.org website
> appearing in search results, we want the title and the summary to have 2
> main properties: (1) Provide search keywords to make it searchable by
> potentially interested people; (2) Provide a title that immediately
> signal that our website contains the information people are looking for.
>
> Now, we need to understand what kind of people may be looking to
> orgmode.org website.
>
> 1. Existing emacs users
>
>    If a Emacs user is faced with "Org mode for Emacs", the word "Emacs"
>    is indeed recognisable. On the other hand, the word "Org mode" does
>    not provide much further info, except that it is a major (or maybe
>    minor?) mode for "Org"??
>    
>    Now, consider "Org mode: your life in plain text".
>    For emacs users, "Org mode" is not just a strange phrase, but a very
>    clear indication that we are talking about Emacs.
>    The "your life in plain text" provides extra information about what
>    "Org mode" refers to. Clearly, text documents and "your life in plain
>    text" should resonate with every Emacs user's soul.
>
>    We can change the second variant of the title to contain "Emacs", but
>    will it add much value? I am not convinced. On the other hand, making
>    title too long or too complex _is_ bad. Long titles tend to be
>    skipped (there was even formal research on this!)
>
> 2. Non-emacs users interested in plain text markup
>
>    These users know nothing about Emacs and "Org mode" has no meaning
>    for them as is. So, we do need something more descriptive.
>    Adding "Emacs" may be fine, but it will serve no purpose for people
>    not familiar with emacs. Just another unknown term making the title
>    longer.
>
> 3. Non-emacs users interested in GTD/project management, etc
>    "Org mode: your life in plain text" is somewhat relevant when people
>    are looking to manage "life" (typically true for GTD enthusiasts).
>
>    Though we can probably do better for this category.
>    Maybe "Org mode: manage your life and notes in plain text"?
>    Though it makes the title less relevant to group #2.
>
> 4. Researchers looking for ipython-like environment
>
>    Not covered, except by reading my proposed site summary. I am not
>    sure how we can improve title for this audience.
>
> 5. ??? (Suggestions are welcome)
>
> Of course, better suggestions for the title are welcome. I just wanted
> to make it clear the reasoning I do not like the current title and how
> the proposed alternative is better (though not ideal).
>
> Finally, I want to emphasise that our aim for search engines is not
> advertising Emacs (we already do it by trapping users inside Org and
> making them switch to Emacs by force :evil_laughter:). The aim is
> encouraging people to use and contribute to Org mode in useful ways
> (even unrelated to writing Elisp or, really, any code at all).
>
> Search result is just an entrance for users to be curious about the new
> beast of "Org mode". The website front page is the means to make users
> try. And the Org mode itself is the way to make users fall in love with
> Org in one way or another (even unrelated to Emacs [at least
> initially]).
>

I think your working off a false premise. Your view is that org mode
should be available in other editors/software so that others can realise
the power and benefits it provides. I can understand that position.

However, the FSF position would be exactly the opposite. They would
argue that orgmode is a powerful and flexible tool that is part of Emacs
and if you want that power and flexibility, you need to use Emacs. Org
mode has probably done more to bring new users to Emacs than any other
Emacs mode in the last 30 years. As a consequence, you will find not
only little support towards making it available in other editors, you
are likely to run into active resistance. As you say, org-mode can be
thought of as a brand name and that is a brand name owned by the FSF as
an official GNU project and a goal of the FSF is to convert people to
use GNU free software. Anything which has the potential to take the
power of org mode and make it available in non-free software (not simply
open source) is not going to be supported or welcomed.

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Bruce D'Arcus]

On Sun, Dec 5, 2021 at 8:42 AM Tim Cross <theophilusx@gmail.com> wrote:

> I think your working off a false premise. Your view is that org mode
> should be available in other editors/software so that others can realise
> the power and benefits it provides. I can understand that position.
>
> However, the FSF position would be exactly the opposite. They would
> argue that orgmode is a powerful and flexible tool that is part of Emacs
> and if you want that power and flexibility, you need to use Emacs.

Perhaps, but Emacs org users benefit from better support for org
documents outside of Emacs.

Bruce

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I think your working off a false premise. Your view is that org mode
> should be available in other editors/software so that others can realise
> the power and benefits it provides. I can understand that position.

A clarification: my premise is that org mode should be available in
other _free_ editors/software. If we provide the means for other free
software to support Org mode (either as markup format or as some subset
of Elist implementation), it will benefit software freedom in general.
Whatever helper information (think of tests) we provide, it should be
licensed under GPLv3 with the following effect:

https://www.gnu.org/licenses/quick-guide-gplv3.html
>> It's always possible to use GPLed code to write software that
>> implements DRM. However, if someone does that with code protected by
>> GPLv3, section 3 says that the system will not count as an effective
>> technological "protection" measure. This means that if you break the
>> DRM, you'll be free to distribute your own software that does that,
>> and you won't be threatened by the DMCA or similar laws.

The fact is that e.g. Github already provides support for Org markup.
They do it for their own profit and we cannot stop them. If we have a
controlled criteria about quality of third-party Org mode support, there
will be means to interfere with non-free software attempting to makes
profits out of Org mode. For example, if Github do not integrate our
recommended test suite (with all the legal consequences defined in
GPLv3), we will be able to have a list of third-party tools and, among
free alternatives, mention that Github support for Org is not verified
and most likely not consistent with other _free_ tools. We cannot do it
now.

> However, the FSF position would be exactly the opposite. They would
> argue that orgmode is a powerful and flexible tool that is part of Emacs
> and if you want that power and flexibility, you need to use Emacs. Org
> mode has probably done more to bring new users to Emacs than any other
> Emacs mode in the last 30 years. As a consequence, you will find not
> only little support towards making it available in other editors, you
> are likely to run into active resistance. As you say, org-mode can be
> thought of as a brand name and that is a brand name owned by the FSF as
> an official GNU project and a goal of the FSF is to convert people to
> use GNU free software. Anything which has the potential to take the
> power of org mode and make it available in non-free software (not simply
> open source) is not going to be supported or welcomed.

I am very much sceptical that third-party tools can provide the level of
Org support Emacs does provide. Emacs is and will remain the most
feature-full tool for people to use Org mode. Org mode's largest power
is configurability thanks to Emacs. On the other hand, Org mode's
support would be welcome in tools like TeXMacs or in forges like
Sourcehut (currently only supporting markdown).

Best,
ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> - The suggested org mode in a browser example is unlikely to be
>   acceptable to the FSF (or RMS). The FSF is very much against cloud
>   based computing services or any web service which uses non-free
>   Javascript (which is most of them and one of the many reasons Github
>   is discouraged by the FSF).

You are not right. I am well aware about the freedom of computation and
proposed organise because it is not actually cloud-based. Organise is a
frontend. It is licensed under AGPL. AGPL is recommended by FSF for
network software.

> A number of the ideas proposed are good ideas for org mode generally -
> for example, a repository of reference documents which could be used for
> testing purposes would be extremely useful for org-mode development and
> testing. Likewise, any effort to clarify the syntax and remove any
> ambiguities is beneficial for org mode itself. However, the emphasis and
> priority needs to remain focused on org mode as a mode for Emacs. The
> use of org mode by other external programs is really outside (but
> related) to the project.

May you clarify which one of the proposed changes has insufficient
emphasis on org mode for Emacs? If you have concrete ideas for
improvement, feel free to propose them.

> As a consequence and to eliminate/remove potential conflicts with FSF
> philosophy and goals, it may be worth considering spinning off a
> separate project. which happens to use the same markup syntax, but is
> not a GNU project (though it would be good to still be GPL'd). 

I think that's what Karl proposed? I created this thread with specific
purpose to adapt his ideas to Org mode as a free software under FSF.

> If you want ot get a feel for the sort of issues which could come up
> when trying to develop/support 3rd party tools, have a look at the
> recent thread on creating an LSP server for emacs-lisp. While I
> personally disagree with most of the fears raised by some contributors
> to that thread and disagree with RMS's view that such a server would not
> be in the best interests of Emacs, the thread does give you a sample of
> the sort of issues which could come up with efforts to support or
> encourage 3rd party libraries for org markup, much of which could be
> avoided if the work is clearly not part of, related to or supported by
> the main org-mode project. 

I have looked through that thread. I do not think that it applies.
Implementing LSP server for Elisp will give little benefit for Emacs while
giving a "free" and large benefit to non-free software at the same time.

Our situation is different. What I propose in a nutshell is: (1) Improve
our technical documentation; (2) Improve our test coverage; (3) Attract
more users to Org mode. Everything gives benefits to Org. In addition to
better integration.

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 3758 bytes --]

Hi Ihor,

Thank you for your email. I have little to add to you analysis and suggestions
other than my strong agreement. However, I will give some of my thoughts that
lead me to this position.

Ultimately, we have a choice. Do we wish to be hostile, or welcoming to interest
in Org outside org-mode? Under the “hostile” or “isolationist” option, we might:
⁃ Talk of Org not as a general format, but the /exclusive/ format of org-mode
⁃ Ignore any and all attempts to support .org files outside Emacs
Under the “welcoming” option, we might:
⁃ Treat OrgMode as a “brand name” for the Org file format, with org-mode as the
  reference implementation
⁃ Take reasonable steps (such as those Ihor suggests) to make Org seem
  relevant/interesting/usable (if inferior) without Emacs
⁃ Encourage efforts to support the format outside Emacs’ org-mode

To me, the choice is clear. I think we loose nothing by choosing by welcoming
interest in Org outside org-mode, but potentially gain much. As Emacs users, we
can think of ourselves as living in a little Emacs bubble (of around 2% of
developers if StackOverflow’s developer survey is to be believed). Like Karl
Voit, I believe Org holds a lot of value as a markup format in and of itself.
The other 98% + some non-developers have good reason in my mind to be curious
about Org. I imagine many of us regularly interact with such people. We see this
interest manifested not only in extensions for various other editors ([neovim],
[atom], [vs code], [sublime], etc.), but also parsers and tools that work with Org
like Hugo, Pandoc, and LogSeq. We do not live in a bubble. We all benefit from
such efforts.

*The more people use Org, in some form, the greater the chance that someone
making a new tool will think to support it.*

Whatever we may make of it, there is clear interest in Org (to some extent)
separately from Emacs. By ignoring that, we only do ourselves and potential
future Org / Emacs org-mode users a disservice.

People are currently making editor extensions and tools for Emacs outside
org-mode. I don’t think this is suddenly going to stop. We might as well help
such efforts. Good tools that work with Org are good for Org / org-mode. By
providing good clear documentation, and a well-defined grammar, we reduce the
risk of different implementations of the syntax and functionality defined by
org-mode. We could even provide some for of “implementation roadmap” (linked to
the syntax specification) to help developers understand what is required to
implement certain functionality (both markup/syntax, and editing features —
more on this idea I’ve had in a future email). Karl Voit’s idea of “levels” of
Org helps make the task less daunting.

Yes, it will take a bit of effort to do this, and in particular to do this well.
I feel it would likely be worth it though. From the efforts we’ve seen so far,
we have nothing to loose and much we could gain.

All the best,
Timothy

p.s. I see some concerns have been raised about freedom and Org outside Emacs.
While the FSF/GNU project are champions of the FOSS movement, there are many
other FOSS projects and FOSS editors. To decry helping non-GNU/FSF
projects/editors because there are non-free projects/editors seems a bit much.
If improving our documentation and being friendlier to non-Emacs users looking
at the project website is anti-freedom/breaks FSF rules, what’s making an Emacs
build for Windows?


[neovim] <https://github.com/nvim-orgmode/orgmode>

[atom] <https://atom.io/packages/org-mode>

[vs code] <https://github.com/vscode-org-mode/vscode-org-mode>

[sublime] <https://packagecontrol.io/packages/OrgExtended>

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Ihor Radchenko writes:

> Ok. Let me explain my thought process.
>
> First of all, there is no burden on users of Org mode in making edits to
> orgmode.org. It is a burden on Org contributors.
>
> One of the aims of my proposal is reducing this burden by involving
> non-emacs users to provide contributions to Org (e.g. by making more
> tests for Org-element parser). To do it, we need to make the
> contribution process for non-emacs developers easier. Ideally, without
> too much effort on our side.
>
> The idea of involving non-emacs users does have a potential because we
> do know that third-party tools are already using Org. The problem is the
> disconnect between those tools and Org mode proper.
>
> The sources of the disconnect are (1) lack of technical "blueprints" for
> Org that do not require knowing Elisp; (2) lack of discovereability of
> Org mode as something that can live outside narrow field of Emacs. In
> this branch of our discussion, I am going to talk about the second
> point.
>
> People simply do not expect to see a markup language when they encounter
> a link with "Org mode for Emacs" title. Someone looking for Org mode
> markup to be used in, say, websites will think that "Org mode for Emacs"
> is limited to Emacs. Someone just interested in plain text markup will
> find no relevance at all.
>
> Title is important. If we care at all about orgmode.org website
> appearing in search results, we want the title and the summary to have 2
> main properties: (1) Provide search keywords to make it searchable by
> potentially interested people; (2) Provide a title that immediately
> signal that our website contains the information people are looking for.
>
> Now, we need to understand what kind of people may be looking to
> orgmode.org website.
>
> 1. Existing emacs users
>
>    If a Emacs user is faced with "Org mode for Emacs", the word "Emacs"
>    is indeed recognisable. On the other hand, the word "Org mode" does
>    not provide much further info, except that it is a major (or maybe
>    minor?) mode for "Org"??
>    
>    Now, consider "Org mode: your life in plain text".
>    For emacs users, "Org mode" is not just a strange phrase, but a very
>    clear indication that we are talking about Emacs.
>    The "your life in plain text" provides extra information about what
>    "Org mode" refers to. Clearly, text documents and "your life in plain
>    text" should resonate with every Emacs user's soul.
>
>    We can change the second variant of the title to contain "Emacs", but
>    will it add much value? I am not convinced. On the other hand, making
>    title too long or too complex _is_ bad. Long titles tend to be
>    skipped (there was even formal research on this!)
>
> 2. Non-emacs users interested in plain text markup
>
>    These users know nothing about Emacs and "Org mode" has no meaning
>    for them as is. So, we do need something more descriptive.
>    Adding "Emacs" may be fine, but it will serve no purpose for people
>    not familiar with emacs. Just another unknown term making the title
>    longer.
>
> 3. Non-emacs users interested in GTD/project management, etc
>    "Org mode: your life in plain text" is somewhat relevant when people
>    are looking to manage "life" (typically true for GTD enthusiasts).
>
>    Though we can probably do better for this category.
>    Maybe "Org mode: manage your life and notes in plain text"?
>    Though it makes the title less relevant to group #2.
>
> 4. Researchers looking for ipython-like environment
>
>    Not covered, except by reading my proposed site summary. I am not
>    sure how we can improve title for this audience.
>
> 5. ??? (Suggestions are welcome)
>
> Of course, better suggestions for the title are welcome. I just wanted
> to make it clear the reasoning I do not like the current title and how
> the proposed alternative is better (though not ideal).
>
> Finally, I want to emphasise that our aim for search engines is not
> advertising Emacs (we already do it by trapping users inside Org and
> making them switch to Emacs by force :evil_laughter:). The aim is
> encouraging people to use and contribute to Org mode in useful ways
> (even unrelated to writing Elisp or, really, any code at all).
>
> Search result is just an entrance for users to be curious about the new
> beast of "Org mode". The website front page is the means to make users
> try. And the Org mode itself is the way to make users fall in love with
> Org in one way or another (even unrelated to Emacs [at least
> initially]).

Ihor, thank you very much for explaining your motivation in detail. I
think I understand it and (on the important points) I share it. In my
case, as an Org Mode user I often feel a mixture of happiness and
frustration. Happiness on using Org. Frustration every time I want to
recommend Org to many of my friends and colleagues, who don't even use
Emacs. GNU Emacs is a great, labyrinthine, fascinating building. Almost
like a city. And Org is a room on one of the upper floors. In the
Org-room (including Org-Roam! :-) there is a lot of fun, great people,
great music. But whoever wants to get there must go through a series of
levels, intricate corridors that are like a kind of learning path. Emacs
is great, but you can't learn to use it in two days. It takes time to
adapt it to your needs, get to know it, even love it. Sorry to be so
metaphorical and poetic, but it is the only way I can find to explain
what is, in many ways, a personal learning process (even the init Emacs
file could be understood as a kind of autobiography...). I wish the entry
into Org was smoother and more direct, but being an Emacs mode, it is
necessary to go through Emacs. And this gets more crude for
non-technical, Humanities users, who I think could be very happy using
Org and not Word or other modern auto-torture methods :-)

I came to Org been an Emacs user already, so I was reasonably familiar
with Emacs and Elisp, although what I used most was AUCTeX (and Markdown
for my docs). I heard about Org but it never caught my attention, until
one day I read the Org compact guide and I was fascinated that such a
thing existed (and it was just the compact guide!).

TL; DR: I understand and share the maneuver of baiting new and potential
users. But I see it difficult. Users who have never used Emacs will have
to go through the Emacs learning process first, especially non-technical
users or those who come from the country of word processors, that is,
Mordor. On the other hand, Org as a lightweight markup language is only
a tiny part of Org. I don't think Org is better or worse as a markup
language than Markdown, asciidoc or other similar formats. I think the
important thing about Org here is the integration of a series of
resources: a whole that is more important than the sum of the parts
(https://en.wikipedia.org/wiki/Emergentism).

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Sun, Dec 5, 2021 at 8:42 AM Tim Cross <theophilusx@gmail.com> wrote:
>
>> I think your working off a false premise. Your view is that org mode
>> should be available in other editors/software so that others can realise
>> the power and benefits it provides. I can understand that position.
>>
>> However, the FSF position would be exactly the opposite. They would
>> argue that orgmode is a powerful and flexible tool that is part of Emacs
>> and if you want that power and flexibility, you need to use Emacs.
>
> Perhaps, but Emacs org users benefit from better support for org
> documents outside of Emacs.
>

That may be true, but it is secondary to the main goal of the FSF. It
isn't about benefit or convenience. It is about freedom. My guess is
that the FSF position is likely to be something along the lines that
while supporting/encouraging org mode in other, possibly non-free,
software might be beneficial to Emacs users, it not only fails to
promote the objectives of the FSF, it may actually work against it by
decreasing the incentives to move away from a closed non-free solution
to an open free one. 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> I think your working off a false premise. Your view is that org mode
>> should be available in other editors/software so that others can realise
>> the power and benefits it provides. I can understand that position.
>
> A clarification: my premise is that org mode should be available in
> other _free_ editors/software. If we provide the means for other free
> software to support Org mode (either as markup format or as some subset
> of Elist implementation), it will benefit software freedom in general.
> Whatever helper information (think of tests) we provide, it should be
> licensed under GPLv3 with the following effect:
>

I don't disagree with this objective. My objection is to changing the
emphasis or priority of org mode as an Emacs mode to a general technical
specification for a small part of what is org-mode, the markup (I will
outline the concerns I have in doing this below). The other point of
course is that if you make it easier to implement org markdown in other
editors, you don't have control over whether they are implemented in
free or non-free solutions. 

> https://www.gnu.org/licenses/quick-guide-gplv3.html
>>> It's always possible to use GPLed code to write software that
>>> implements DRM. However, if someone does that with code protected by
>>> GPLv3, section 3 says that the system will not count as an effective
>>> technological "protection" measure. This means that if you break the
>>> DRM, you'll be free to distribute your own software that does that,
>>> and you won't be threatened by the DMCA or similar laws.
>
> The fact is that e.g. Github already provides support for Org markup.
> They do it for their own profit and we cannot stop them. If we have a
> controlled criteria about quality of third-party Org mode support, there
> will be means to interfere with non-free software attempting to makes
> profits out of Org mode. For example, if Github do not integrate our
> recommended test suite (with all the legal consequences defined in
> GPLv3), we will be able to have a list of third-party tools and, among
> free alternatives, mention that Github support for Org is not verified
> and most likely not consistent with other _free_ tools. We cannot do it
> now.
>

There is a fatal flaw in this argument. The GPL licenses code, not the
underlying idea (which is essentially what patents are about). We can
define all the quality criteria we like for 3rd party implementations,
but we cannot enforce them unless they are also using the GPL'd code. As
this code is elisp, it is extremely unlikely any 3rd party
implementation will be using it. In short, defining clear specifications
and minimal quality criteria will only have baring on code added to org
mode - it would, for example, have no effect on what github has/is
implementing. 

>> However, the FSF position would be exactly the opposite. They would
>> argue that orgmode is a powerful and flexible tool that is part of Emacs
>> and if you want that power and flexibility, you need to use Emacs. Org
>> mode has probably done more to bring new users to Emacs than any other
>> Emacs mode in the last 30 years. As a consequence, you will find not
>> only little support towards making it available in other editors, you
>> are likely to run into active resistance. As you say, org-mode can be
>> thought of as a brand name and that is a brand name owned by the FSF as
>> an official GNU project and a goal of the FSF is to convert people to
>> use GNU free software. Anything which has the potential to take the
>> power of org mode and make it available in non-free software (not simply
>> open source) is not going to be supported or welcomed.
>
> I am very much sceptical that third-party tools can provide the level of
> Org support Emacs does provide. Emacs is and will remain the most
> feature-full tool for people to use Org mode. Org mode's largest power
> is configurability thanks to Emacs. On the other hand, Org mode's
> support would be welcome in tools like TeXMacs or in forges like
> Sourcehut (currently only supporting markdown).
>

I don't disagree about the benefit of org markup being supported in 3rd
party tools. I disagree with the proposal to change the emphasis of the
org-mode project from being an Emacs mode to being a more general
technology.

Consider this contrived scenario.

We have adopted a change in emphasis to promote org mode as a more
general solution and clarified the specification to make it easier for
3rd parties to implement.

Meanwhile, Emacs development continues and new features/capabilities
continue to be added. In particular, a new feature is added which is
extremely powerful and would be a huge benefit for Emacs org-mode users.
However, there is a problem. In order to take advantage of this new
feature, significant changes are required for the specification. This
will result in implementations requiring considerable work in order to
update them to the new specification. Worse still, the nature of this
change means that only Emacs org-mode users will benefit from this
change. The change will not realise any benefit for 3rd party
implementations.

Now we have a problem. Either Emacs org mode users must give up on this
great new feature or we break compatibility with 3rd party libraries.
However, if the user base of the 3rd party implementations is much
larger than the Emacs org mode user base, there will be strong pressure
to not make this change.

An extreme scenario to emphasise the point that moving org-mode from
being primarily and Emacs mode to being a more general technical
solution could easily have adverse impact on the development of org mode
for Emacs users.

I also have a more fundamental problem with the whole premise regarding
org markdown and 'bringing it to the world'. This will likely be
controversial, but ....

There is nothing fundamentally better or more powerful about the
markdown dialect used by org mode over any other markdown dialect. In
fact, other markdown dialects are possibly even better than the one used
by org mode.

The power of org-mode is in all the non-markdown functionality - section
folding, table formulas, executable source code blocks, powerful link
definition facilities, noweb/literate programming support,
planning/tasks/scheduling/time tracking, multiple back end target
exports etc.

To some extent, all these issues about markdown and compatibility in 3rd
party libraries and editors is actually a consequence of org mode being
defined at a point where markdown was relatively new and still somewhat
immature. I see parallels here with much of the Emacs terminology which
new users find very alien. The Emacs use of frames, windows and buffers
or even the default key bindings for cut, copy and paste seem weird
these days. However, they were novel new concepts when Emacs implemented
them.

It is similar with org markdown. At the time it was implemented, it was
compatible with one of the best known and popular markdown dialects and
one of the few to have actually written down it's specification.
However, things evolved and now we have a handful of popular markdown
dialects and the one org adopted is no longer as popular. If org had
been developed later, it could well have adopted a different markdown
syntax, possibly github flavoured for example. If this had occurred, we
would have been more compatible with many 3rd party libraries and much
of this debate would not exist.

We have had many threads discussing how to increase compatibility of org
mode in 3rd party applications and many good suggestions. Unfortunately,
they usually amount to nothing because while ideas are cheap, action is
not and often we get caught up on side issues which are not actually
that important (at this stage at least). Forget about changing the
emphasis or search engine results etc at this time. Focus on the more
concrete components, like a clear specification of syntax, good unit
tests and definitive reference documents. Just these three items
represent a huge amount of work and all are likely prerequisites for
establishing a vibrant 3rd party ecosystem. Worry about the rest later.

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Sun, Dec 05, 2021 at 06:59:20PM +0000, Juan Manuel Macías wrote:
> Frustration every time I want to recommend Org to many of my friends
> and colleagues, who don't even use Emacs.

I think this is the core of every interoperability argument: "Why do
we have to use Emacs to use Org?" It's called Emacs Org-mode for a
reason. Org-mode doesn't work outside of Emacs.

I've often told users that it's worth learning enough Emacs to use
Org, and have successfully moved several non-power users over to
Emacs. They know enough to consistently open their files, edit Org,
and quit. That's enough. They complain it's "ugly" compared to
"modern" GUI tools, but they can't deny the utility.

> I came to Org been an Emacs user already, so I was reasonably familiar
> with Emacs and Elisp, although what I used most was AUCTeX (and Markdown
> for my docs).

I'd have a hard time convincing users in Mordor to use a plain text
format, much less Org without Emacs.

> On the other hand, Org as a lightweight markup language is only
> a tiny part of Org. I don't think Org is better or worse as a markup
> language than Markdown, asciidoc or other similar formats.

What makes Org dramatically different is the editing experience in
Emacs. Collapsing the outline, filtering on metadata, exports, agenda,
etc. Those are Emacs features, not specific to the actual markup
format.

My impression is we already have stretched our resources thin trying
to maintain Org. Pushing to provide compatibility with non-Emacs tools
seems a poor use of their time, and rude to ask of them.

If non-Emacs users and coders want to use Org formatted files, they
are free to spend their time customizing their tools to make it
work. If the Org project owes anything to anyone, it's a consistent
experience for the users who have used Org for years. Not changes to
satisfy potential users or follow trendy fads.

My experience has been that Org's markup is so simple and could be
summarized in a few lines. Anything more complex enters the territory
of Emacs only features (ie: drawers. exports, view control, source
blocks, reporting). Those are unlikely to be portable, so we're back
to "use Emacs".

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Russell Adams writes:

> What makes Org dramatically different is the editing experience in
> Emacs. Collapsing the outline, filtering on metadata, exports, agenda,
> etc. Those are Emacs features, not specific to the actual markup
> format.
>
> My impression is we already have stretched our resources thin trying
> to maintain Org. Pushing to provide compatibility with non-Emacs tools
> seems a poor use of their time, and rude to ask of them.
>
> If non-Emacs users and coders want to use Org formatted files, they
> are free to spend their time customizing their tools to make it
> work. If the Org project owes anything to anyone, it's a consistent
> experience for the users who have used Org for years. Not changes to
> satisfy potential users or follow trendy fads.
>
> My experience has been that Org's markup is so simple and could be
> summarized in a few lines. Anything more complex enters the territory
> of Emacs only features (ie: drawers. exports, view control, source
> blocks, reporting). Those are unlikely to be portable, so we're back
> to "use Emacs".

I think that I cannot agree more with this. Org Mode is GNU Emacs, and
the magic of Org Mode is the magic of GNU Emacs. That's why I insist
that going to Org means going to Emacs.

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

Hi Juan,

> I think that I cannot agree more with this. Org Mode is GNU Emacs, and
> the magic of Org Mode is the magic of GNU Emacs. That’s why I insist
> that going to Org means going to Emacs.

I don’t think Ihor is suggesting we stop indicating that org-mode is part of
Emacs. I think there’s been a fair bit of misinterpretation in this thread.
Let’s treat the proposal as what it is. Namely:
⁃ Indicating that Org as a concept has spread beyond Emacs (it has, look at the
  number of tools as extensions for it)
⁃ Perhaps use some of the tools that people have developed to provide a
  no-install in browser demo of some of the functionality
⁃ Improve our technical documentation and testing
⁃ Let people know what to expect from other tools/extensions that offer “Org support”

That’s it.

All the best,
Timothy

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Hi Timohy,

Timothy writes:

> I don’t think Ihor is suggesting we stop indicating that org-mode is part of
> Emacs.

Of course, I am convinced that Ihor is not saying that Org is not part
of Emacs, and I have to make it clear, that I have never suggested such
a thing. What's more, I understand and applaud his conciliatory effort
on this issue, since since a series of debates and noise have arisen
here on these matters.

What worries me are the consequences of insisting here on these debates.
That is why I have underlined what Russell has commented, as I think he
is right.

On the other hand, we must not forget that Org, as part of Emacs, is
part of GNU, and this is a mailing list from the GNU project. I think
everything related to the (possible) extension of GNU Org Mode outside
of GNU Emacs (even in software incompatible with the ethics and
philosophy of the GNU project) should be considered offtopic here and be
discussed in other forums. Otherwise it would only create confusion
among users.

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Greg Minshall]

Juan Manuel (and, Tim, i think),

> On the other hand, we must not forget that Org, as part of Emacs, is
> part of GNU, and this is a mailing list from the GNU project. I think
> everything related to the (possible) extension of GNU Org Mode outside
> of GNU Emacs (even in software incompatible with the ethics and
> philosophy of the GNU project) should be considered offtopic here and
> be discussed in other forums. Otherwise it would only create confusion
> among users.

i hope we don't adopt such an "official policy" regarding discussions on
this list.  i don't think we've had any problems where non-FSF/GNU
topics have somehow swamped our discussions.

cheers, Greg

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Greg Minshall]

Ihor,

> Search result is just an entrance for users to be curious about the
> new beast of "Org mode". The website front page is the means to make
> users try. And the Org mode itself is the way to make users fall in
> love with Org in one way or another (even unrelated to Emacs [at least
> initially]).

i *do* agree that our hope is that people will see the great benefit of,
and use, "emacs org mode".  however, along with others, i don't want
people to fall in love with "org mode" itself, but, rather with the full
"running org mode with emacs".

at the same time, i have a desire that "the general public" will be able
to "work with" "emacs org mode" *files*; i.e., treat them as somewhat
more than a collection of octets.  *how much* more is tbd; i sort of saw
Karl Voit's proposal of "level 1", ..., as being an attempt to define
ways of extracting (limited) utility from an org mode file.

cheers, Greg

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Max Nikulin]

On 05/12/2021 14:35, Ihor Radchenko wrote:
> 
> The recent spike of discussions following Karl's presentation in
> Emacsconf 2021 revealed a lot of controversy among Org and Emacs
> enthusiasts. Yet, Karl named a number of very real problems surrounding
> Org mode usage outside Emacs.
> 
> WDYT?

Ihor, I like your ideas and, I hope, they may be expressed in a way that 
does not cause such irritation and negative reaction.

After all, I suppose, nobody is regretting that it is possible to 
convert text from a set of formats to Org using pandoc which it is not a 
part of Emacs.

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Juan Manuel Macías]

Hi Greg,

Greg Minshall writes:

> i hope we don't adopt such an "official policy" regarding discussions on
> this list.  i don't think we've had any problems where non-FSF/GNU
> topics have somehow swamped our discussions.

Not that I want to put on a censor hat, far from it :-). But this is
still an 'official' list. It's not reddit or anything like that. I think
it is necessary for the user to know what to expect here and what not to
expect. And I think there is a border between offtopic and explicitly
discussing extending Org beyond its natural limits or even flirting with
applications and software that do not respect user freedom and are
outside of GNU ethics and philosophy. And for the record, I am not
talking about Ihor's specific proposals in this thread --- which, as I
have already said, seems to me to be a commendable conciliatory effort,
although I do not share some points--- but rather from the previous
debate on the "new" name of the Org syntax, and other such things. I
think things like the 'orgdown' topic, which are not productive here,
should have their own place and forums outside here.

Best regards,

Juan Manuel 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tom Gillespie]

Hi all,
I have a much longer mail in the works, a quick one for now.

I think it is a major strategic mistake to exclude discussions
about interoperability from this list. As Bastien pointed out in
his talk at Emacsconf there is only a single list for both users
and developers. Discussion about interoperability with tools for
working with Org are entirely valid subjects for the user
list. Obviously help and support for other tools is not valid for
the list, but questions about interoperability or incorrectness
of some external tool should always be valid.

We must provide strong technical leadership for all tools that
want to work with Org syntax otherwise we risk it spiraling out
of control. Forcing discussions off list will split the community
and I think the fact that Karl's work made it to this list so
late in the process shows the danger of trying to exclude certain
discussions.

I follow this list, I keep the community up to date with my work,
I have no idea where to look for other Org related dicussions,
nor frankly do I have time to look for them. I suspect I am not
alone in this.

Whether a certain portion of the Org community likes it or not,
there is another portion for whom Org syntax already has a life
beyond Org mode (e.g. academic papers and computation notebook
style workflows). For some workflows documents written in Org
syntax are a primary exchange format and format of record, not
just an internal format from which documents for sharing are
generated. The plain text nature of Org syntax and the freedom
that it enables also means freedom from Emacs. Empowering users
to own and control their own data to use with their own tools is
the whole point. The fact that this means that it works outside
Emacs is a critical feature for many data preservation use cases.

Enough for now. Best!
Tom

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[M. ‘quintus’ Gülker]

Am Montag, dem 06. Dezember 2021 schrieb Tom Gillespie:
> [On not excluding discussions about org markup from the mailing list]

Thank you for writing this up. I agree with it. This discussion has
taken routes which I would never have expected. We started with an
interoperability topic and now we are discussing whether the intent is
to take away software freedom from Emacs org users. I cannot help but to
find this connection far-fetched. Nobody is suggesting to hamper
org-mode-in-emacs' further development. All that was asked was if there
is interest in someone outside of org-mode-in-emacs writing up
“compatibility levels” for the org markup. Whether or not that is a
daunting task is up to those who want to pick it up, and it certainly is
not "owed" by the org developers to cater for them. If there are
synergies -- there appear to be some, since the org formalising efforts
do predate Karl Voit's proposal apparently -- they should be taken
advantage of and not be banned from discussion on this mailing list.

I do not expect the permissal of such discussions to turn this mailing
list into a hub for help requests for non-org, or, heaven forbid,
non-free software implementing org markup.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu | PGP: Siehe Webseite
Passau, Deutschland  | kontakt@guelker.eu    | O<

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Mon, Dec 06, 2021 at 09:59:42AM -0800, Tom Gillespie wrote:
> I think it is a major strategic mistake to exclude discussions
> about interoperability from this list.

I don't think discussion on the list (or irc) is a problem. It's all
on topic if it's related to Org-mode. As you said, users and
developers share the mailing list. Just understand as an Emacs mode,
it's Emacs oriented and Emacs is the priority. The truth is Org
doesn't work outside Emacs.

The issue is when non-Emacs enhancements or feature requests are made
to the maintainers and coders. I don't think anyone is hostile to
interoperability, but hostile to additional workload.

I've watched Org since shortly after it's creation snowball features
nonstop until it burned out coders. I feel like every power user with
a new edge case feature wants it added to the Org core, and that's
just not sustainable. That's why I'm very cautious about advocating
for new features, or potential burdens external interoperability may
place on our volunteers.

> I follow this list, I keep the community up to date with my work,
> I have no idea where to look for other Org related dicussions,

IRC. #org-mode on Libera.

> Whether a certain portion of the Org community likes it or not,
> there is another portion for whom Org syntax already has a life
> beyond Org mode (e.g. academic papers and computation notebook
> style workflows).

Ideally written with Emacs, and the source blocks processed by
Emacs. I can't imagine any of the data science source blocks working
in any environment outside Emacs today.

If other programs try to replicate Org's features that's fine in the
spirit of open source, but what support do we owe their
implementations? At what point does their project impose a maintenance
burden on our volunteers? That's what we need to be careful of.

Perhaps it's worth noting the clear distinction between Org's plain
text format and the Org experience inside Emacs. I think that the
plain text format in it's simplest forms may be interpreted by
external tools (ie: README.org on Github is HTML formatted). I don't
expect any tools outside of Emacs to provide the Org editing
experience.

> The plain text nature of Org syntax and the freedom that it enables
> also means freedom from Emacs. Empowering users to own and control
> their own data to use with their own tools is the whole point. The
> fact that this means that it works outside Emacs is a critical
> feature for many data preservation use cases.

Certainly Org is future proof because it's plain text. There's a big
difference between future proofed and openly legible versus
programming two way interoperability between Emacs and an external
tool. Just ask any synchronization tool. ;]

In summary, don't assume hostility to interoperability. Please respect
the limited time of our coders and maintainers, and the additional
burdens on them that may be implied by supporting external programs.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Mon, Dec 06, 2021 at 07:25:02PM +0100, M. ‘quintus’ Gülker wrote:
>
> We started with an interoperability topic and now we are discussing
> whether the intent is to take away software freedom from Emacs org
> users. I cannot help but to find this connection far-fetched. Nobody
> is suggesting to hamper org-mode-in-emacs' further development. All
> that was asked was if there is interest in someone outside of
> org-mode-in-emacs writing up “compatibility levels” for the org
> markup.

These kind of issues snowball because we are also indirectly asking
for our coders and maintainers to consider those external tools while
continuing to support Org. How many syntax documents are we supposed
to maintain outside of the working implementation in Emacs and the
manual?

The topic of software freedom comes up because by definition, other
tools are outside of Emacs and may be non-free. It's important to
consider, but isn't a reason to not discuss features. The key is our
volunteers should not be required to code features for non-free tools
outside of Emacs.

An implied support requirement to preserve interoperability with
external tools is a large commitment, and could also run into the
non-free software issue. Expect people to have strong opinions about
these matters.

> ... not be banned from discussion on this mailing list.

I don't think banning topics is appropriate. I think we were reminded
that this is an Emacs list and FSF software ethics apply. Any official
stance Org maintainers take would have to be in line with that moral
code.

Discussions are often fruitful for all involved and shouldn't be a
problem when conducted in a respectful manner. Expect critical
opinions at times, but we should keep it civil.


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Sun, Dec 05, 2021 at 03:35:39PM +0800, Ihor Radchenko wrote:
> Dear Fellow Orgers,
>
> The recent spike of discussions following Karl's presentation in
> Emacsconf 2021 revealed a lot of controversy among Org and Emacs
> enthusiasts. Yet, Karl named a number of very real problems surrounding
> Org mode usage outside Emacs.

On a lighter note, I preferred the format be named "Orgish".

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

Hi Russell,

I have a few comments on your comments :)

> These kind of issues snowball because we are also indirectly asking
> for our coders and maintainers to consider those external tools while
> continuing to support Org.

As I read it, considering other tools was just in the respect of indicating what
degree of support different tools have for Org as defined by org-mode. Not
changing anything we do based on those tools.

> How many syntax documents are we supposed to maintain outside of the working
> implementation in Emacs and the manual?

Just the one. I have some ideas on this that need to be written up, but I see
this more as polishing our syntax specification such that it’s is more
approachable for someone interested in supporting Org. IMO this leads to a
syntax document which is just better, period.

> The topic of software freedom comes up because by definition, other
> tools are outside of Emacs and may be non-free. It’s important to
> consider, but isn’t a reason to not discuss features. The key is our
> volunteers should not be required to code features for non-free tools
> outside of Emacs.

I don’t think anybody has proposed this. Personally I’m not even sure how we
went from making the website/docs more approachable to supporting 3rd party
tools ourselves, or changing anything in org-mode itself for them…

> An implied support requirement to preserve interoperability with
> external tools is a large commitment, and could also run into the
> non-free software issue. Expect people to have strong opinions about
> these matters.

With regards to this, by the very nature of things, any change that would break
interoperability with external tools would have to be a breaking change to the
syntax (with all the relevant implications for org-mode itself). In such a
situation, this would be the major concern, not external tools, so I see this
line of reasoning as being a bit moot.

> Discussions are often fruitful for all involved and shouldn’t be a
> problem when conducted in a respectful manner. Expect critical
> opinions at times, but we should keep it civil.

Indeed. I do find myself wishing that some discussions stayed more on-topic
though…

All the best,
Timothy

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Gerry Agbobada]

Hello everybody,

On Mon, Dec 6, 2021, at 18:59, Tom Gillespie wrote:
> I follow this list, I keep the community up to date with my work,
> I have no idea where to look for other Org related dicussions,
> nor frankly do I have time to look for them. I suspect I am not
> alone in this.

Just not to leave this be a wild guess or a lone data-point, I want to say that I’m exactly in the same case, and I really don’t want to bring up anything I do related to org-mode here because of this kind of backlash without which I feel really better. Too bad I guess, I’ll just try to communicate here and there through github issues as I’ve been doing until now.

Regards,
G
-- 
Gerry Agbobada

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Tue, Dec 07, 2021 at 02:47:28AM +0800, Timothy wrote:
> I have a few comments on your comments :)

To my esteemed colleague, I have a few comments for your comments on
my comments. ;]

> > How many syntax documents are we supposed to maintain outside of the working
> > implementation in Emacs and the manual?
>
> Just the one. I have some ideas on this that need to be written up, but I see
> this more as polishing our syntax specification such that it’s is more
> approachable for someone interested in supporting Org. IMO this leads to a
> syntax document which is just better, period.

I'm all for the idea of tightening up documentation to make Org a more
polished product. The issue is when the justification for that effort
is interoperability with tools outside Emacs.

> > Discussions are often fruitful for all involved and shouldn’t be a
> > problem when conducted in a respectful manner. Expect critical
> > opinions at times, but we should keep it civil.
>
> Indeed. I do find myself wishing that some discussions stayed more on-topic
> though…

My goal is to remind everyone that maintainer and coder time is a
scarce resource, and I'm very protective of asking them to commit to
anything. An indirect commitment can still feel like a commitment,
even if it's only implied by popular opinion and not agreed to.

As a free program with free and plain text results, anyone can code
anything they want to work with it. Asking Org volunteers to do
something outside the scope of Emacs should be critically examined
before we can justify asking for volunteer time. That doesn't mean
topics shouldn't be discussed, that discussion is the process of
critical examination. There's also plenty of room for "fluff", but not
flame.

For instance, Karl clearly spent a lot of time on his proposal. I
thought he was trying to clearly articulate issues from the recent
discussions regarding interoperability. I appreciate his efforts even
when I don't agree with every conclusion (I vote "Orgish"!). I wish I
knew enough about the underlying code to make an informed opinion so I
have abstained from on commenting on the details.

I don't have to agree or disagree with every point, I'm just watching
the debate and trying to place some gentle reminders in the discussion
to keep it civil and remember our volunteers are just that,
volunteers.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

Hi Russell,

> To my esteemed colleague, I have a few comments for your comments on
> my comments. ;]

Lovely. I happen to have one or two remarks on your comments^{2} :P

> I’m all for the idea of tightening up documentation to make Org a more
> polished product. The issue is when the justification for that effort
> is interoperability with tools outside Emacs.

I think this is always a good thing. Regarding the motivation, I’m not sure how
crucial it is. The idea I have had and am yet to articulate (hopefully I’ll sit
down and do so this week) is originally motivated by considerations for people
who want to make tools that work with Org, but upon reflection I think has
broader benefits. I think the framing is also rather important here, not
changing Org to fit other tools, but making it easier for other tools to shape
themselves to be interoperable with org-mode — which I see as the crux of this
topic.

> My goal is to remind everyone that maintainer and coder time is a
> scarce resource, and I’m very protective of asking them to commit to
> anything. An indirect commitment can still feel like a commitment,
> even if it’s only implied by popular opinion and not agreed to.

It’s always nice to see you advocating for consideration of maintainers and
contributors free time. With this specific topic though, given that we have one
maintainer/significant contributor (Ihor) asking for feeback on proposed changes
w/ motivation and another one (me) very interested in it , I’m not sure that
this concern is particularly relevant.

All the best,
Timothy

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Karl Voit]

Hi,

* Ihor Radchenko <yantar92@gmail.com> wrote:
>
> The fact is that e.g. Github already provides support for Org markup.
> They do it for their own profit and we cannot stop them. If we have a
> controlled criteria about quality of third-party Org mode support, there
> will be means to interfere with non-free software attempting to makes
> profits out of Org mode. For example, if Github do not integrate our
> recommended test suite (with all the legal consequences defined in
> GPLv3), we will be able to have a list of third-party tools and, among
> free alternatives, mention that Github support for Org is not verified
> and most likely not consistent with other _free_ tools. We cannot do it
> now.

This is the main reason why I came up with the concept of Orgdown
levels in the first place. Tools may choose their featureset
independently and Orgdown assessments show the users the amount of
features they can get.

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Karl Voit]

Hi,

* Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Now, we need to understand what kind of people may be looking to
> orgmode.org website.
>
> 1. Existing emacs users
> 2. Non-emacs users interested in plain text markup
> 3. Non-emacs users interested in GTD/project management, etc
>    "Org mode: your life in plain text" is somewhat relevant when people
>    are looking to manage "life" (typically true for GTD enthusiasts).
> 4. Researchers looking for ipython-like environment
> 5. ??? (Suggestions are welcome)

I don't think that you can come up with a good website that is able
to serve all of them properly.

For Orgdown I had in mind number 2 and 3. For supporting the idea
for Orgdown, I thought of number 1. (Well, that turned out in a
mixed way[1]).

If you do want to have a go-to for people who should use the
Org-mode syntax without the GNU/Emacs tool, you need to have a
separate site.

[1] https://karl-voit.at/2021/12/02/Orgdown-feedback

-- 
get mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML into Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <
Personal Information Management > http://Karl-Voit.at/tags/pim/
Emacs-related > http://Karl-Voit.at/tags/emacs/

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Vincent Breton]

Hi,

Org mode needs to have his pdf  documentation on the official web page
of documentation :  https://www.gnu.org/software/emacs/manual/org.html

How much time for example to copy  http://www.presentiel.com/org/org.pdf
file  on the official web site for gnu documentation, or if you prefer
to generate it again using sources?

In my opinion the link
https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html is
not appropriate and must be changed to
https://www.gnu.org/software/emacs/manual/emacs.html in the web
documentation page of Emacs. People must have the choice to choose
easily the format of files to read when possible and not be forced to be
oriented with or forced to use HTML format

Design web site  musn't reduce the access of documentation of products
to different formats. A direct link to manuals to differents formats
must be provided and others formats musn't be neglated, more
particularly when tools integrates sources and options to generate them.

Yes, separate site is good but it don't forget a easy and direct link to
the official doc to different formats...

Vincent

On 06/12/2021 20:45, Karl Voit wrote:
> Hi,
>
> * Ihor Radchenko <yantar92@gmail.com> wrote:
>> Now, we need to understand what kind of people may be looking to
>> orgmode.org website.
>>
>> 1. Existing emacs users
>> 2. Non-emacs users interested in plain text markup
>> 3. Non-emacs users interested in GTD/project management, etc
>>    "Org mode: your life in plain text" is somewhat relevant when people
>>    are looking to manage "life" (typically true for GTD enthusiasts).
>> 4. Researchers looking for ipython-like environment
>> 5. ??? (Suggestions are welcome)
> I don't think that you can come up with a good website that is able
> to serve all of them properly.
>
> For Orgdown I had in mind number 2 and 3. For supporting the idea
> for Orgdown, I thought of number 1. (Well, that turned out in a
> mixed way[1]).
>
> If you do want to have a go-to for people who should use the
> Org-mode syntax without the GNU/Emacs tool, you need to have a
> separate site.
>
> [1] https://karl-voit.at/2021/12/02/Orgdown-feedback
>

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

"Gerry Agbobada" <emacs-orgmode@gagbo.net> writes:

> Just not to leave this be a wild guess or a lone data-point, I want to say that I’m exactly in the same case, and I really don’t want to bring up anything I do related to org-mode here because of this kind of backlash without which I feel really better. Too bad I guess, I’ll just try to communicate here and there through github issues as I’ve been doing until now.

I am sorry about your experience.

Judging from your previous emails, you are very interested in grammar
parsing and asked several questions about external implementations of
Org grammar.

Note that your emails (even unanswered) did not go unnoticed. The main
problem is that we do not actually have a consistent uniform grammar, we
just _mostly_ have it. Nicolas is making a long-time effort to write the
formal grammar specification while simultaneously identifying and moving
bits of vague parsing Elisp routines into org-element.el (which he
created). I am too partially interested in making Org parser more
uniform and knowing the actual state of underlying Elisp code makes me
reluctant to reply to "official parser" requests immediately (all those
email replies are in my todo-list).

This thread is partially a realisation that without even half-cooked and
accessible grammar description, things can spin out of control.

Note that one of my ideas is creating a set of testing files that should
be easier to contribute without diving into org testing code.

As I mentioned in my other call for patches [1], improvements to
existing patch suite (including testing the grammar) would be very much
appreciated and definitely not frowned on.

[1] https://list.orgmode.org/871r342z6g.fsf@localhost/T/#t

Best,
Ihor

P.S.

One of grammar-related emails in my todo-list:

****** TICKLER [#A] Tom Gillespie [ML:Org mode] (2021) A formal grammar for Org :BOOKMARK:misc:email:
SCHEDULED: <2021-12-20 Mon>
:PROPERTIES:
:TITLE:    A formal grammar for Org
:BTYPE:    misc
:ID:       ML:Org-mode-<tgbugs@gmail.com>2021-formal-grammar-org-171
:AUTHOR:   Tom Gillespie <tgbugs@gmail.com>
:CREATED:  [2021-09-22 Wed 13:27]
:HOWPUBLISHED: ML:Org mode
:LINK:     notmuch:id:CA+G3_PNj6Pekqv+tWFkwbD778XhW9WSfx+kjJhjSOREpLHUpRQ@mail.gmail.com
:NOTE:     Online; accessed 22 September 2021
:TYPEALT:  email
:YEAR:     2021
:DRILL_LAST_INTERVAL: 35.8039
:DRILL_REPEATS_SINCE_FAIL: 5
:DRILL_TOTAL_REPEATS: 7
:DRILL_FAILURE_COUNT: 1
:DRILL_AVERAGE_QUALITY: 3.143
:DRILL_EASE: 1.9
:END:
:LOGBOOK:
- Refiled on [2021-09-22 Wed 14:17]
:END:

(7 times postponed...)

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Vincent Breton <emacs-orgmode@presentiel.fr> writes:

> Hi,
>
> Org mode needs to have his pdf  documentation on the official web page
> of documentation :  https://www.gnu.org/software/emacs/manual/org.html
>
> How much time for example to copy  http://www.presentiel.com/org/org.pdf
> file  on the official web site for gnu documentation, or if you prefer
> to generate it again using sources?

We have seen the request you posted in separate thread. Please, do not
clutter this thread by re-sending the message you have already posted.

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Karl Voit <devnull@Karl-Voit.at> writes:

>> Now, we need to understand what kind of people may be looking to
>> orgmode.org website.
>>
>> 1. Existing emacs users
>> 2. Non-emacs users interested in plain text markup
>> 3. Non-emacs users interested in GTD/project management, etc
>>    "Org mode: your life in plain text" is somewhat relevant when people
>>    are looking to manage "life" (typically true for GTD enthusiasts).
>> 4. Researchers looking for ipython-like environment
>> 5. ??? (Suggestions are welcome)
>
> I don't think that you can come up with a good website that is able
> to serve all of them properly.

> If you do want to have a go-to for people who should use the
> Org-mode syntax without the GNU/Emacs tool, you need to have a
> separate site.

There is quite a bit of misunderstanding going on in this thread. I did
not suggest to change the orgmode.org website for users who _will never
use Emacs_. One of the ideas was rather to make the website more
approachable for the listed groups of people and encourage them to use
Org mode _for Emacs_. (The other one was related to third-party
developers, but it is not what my reply above was about).

The message above is simply about how orgmode.org appears in web search
results. The current website title looks [for non-Emacs users] like
an agglomeration of meaningless words and I suggested to make a little
bit of SEO and change the title into something with more readable
keywords. That's it.

> For Orgdown I had in mind number 2 and 3. For supporting the idea
> for Orgdown, I thought of number 1. (Well, that turned out in a
> mixed way[1]).

I understand and I like the idea of Orgdown in general (and I do not
really care about the naming - you already did a great job making it
sufficiently unique as can be easily seen in
https://duckduckgo.com/?q=orgdown&ia=web). This thread is my attempt to
adapt some ideas you introduced within narrow scope org Org mode for
Emacs and our orgmode.org website. We can really benefit from thinking
about luring non-Emacs users into Org mode for Emacs and from improving
Org grammar documentation (aka Orgdown ∞).

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I don't disagree with this objective. My objection is to changing the
> emphasis or priority of org mode as an Emacs mode to a general technical
> specification for a small part of what is org-mode, the markup (I will
> outline the concerns I have in doing this below).

I think I was not clear enough in my topic-starter when I framed it as
"From the narrow perspective of this mailing list". I do not suggest to
change emphasis of the whole Org mode project. Instead, I am trying to
suggest a set of improvements to orgmode.org aiming to lure broader
auditory to _Org mode for Emacs_ and a change to Org test suite that
would allow easier contribution to _Org mode for Emacs_ from third-party
developers

> ... The other point of
> course is that if you make it easier to implement org markdown in other
> editors, you don't have control over whether they are implemented in
> free or non-free solutions. 

To be clear, writing Org syntax spec has been decided long time ago.
See https://list.orgmode.org/87r1qt9cf0.fsf@gnu.org/ and
https://orgmode.org/worg/dev/org-syntax.html (started long before that
thread). I just suggested to link it to orgmode.org

>> The fact is that e.g. Github already provides support for Org markup.
>> They do it for their own profit and we cannot stop them. If we have a
>> controlled criteria about quality of third-party Org mode support, there
>> will be means to interfere with non-free software attempting to makes
>> profits out of Org mode. For example, if Github do not integrate our
>> recommended test suite (with all the legal consequences defined in
>> GPLv3), we will be able to have a list of third-party tools and, among
>> free alternatives, mention that Github support for Org is not verified
>> and most likely not consistent with other _free_ tools. We cannot do it
>> now.
>>
>
> There is a fatal flaw in this argument. The GPL licenses code, not the
> underlying idea (which is essentially what patents are about). We can
> define all the quality criteria we like for 3rd party implementations,
> but we cannot enforce them unless they are also using the GPL'd code. As
> this code is elisp, it is extremely unlikely any 3rd party
> implementation will be using it. In short, defining clear specifications
> and minimal quality criteria will only have baring on code added to org
> mode - it would, for example, have no effect on what github has/is
> implementing. 

It is indeed unlikely and not using our tests will degrade its quality
if they just use the existing https://orgmode.org/worg/dev/org-syntax.html
So, I only see tests as an improvement.

> Consider this contrived scenario.

As I tried to stress above, I do not suggest to change Org mode project
emphasis. Anyway, some of my replies to your statements below might be
of interest.

> Meanwhile, Emacs development continues and new features/capabilities
> continue to be added. In particular, a new feature is added which is
> extremely powerful and would be a huge benefit for Emacs org-mode users.
> However, there is a problem. In order to take advantage of this new
> feature, significant changes are required for the specification. This
> will result in implementations requiring considerable work in order to
> update them to the new specification.

I disagree. We already need to care about back-compatibility of Org
syntax (think of org documents written years ago). Major changes to
syntax are very unlikely even without considering third-party software.
And, by the way, remember the existing "third party" Elisp packages
(think of Org roam, for example). We do not want to break them.

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Meanwhile, Emacs development continues and new features/capabilities
>> continue to be added. In particular, a new feature is added which is
>> extremely powerful and would be a huge benefit for Emacs org-mode users.
>> However, there is a problem. In order to take advantage of this new
>> feature, significant changes are required for the specification. This
>> will result in implementations requiring considerable work in order to
>> update them to the new specification.
>
> I disagree. We already need to care about back-compatibility of Org
> syntax (think of org documents written years ago). Major changes to
> syntax are very unlikely even without considering third-party software.
> And, by the way, remember the existing "third party" Elisp packages
> (think of Org roam, for example). We do not want to break them.
>

Backwards compatibility is important and changes should never be done
lightly. However, that doesn't mean they don't occur (we have already
had breaking changes, so old org files are likely to have issues
already). Backwards compatibility can also become a burden and
sometimes, needs to be sacrificed to maintain the viability or
maintainability of the system. So while I totally agree we should work
very hard not to break compatibility or adversely affect other projects
which are built on top of org mode, like org-roam, we also don't want to
find ourselves in a position where we cannot improve/enhance org mode
because of the impact it has on other projects. The priority should
always be org-mode as an Emacs mode. When there is a need for a breaking
change, that needs to be managed in a way which provides other
dependent libraries and projects a reasonable time to adapt.

Having thought about this whole thread and other recent posts, I still
feel any concern or reference to third party libraries etc is misguided
or at the least, irrelevant. Most of the suggestions are fine and would
be beneficial to org mode (such as clearly defined, consistent and
documented syntax). The fact 3rd party libraries would also benefit from
this is a bonus, but largely irrelevant. I'm not convinced that the
perceived lack of such documentation or specification is actually the
impediment to a 3rd party org mode. I think the real problem and the
real reason you are unlikely to get a version of org-mode which is
popular for non-Emacs users (and would facilitate collaboration with
non-Emacs users) is because what makes org-mode so great has little to
do with the markup. The org-mode markup is no better or worse than other
'markdown' dialects. What makes org-mode such a great system is
intrinsically interwoven with Emacs and the facilities Emacs provides.
The amount of work which would be required in another editor to get even
close to the experience and benefits of org mode is simply too high. The
best you can hope for is some baic rendering and syntax
highlighting/font-locking, which is unlikely to be sufficient to make
people switch from the existing markdown they already use.

I think a far more likely scenario is that we will see some/many of the
ideas found in org-mode adapted and implemented in other editors, but
without concern for compatibility. This has little to do with Emacs
org-mode's documentation or org-modes specification, but rather is about
how the ideas which are encapsulated in org-mode can be implemented in
other systems and within the limitations of those systems. I'm actually
surprised there hasn't been more org-mode clones already, but that could
be a reflection of the amount of work it would take to create something
which wasn't just another markdown module that renders a reasonable
HTML/PDF version of it's contents. .

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]


Tim Cross <theophilusx@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> Meanwhile, Emacs development continues and new features/capabilities
>>> continue to be added. In particular, a new feature is added which is
>>> extremely powerful and would be a huge benefit for Emacs org-mode users.
>>> However, there is a problem. In order to take advantage of this new
>>> feature, significant changes are required for the specification. This
>>> will result in implementations requiring considerable work in order to
>>> update them to the new specification.
>>
>> I disagree. We already need to care about back-compatibility of Org
>> syntax (think of org documents written years ago). Major changes to
>> syntax are very unlikely even without considering third-party software.
>> And, by the way, remember the existing "third party" Elisp packages
>> (think of Org roam, for example). We do not want to break them.
>>
>
> Backwards compatibility is important and changes should never be done
> lightly. However, that doesn't mean they don't occur (we have already
> had breaking changes, so old org files are likely to have issues
> already). Backwards compatibility can also become a burden and

I already spent several hours fixing old presentations, because of org
format changes, so I want to put in a strong vote for backwards
compatibility.

If you have 1400 slides of lectures, all carefully laid out to convey
information as best as possible, and you realize a few days before the
lecture when you want to update them that the layout is broken, because
of some minor change in interpretation of empty headlines in org-beamer
export so you have to go over each slide individually to make sure that
nothing is cut off and no layout is broken — and check the compile to
latex many times until the layout is working again — that is a huge
cost.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Wed, Dec 08, 2021 at 05:16:20PM +0100, Dr. Arne Babenhauserheide wrote:
>
> Tim Cross <theophilusx@gmail.com> writes:
>
> > Backwards compatibility is important and changes should never be done
> > lightly. However, that doesn't mean they don't occur (we have already
> > had breaking changes, so old org files are likely to have issues
> > already). Backwards compatibility can also become a burden and
>
> I already spent several hours fixing old presentations, because of org
> format changes, so I want to put in a strong vote for backwards
> compatibility.

I agree completely. Luckily org-lint provides great insights into
changes. Reading the release notes between major versions is a good
idea. I have found that anytime I've had a problem it was well
documented in the release notes, and that I simply neglected to read
them.

> If you have 1400 slides of lectures, all carefully laid out to convey
> information as best as possible, and you realize a few days before the
> lecture when you want to update them that the layout is broken, because
> of some minor change in interpretation of empty headlines in org-beamer
> export so you have to go over each slide individually to make sure that
> nothing is cut off and no layout is broken — and check the compile to
> latex many times until the layout is working again — that is a huge
> cost.

I don't see this as much different from the issues encountered with
compiling code with libraries. During development you have to freeze
libraries you're working against. After an update, you'll have to
check again.

I've had this come up in my professional documents on occasion, and
I've developed habits to help. For instance:

 - Every file gets an export header template and all settings are done
   there.

 - Exported documents must never depend on variables in my
   init.el. All variables must be stored as file local variables if
   they required customization against Org defaults.

 - I run org-lint first if I suspect a problem.

 - I pay latex experts to make my templates so I don't have to.

 - Anything outside of basic Org syntax, tables and source blocks I do
   directly in latex. Images are a good example. I will use latex code
   for the image, sizing, orientation, etc instead of relying on Org's
   extended syntax for image links, caption, and attributes.

As a result my publishing has been pretty consistent for customer
documents. I also only update my Org between projects. ;]

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 4578 bytes --]


Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> On Wed, Dec 08, 2021 at 05:16:20PM +0100, Dr. Arne Babenhauserheide wrote:
>>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>> > Backwards compatibility is important and changes should never be done
>> > lightly. However, that doesn't mean they don't occur (we have already
>> > had breaking changes, so old org files are likely to have issues
>> > already). Backwards compatibility can also become a burden and
>>
>> I already spent several hours fixing old presentations, because of org
>> format changes, so I want to put in a strong vote for backwards
>> compatibility.
>
> I agree completely. Luckily org-lint provides great insights into
> changes. Reading the release notes between major versions is a good
> idea. I have found that anytime I've had a problem it was well
> documented in the release notes, and that I simply neglected to read
> them.
>
>> If you have 1400 slides of lectures, all carefully laid out to convey
>> information as best as possible, and you realize a few days before the
>> lecture when you want to update them that the layout is broken, because
>> of some minor change in interpretation of empty headlines in org-beamer
>> export so you have to go over each slide individually to make sure that
>> nothing is cut off and no layout is broken — and check the compile to
>> latex many times until the layout is working again — that is a huge
>> cost.
>
> I don't see this as much different from the issues encountered with
> compiling code with libraries. During development you have to freeze
> libraries you're working against. After an update, you'll have to
> check again.
>
> I've had this come up in my professional documents on occasion, and
> I've developed habits to help. For instance:
>
>  - Every file gets an export header template and all settings are done
>    there.
>
>  - Exported documents must never depend on variables in my
>    init.el. All variables must be stored as file local variables if
>    they required customization against Org defaults.

I actually have separate .emacs.d folders and autotools setups for most
of my org-mode projects. But that’s to separate it from my emacs setup,
not as protection against a potentially volatile¹ document format.

>  - I run org-lint first if I suspect a problem.
>
>  - I pay latex experts to make my templates so I don't have to.
>
>  - Anything outside of basic Org syntax, tables and source blocks I do
>    directly in latex. Images are a good example. I will use latex code
>    for the image, sizing, orientation, etc instead of relying on Org's
>    extended syntax for image links, caption, and attributes.

> As a result my publishing has been pretty consistent for customer
> documents. I also only update my Org between projects. ;]

If I had needed a stronger argument for more backwards compatibility,
this list of habits is it. That should not be required to keep your
org-mode documents working.

I use org-mode for a lecture I give as small side-job because I like
teaching. It is not my main job.

And I use org-mode for hobby-projects. Yes, the hobby project is a 400
page RPG rulebook, but it is still a hobby project.

And I use org-mode to publish my personal website (also as a Hobby),
with about 150% that size.

And my projects do not end. Some of these documents are already in use
for a decade.

Org-mode is not just a library, it keeps user-data. It should really not
be volatile¹.

If I can’t trust org-mode to keep working but have to check the
documents every time I come back to them — and might have to spend hours
fixing them — then it not suitable for writing, as much as that would
pain me (because it would cast into doubt most of my decisions around
writing of the past decade).

To date, I only had a bigger problem once (and that hurt a lot, because
it was just before giving a lecture, so I had to ditch most of the
improvements I wanted to do and instead spend all the time fixing the
document), but the talk here about “sometimes you have to break
compatibility” goes into a direction I consider as very dangerous.

Please do not make org-mode volatile.¹

Org-Mode and Emacs have mostly been stable the past 15 years. And it is
good to be stable; a strength that is highlighted much too seldomly.

Best wishes,
Arne

¹: https://stevelosh.com/blog/2012/04/volatile-software/
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Wed, Dec 08, 2021 at 08:22:31PM +0100, Dr. Arne Babenhauserheide wrote:
> >  - Anything outside of basic Org syntax, tables and source blocks I do
> >    directly in latex. Images are a good example. I will use latex code
> >    for the image, sizing, orientation, etc instead of relying on Org's
> >    extended syntax for image links, caption, and attributes.
>
> > As a result my publishing has been pretty consistent for customer
> > documents. I also only update my Org between projects. ;]
>
> If I had needed a stronger argument for more backwards compatibility,
> this list of habits is it. That should not be required to keep your
> org-mode documents working.

I think this may be a problem regarding expectations.

I expect Org to be great at handling it's own format, and to give me
the editing experience within Emacs that I have come to expect.

That Org can also be used to export to other formats is both a
blessing and a curse. Org can only do high level constructs in the
languages it exports to, and really should only be expected to do just
that. It's a paper thin macro or template over a much more complicated
document language.

Org's lightweight markup has had things bolted onto it repeatedly for
years. Typically issues have resulted in changes in the export engine
defaults (ie: html moving to using css), and not Org itself changing
the editing experience in Emacs.

> Org-mode is not just a library, it keeps user-data. It should really not
> be volatile¹.

Org's format isn't volatile. You could view those anytime in Org and
see what you expect to see. The issue you are having is that an old
document may not export perfectly over time.

What if Org didn't diverge, the underlying format did?

> If I can’t trust org-mode to keep working but have to check the
> documents every time I come back to them — and might have to spend hours
> fixing them — then it not suitable for writing, as much as that would
> pain me (because it would cast into doubt most of my decisions around
> writing of the past decade).

You can absolutely trust Org to open, view, and edit it's own files
even decades old. It's plain text, so there's no risk in experiencing
a permanent loss of data.

The exporting is the difference in expectations. Org's lightweight
markup is quite simple, and the documents it produces should be as
well. This is much like the original HTML specification. Look how
complicated it is to write HTML now with CSS and Javascript emulating
mundane functions after decades of bolt on "standards".

If I had a document which had a highly sensitive output format which
had to remain perfect over decades, I would argue that perhaps Org
wasn't the correct markup to write it in.

Much like plain text vs original simple HTML, vs Latex. Text was plain
and simple, with little formatting. Durable and ugly at times, but
always legible. The original HTML had more markup required, but it was
hyperlinks and some simple fonts and formats. Prettier, variable
fonts, colors, pictures. Latex can make pixel perfect PDFs with
multiple medias and professional results, however it has a very
specific format and this may be poor for writing in dynamically. HTML
required decades of tweaks to become "pixel perfect", and HTML a
decade old rarely renders properly in a "modern" browser.

At some point with each of these languages, the formatting became more
important than the content.

I write all my customer documentation in Org, with custom Latex
templates. I've only had to make major changes once, I think between
v8 and v9. Yes, my old documents won't export identically without the
changes. The likelihood they still export is high, and 100% that I can
view and edit them correctly in Org. It's only the polished result
which could degrade. I may have to tweak them to make them export the
same way again, but I expect they can without too much effort. I'm OK
with that.

> Please do not make org-mode volatile.¹

I think our maintainers have done an excellent job of minimizing the
impact of any changes. However when changes are needed, I trust their
judgement to have good reason to make a change and document it
thoroughly.

However I only export Org to be backwardly compatible with itself, not
the languages it makes exports to.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> [[PGP Signed Part:Undecided]]
>
> Russell Adams <RLAdams@AdamsInfoServ.Com> writes:
>
>> On Wed, Dec 08, 2021 at 05:16:20PM +0100, Dr. Arne Babenhauserheide wrote:
>>>
>>> Tim Cross <theophilusx@gmail.com> writes:
>>>
> To date, I only had a bigger problem once (and that hurt a lot, because
> it was just before giving a lecture, so I had to ditch most of the
> improvements I wanted to do and instead spend all the time fixing the
> document), but the talk here about “sometimes you have to break
> compatibility” goes into a direction I consider as very dangerous.
>
> Please do not make org-mode volatile.¹
>
> Org-Mode and Emacs have mostly been stable the past 15 years. And it is
> good to be stable; a strength that is highlighted much too seldomly.
>

Nobody is suggesting we make org-mode volatile. However, it expect that
there will never be breaking change is idealistic. I cannot think of a
single piece of software which hasn't at some point had some level of
breaking change. 

As I stated in my post, backwards compatibility is important and no
breaking change should be taken lightly. However, at times, it is
necessary and cannot be avoided. It might even be outside org-mode's
control - for example, a breaking change in Emacs might result in the
need for a change in org-mode or a security vulnerability might be
discovered which cannot be fixed without a breaking change.

Change is inevitable. It cannot be prevented. All we can do is try to
mitigate the impact of that change as best we can. Of course you also
have the choice to avoid such changes simply by not upgrading. While
this will mean you don't get bug fixes or enhancements, it is really the
only way to guarantee your documents are not impacted.

I think org-mode has a pretty good track record. There have been
breaking changes, but those changes have been in the main, justified and
never done lightly. They have bene documented and included in the NEWS
file and org has provided tools like rog-lint and conversion functions
to help with the transition required for such change.

Change is inevitable and sometimes, breaking change cannot be avoided.
It is a fact of life we have to deal with. As developers, we need to try
and ensure the impact from change is as minimal as possible and when it
is inevitable, we implement the change in a planned manner which tries
to reduce that impact (communication, conversion facilities and
conversion functions, stage implementation, deprecation periods etc).

What really doesn't help is to immediately jump to extremes and start
talking about making something volatile just because change is
mentioned.

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> On Wed, Dec 08, 2021 at 08:22:31PM +0100, Dr. Arne Babenhauserheide wrote:
>> >  - Anything outside of basic Org syntax, tables and source blocks I do
>> >    directly in latex. Images are a good example. I will use latex code
>> >    for the image, sizing, orientation, etc instead of relying on Org's
>> >    extended syntax for image links, caption, and attributes.
>>
>> > As a result my publishing has been pretty consistent for customer
>> > documents. I also only update my Org between projects. ;]
>>
>> If I had needed a stronger argument for more backwards compatibility,
>> this list of habits is it. That should not be required to keep your
>> org-mode documents working.
>
> I think this may be a problem regarding expectations.
>
> I expect Org to be great at handling it's own format, and to give me
> the editing experience within Emacs that I have come to expect.
>
> That Org can also be used to export to other formats is both a
> blessing and a curse. Org can only do high level constructs in the
> languages it exports to, and really should only be expected to do just
> that. It's a paper thin macro or template over a much more complicated
> document language.
>
> Org's lightweight markup has had things bolted onto it repeatedly for
> years. Typically issues have resulted in changes in the export engine
> defaults (ie: html moving to using css), and not Org itself changing
> the editing experience in Emacs.
>
>> Org-mode is not just a library, it keeps user-data. It should really not
>> be volatile¹.
>
> Org's format isn't volatile. You could view those anytime in Org and
> see what you expect to see. The issue you are having is that an old
> document may not export perfectly over time.
>
> What if Org didn't diverge, the underlying format did?
>
>> If I can’t trust org-mode to keep working but have to check the
>> documents every time I come back to them — and might have to spend hours
>> fixing them — then it not suitable for writing, as much as that would
>> pain me (because it would cast into doubt most of my decisions around
>> writing of the past decade).
>
> You can absolutely trust Org to open, view, and edit it's own files
> even decades old. It's plain text, so there's no risk in experiencing
> a permanent loss of data.
>
> The exporting is the difference in expectations. Org's lightweight
> markup is quite simple, and the documents it produces should be as
> well. This is much like the original HTML specification. Look how
> complicated it is to write HTML now with CSS and Javascript emulating
> mundane functions after decades of bolt on "standards".
>
> If I had a document which had a highly sensitive output format which
> had to remain perfect over decades, I would argue that perhaps Org
> wasn't the correct markup to write it in.
>
> Much like plain text vs original simple HTML, vs Latex. Text was plain
> and simple, with little formatting. Durable and ugly at times, but
> always legible. The original HTML had more markup required, but it was
> hyperlinks and some simple fonts and formats. Prettier, variable
> fonts, colors, pictures. Latex can make pixel perfect PDFs with
> multiple medias and professional results, however it has a very
> specific format and this may be poor for writing in dynamically. HTML
> required decades of tweaks to become "pixel perfect", and HTML a
> decade old rarely renders properly in a "modern" browser.
>
> At some point with each of these languages, the formatting became more
> important than the content.
>
> I write all my customer documentation in Org, with custom Latex
> templates. I've only had to make major changes once, I think between
> v8 and v9. Yes, my old documents won't export identically without the
> changes. The likelihood they still export is high, and 100% that I can
> view and edit them correctly in Org. It's only the polished result
> which could degrade. I may have to tweak them to make them export the
> same way again, but I expect they can without too much effort. I'm OK
> with that.
>
>> Please do not make org-mode volatile.¹
>
> I think our maintainers have done an excellent job of minimizing the
> impact of any changes. However when changes are needed, I trust their
> judgement to have good reason to make a change and document it
> thoroughly.
>
> However I only export Org to be backwardly compatible with itself, not
> the languages it makes exports to.
>

Those are excellent points and highlight the fact much of what org does
is not always under the control of org. As you point out, the HTML
specification has changed a lot in the last 30 years. It use to be
'standard' to use upper case for HTML tags and closing tags were not
always required. However, html5 requires tags in lower case and now
expects closing tags.

Irony here is we are dealing with a mediam which is inherently
susceptible to change. Talk to any archivist and they will tell you
about the huge challenges they 'digital age' has created for them. They
will tell you about all the data they have stored in various formats
which either cannot be read (there are miles of tapes out there where
you cannot even find a hardware device capable of reading them) or which
have degraded too much. The life expectancy of digital media is far less
than printed media. Much of what large archiving sites do these days is
constantly copying digital artefacts from one bit of media to another.
Often, this isn't even because of format change, but simply due to the
way digital data degrades over time. I'm still amazed when people seem
shocked when I tell them that those CDs or DVDs they burnt with all that
gigabytes of critical data will begin to degrade after about 5 years and
after 10 years you will see increasing instances of data corruption. In
some instances, critical digital data is still printed to paper because
the printed format still has a much, much longer shelf life than digital
data. We can still read books printed by Gutenberg, Meanwhile, there
are disks of documents you saved in 2000 which will likely have access
problems, assuming you still have a device which can read that format.

As you point out, the big benefit of org mode is that the files are
plain text. This means you will always be able to 'fix' any issues which
arise from change. It might not be convenient and you may be frustrated
by such change, but you will likely have a much better outcome than you
would with any other document formatting system which is not based on
plain text. 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]


Tim Cross <theophilusx@gmail.com> writes:

> What really doesn't help is to immediately jump to extremes and start
> talking about making something volatile just because change is
> mentioned.

I am wording this so strongly because we currently have talk about
creating more abstract org syntax.

This is the situation in which the temptation to skip backwards
compatibility is highest — as is the cost of that, because not updating
will quickly not be an option (because dependencies will follow).

In another situation I would be much more relaxed about this discussion,
but when larger refactoring is on the table, it is important that
backwards compatibility is high in the priorities.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2603 bytes --]


Tim Cross <theophilusx@gmail.com> writes:



> Russell Adams <RLAdams@AdamsInfoServ.Com> writes:
>> That Org can also be used to export to other formats is both a
>> blessing and a curse. Org can only do high level constructs in the
>> languages it exports to, and really should only be expected to do just
>> that. It's a paper thin macro or template over a much more complicated
>> document language.
…
>> The exporting is the difference in expectations. Org's lightweight
>> markup is quite simple, and the documents it produces should be as
>> well. This is much like the original HTML specification. Look how
>> complicated it is to write HTML now with CSS and Javascript emulating
>> mundane functions after decades of bolt on "standards".


Yes, the language-specific hacks are outside its scope. When I do
#+latex: …
or
@@html:…@@

I am in the export formats.

What should not happen is that 

>>> Please do not make org-mode volatile.¹
>> …
>> I think our maintainers have done an excellent job of minimizing the
>> impact of any changes.

Yes, they have. That’s why I wrote that it is a too rarely highlighted
strength, that Emacs is stable.

>> However I only export Org to be backwardly compatible with itself, not
>> the languages it makes exports to.

One of the big strengths of org-mode is in its integrations. When the
languages change that it exports to, it cannot do much. But where they
don’t, an update of org-mode should not break the export.

From the expectations-side, that’s the extended 80/20 rule: 80% of your
users only use 20% of the features¹, but they don’t all use the same
20%. So everytime you break something that’s not in the core 20%, you
lose some users until only a small core is left that actually did not
use anything outside those 20%.

¹: for org-mode it migh rather be 5% :-)

> As you point out, the big benefit of org mode is that the files are
> plain text. This means you will always be able to 'fix' any issues which
> arise from change. It might not be convenient and you may be frustrated
> by such change, but you will likely have a much better outcome than you
> would with any other document formatting system which is not based on
> plain text. 

Yes — that’s also one of the reasons why I’m using org-mode.

It’s awesome integrations and tooling are why I use org-mode instead of
markdown or asciidoc or similar. It binds all my work in Emacs together.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

Hi Arne,

> I am wording this so strongly because we currently have talk about
> creating more abstract org syntax.
>
> This is the situation in which the temptation to skip backwards
> compatibility is highest — as is the cost of that, because not updating
> will quickly not be an option (because dependencies will follow).
>
> In another situation I would be much more relaxed about this discussion,
> but when larger refactoring is on the table, it is important that
> backwards compatibility is high in the priorities.

For the sake of staying vaguely on-track, I think it’s worth noting that Ihor’s
comments make no mention of changing the Org syntax, or creating an abstract
definition (that has existed as a WIP for years).

There’s been a bit too much speculation[1] in this thread methinks…

All the best,
Timothy



Footnotes
─────────

[1] Don’t get me started on speculations
building on speculation.

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Timothy <tecosaur@gmail.com> writes:

> For the sake of staying vaguely on-track, I think it’s worth noting that Ihor’s
> comments make no mention of changing the Org syntax, or creating an abstract
> definition (that has existed as a WIP for years).

I think Dr. Babenhauser referred to another ongoing thread "Raw Org AST
snippets for "impossible" markup".

> There’s been a bit too much speculation[1] in this thread methinks…

Yep, though it is still fairly relevant to some of the interpretations
of the thread title. So, let it be. The discussion is not useless even
though it is not the feedback I was counting on.

I plan to open a new, more technical thread to discuss the changes I
wanted to make.

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Eric S Fraga]

On Wednesday,  8 Dec 2021 at 17:16, Dr. Arne Babenhauserheide wrote:
> If you have 1400 slides of lectures, all carefully laid out to convey

Been there, done that. 😞 I've learned to not upgrade org (or Emacs) for
a few weeks before the start of term!

I am a big fan of backwards compatibility but not if it becomes a
stranglehold on significant improvements.  Org v8 in particular was a
major step forward but broke many of my org files.  I was happy with the
trade-off: immediate pain but much better basis for future developments.

-- 
: Eric S Fraga, with org release_9.5.1-254-g14ed65 in Emacs 29.0.50
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Russell Adams]

On Thu, Dec 09, 2021 at 10:46:34AM +0000, Eric S Fraga wrote:
> Org v8 in particular was a major step forward but broke many of my
> org files.

I know we're beating a dead horse, but can you clarify.

Did Org break your Org editing experience in Emacs for your Org files,
or did this change just break some of the finer formatting details of
your exported Org file?

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Eric S Fraga]

On Thursday,  9 Dec 2021 at 16:21, Russell Adams wrote:
> Did Org break your Org editing experience in Emacs for your Org files,
> or did this change just break some of the finer formatting details of
> your exported Org file?

It's been a while but, IIRC, the latter to a large extent; I should have
been more clear.

However, there have been changes in the past that affected the editing
experience but I cannot remember if they were with v8 or v9 or somewhere
in between, particularly dealing with locations of :LOGBOOK: and
scheduling directives.  These happened when Nikolas cleaned up the
parsing, to the benefit of org but at the cost of making some documents
non-compliant.

Nevertheless, as others have stated, given the stability provided by
Emacs (parts of my .emacs date back to 1985!), and org being all text,
even breaking changes are usually trivial to correct or adjust to.

-- 
: Eric S Fraga, with org release_9.5.1-243-gad53c5 in Emacs 29.0.50
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Samuel Wales]

i think the big change was v9.

On 12/9/21, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> On Thursday,  9 Dec 2021 at 16:21, Russell Adams wrote:
>> Did Org break your Org editing experience in Emacs for your Org files,
>> or did this change just break some of the finer formatting details of
>> your exported Org file?
>
> It's been a while but, IIRC, the latter to a large extent; I should have
> been more clear.
>
> However, there have been changes in the past that affected the editing
> experience but I cannot remember if they were with v8 or v9 or somewhere
> in between, particularly dealing with locations of :LOGBOOK: and
> scheduling directives.  These happened when Nikolas cleaned up the
> parsing, to the benefit of org but at the cost of making some documents
> non-compliant.
>
> Nevertheless, as others have stated, given the stability provided by
> Emacs (parts of my .emacs date back to 1985!), and org being all text,
> even breaking changes are usually trivial to correct or adjust to.
>
> --
> : Eric S Fraga, with org release_9.5.1-243-gad53c5 in Emacs 29.0.50
> : Latest paper written in org: https://arxiv.org/abs/2106.05096
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]


Russell Adams <RLAdams@AdamsInfoServ.Com> writes:

> Did Org break your Org editing experience in Emacs for your Org files,
> or did this change just break some of the finer formatting details of
> your exported Org file?

The change to electric indent broke my workflow badly (always having to
undo the indentation after every new headline), and it took long until I
found out how to avoid that.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

> [[PGP Signed Part:Undecided]]
>
> Russell Adams <RLAdams@AdamsInfoServ.Com> writes:
>
>> Did Org break your Org editing experience in Emacs for your Org files,
>> or did this change just break some of the finer formatting details of
>> your exported Org file?
>
> The change to electric indent broke my workflow badly (always having to
> undo the indentation after every new headline), and it took long until I
> found out how to avoid that.
>

That is probably a good example of how change can be imposed by external
events outside the control of org-mode. While I would agree that more
analysis of that change may have resulted in better initial
documentation and in turn, less inconvenience to users. that is only
obvious now with the benefit of hindsight. The fact it was a change
triggered by a change in Emacs rather than a change initiated by org
demonstrates that at times, org has to adapt to its evolving
environment. While this change may have 'broken' your workflow, the
previous behaviour was breaking other users workflows because org did
not honour electric-indent-mode and was not consistent with other core
emacs modes. 

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]


Tim Cross <theophilusx@gmail.com> writes:

> "Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:
>> The change to electric indent broke my workflow badly (always having to
>> undo the indentation after every new headline), and it took long until I
>> found out how to avoid that.
> environment. While this change may have 'broken' your workflow, the
> previous behaviour was breaking other users workflows because org did
> not honour electric-indent-mode and was not consistent with other core
> emacs modes. 

This stretches the definition of workflow. Other users had a less
consistent workflow, but since orgmode had never done electric indent,
they did not have a workflow requiring electric indent in org which
could be broken by the existing behaviour.

This is rather an example where going for consistency over stability
creates problems.

(but still: that happened once; emacs is not perfectly stable, but when
staying close to vanilla, it is mostly stable)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> ... while I totally agree we should work
> very hard not to break compatibility or adversely affect other projects
> which are built on top of org mode, like org-roam, we also don't want to
> find ourselves in a position where we cannot improve/enhance org mode
> because of the impact it has on other projects.

Well. We have no direct control on the other projects. However, not
doing anything about the fact that other project keep appearing is
nothing but a call for more compatibility issues. If we do not clearly
specify relatively stable syntax or API, the other projects will
inevitably use internal implementation details and could be broken more
easily. For example, my recent patch to org-element broke org-roam
because org-roam relied on some undocumented behaviour of
org-element-at-point.

> Having thought about this whole thread and other recent posts, I still
> feel any concern or reference to third party libraries etc is misguided
> or at the least, irrelevant. Most of the suggestions are fine and would
> be beneficial to org mode (such as clearly defined, consistent and
> documented syntax). The fact 3rd party libraries would also benefit from
> this is a bonus, but largely irrelevant.

You read "Org mode third-party integration" as benefit for third-party
libraries. I read it as benefit for Org from third-party libraries (as
opposed to no benefit and potential complains from third-party library
users).

> I think a far more likely scenario is that we will see some/many of the
> ideas found in org-mode adapted and implemented in other editors, but
> without concern for compatibility. This has little to do with Emacs
> org-mode's documentation or org-modes specification, but rather is about
> how the ideas which are encapsulated in org-mode can be implemented in
> other systems and within the limitations of those systems. I'm actually
> surprised there hasn't been more org-mode clones already, but that could
> be a reflection of the amount of work it would take to create something
> which wasn't just another markdown module that renders a reasonable
> HTML/PDF version of it's contents. .

There are some "clones" like smos. However, org-mode is nothing but a
compilation of existing ideas. There are many other (mostly proprietary)
tools implementing parts of org's functionality: roam research, notion,
evernote, wunderlist, zettelkasten (app), hypothes.is,
ipython, Mathematica, taskwarrior, remember the milk, Doug Engelbart’s
ideas, etc. Even the damned Microsoft Word has built-in outliner (don't
ask how I know).

Best,
Ihor

Re: Concrete suggestions to improve Org mode third-party integration :: an afterthought following Karl Voit's Orgdown proposal

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> ... while I totally agree we should work
>> very hard not to break compatibility or adversely affect other projects
>> which are built on top of org mode, like org-roam, we also don't want to
>> find ourselves in a position where we cannot improve/enhance org mode
>> because of the impact it has on other projects.
>
> Well. We have no direct control on the other projects. However, not
> doing anything about the fact that other project keep appearing is
> nothing but a call for more compatibility issues. If we do not clearly
> specify relatively stable syntax or API, the other projects will
> inevitably use internal implementation details and could be broken more
> easily. For example, my recent patch to org-element broke org-roam
> because org-roam relied on some undocumented behaviour of
> org-element-at-point.
>
>> Having thought about this whole thread and other recent posts, I still
>> feel any concern or reference to third party libraries etc is misguided
>> or at the least, irrelevant. Most of the suggestions are fine and would
>> be beneficial to org mode (such as clearly defined, consistent and
>> documented syntax). The fact 3rd party libraries would also benefit from
>> this is a bonus, but largely irrelevant.
>
> You read "Org mode third-party integration" as benefit for third-party
> libraries. I read it as benefit for Org from third-party libraries (as
> opposed to no benefit and potential complains from third-party library
> users).
>

I think 3rd party libraries are a benefit to org users. On the whole,
they are of no direct benefit to org-mode (if they are, then they are a
good candidate for inclusion into org mode). The relationship is very
similar to what Emacs has with all the external packages which are not
part of Emacs. There is no direct benefit to Emacs from all these
packages, but there is huge benefit for the Emacs community. Emacs
changes and evolves as necessary to keep it relevant, maintainable and
up-to-date with user expectations. At times, it will make changes that
are breaking on 3rd party libraries or require 3rd party libraries to
update/modify how they interact with Emacs. These changes are not done
lightly and are done so as to minimise impact to these 3rd parties.
However, Emacs does not concern itself with 3rd party libraries - it
focuses on making Emacs better and leaves 3rd party libraries to
themselves.

Placing too much focus on 3rd party libraries runs the risk of the tail
wagging the dog. Org mode should focus on what org-mode needs to do to
be performant, maintainable and minimise bugs. Having clear syntax
specifications, good unit tests, a clear and consistent API and
documentation all contribute to org-mode stability and maintainability.
Coincidentally, these are also the things which will most benefit 3rd
party libraries. However, should there be some justified reason to
change the syntax or the API in order to improve performance,
or maintainability, we should not feel constrained from doing so because
it will impact on 3rd party libraries. Instead, we should make such
changes in a staged manner and provide a reasonable time for 3rd parties
to be updated to work with the changes and only introduce breaking
change in new major versions.