Re: Org mode and Emacs

Re: Org mode and Emacs

[Payas Relekar]

Hi Bastien,

Bastien <bzg@gnu.org> writes:

> Here is the email I sent when I accepted to switch to using .org as
> the native format for Org documentation:
>
> https://list.orgmode.org/orgmode/87371gfas7.fsf@bzg.fr/
>
> As you can read, I wanted to make the switch as an experiment to see
> if we were really solving a problem here.
>
> I believe we didn't get more contributions to the manual by switching
> to .org, so I'd be in favor of switching back to using .texi as the
> native format for Org's documentation.  (Not for 9.6, obviously, more
> probably for 10.0 -- I'll discuss this with other Org maintainers.)

From your mail, below were the motivators for change:

- Let's stabilize editing standards around the org.org file.

- Let's test org capabilities against a giant .org file.

- Let's make `C-x 4 a' do something useful in an .org section.

- Let's write more non-emacs parsers and exporters.

- Let's see if we have more contributions to the manual and if
  we really solved a problem here.

While you're best to judge the number of contributions, #1 and #2, or
the dogfooding opportunities provided by the switch are immense.

One doesn't occasionally run into org documents the size of org.org. It
has already resulted in gc enhancements as it was slowing down Emacs
build and was optimized. I'll say that alone is a benefit worth keeping.

There is also more progress being made on non-emacs parsers[0][1], and
perhaps we can reach out if they find org.org useful. Generally
speaking, these projects are even smaller (number of contributors-wise)
than org-mode, but a shared burden is always nicer.

[0]: https://github.com/nvim-orgmode/orgmode
[1]: https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf

> The other topics in this thread (make Org's Texinfo exporter provide
> good .texi manuals, make Org more modular, etc.) are interesting, but
> they are really separate questions IMHO.

Thanks,
Payas
--

Re: Org mode and Emacs

[Bastien]

Hi Payas,

Payas Relekar <relekarpayas@gmail.com> writes:

> From your mail, below were the motivators for change:

Re-reading it, the list was a mix of (1) things to be done for the
switch to be sensible (and that will be boosted if the switch happens)
and (2) possible nice outcomes.

> - Let's stabilize editing standards around the org.org file.

(=> 1) Something that we did before the switch.

> - Let's test org capabilities against a giant .org file.

(=> 2) Yes, this can lead to enhancements in Org like it did recently,
but I don't think this is a good reason enough to justify the switch.

> - Let's make `C-x 4 a' do something useful in an .org section.

(=> 1) Also something now available.

> - Let's write more non-emacs parsers and exporters.

(=> 2) This was an illusion: I don't think projects like Pandoc use
the org-manual.org file to test whether they are good parsers and
exporters.  For a good reason: nobody really needs to use Pandoc for
parsing/exporting the Org manual.

> - Let's see if we have more contributions to the manual and if
>   we really solved a problem here.

(=> 2) This didn't happen.

> While you're best to judge the number of contributions, #1 and #2, or
> the dogfooding opportunities provided by the switch are immense.
>
> One doesn't occasionally run into org documents the size of org.org. It
> has already resulted in gc enhancements as it was slowing down Emacs
> build and was optimized. I'll say that alone is a benefit worth keeping.

I'm not convinced: slowing down Emacs build to create opportunities
for enhancing the Org parser and exporter does not strike me as a good
reason.

Respecting the GNU standards about manuals ("The preferred document
format for the GNU system is the Texinfo formatting language.") and
the recent discussions provide good reasons for switching back to
.texi, if all maintainers agree.

> There is also more progress being made on non-emacs parsers[0][1], and
> perhaps we can reach out if they find org.org useful. Generally
> speaking, these projects are even smaller (number of contributors-wise)
> than org-mode, but a shared burden is always nicer.
>
> [0]: https://github.com/nvim-orgmode/orgmode
> [1]: https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf

Of course, feel free to contact them, but why would they want to try
solving the challenge for exporting org-manual.org?  They can create
any .org file with the complexity they desire if they need it.  Also,
independently from this discussion, _we_ should certainly provide an
example document containing alls things a parser/exporter should be
able to handle.

All best,

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

>> While you're best to judge the number of contributions, #1 and #2, or
>> the dogfooding opportunities provided by the switch are immense.
>>
>> One doesn't occasionally run into org documents the size of org.org. It
>> has already resulted in gc enhancements as it was slowing down Emacs
>> build and was optimized. I'll say that alone is a benefit worth keeping.
>
> I'm not convinced: slowing down Emacs build to create opportunities
> for enhancing the Org parser and exporter does not strike me as a good
> reason.

As the outcome of that thread, I managed to reduce org->texinfo
conversion down to 4sec. https://yhetil.org/emacs-devel/87czf9np98.fsf@localhost
I do not think that the overall impact on the build process is something
we need to worry about in this context.

> Respecting the GNU standards about manuals ("The preferred document
> format for the GNU system is the Texinfo formatting language.") and
> the recent discussions provide good reasons for switching back to
> .texi, if all maintainers agree.

Do note that RMS expressed interest in changing this standard to Org,
given that Org provides all the necessary tooling to produce
high-quality manual.
https://yhetil.org/emacs-devel/E1nzQh5-0001OB-22@fencepost.gnu.org

I also spawned a thread about this in Org ML.
https://list.orgmode.org/87k09frdsv.fsf@localhost/

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> As the outcome of that thread, I managed to reduce org->texinfo
> conversion down to 4sec. https://yhetil.org/emacs-devel/87czf9np98.fsf@localhost
> I do not think that the overall impact on the build process is something
> we need to worry about in this context.

Thanks a lot for working on this, this is a great achievement.

To be fair, the impact on the build process was not really my main
concern.  My concern is that Org does not respect the GNU standards.

>> Respecting the GNU standards about manuals ("The preferred document
>> format for the GNU system is the Texinfo formatting language.") and
>> the recent discussions provide good reasons for switching back to
>> .texi, if all maintainers agree.
>
> Do note that RMS expressed interest in changing this standard to Org,
> given that Org provides all the necessary tooling to produce
> high-quality manual.
> https://yhetil.org/emacs-devel/E1nzQh5-0001OB-22@fencepost.gnu.org

I'm aware of this possibility, but there are many "if" :)

Also, we can work toward this possibility without using .org for the
Org manual.

What do you think of my proposal to produce an example document,
demonstrating the Org syntax and its possibilities?  

It would surely help us test our own code (parser and exporters) and
help others write parsers/exporters (outside of Emacs).

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

> To be fair, the impact on the build process was not really my main
> concern.  My concern is that Org does not respect the GNU standards.

We do. Org does provide manual in Texinfo format. The fact that our
texinfo version is auto-generated is irrelevant wrt GNU Standards.

> What do you think of my proposal to produce an example document,
> demonstrating the Org syntax and its possibilities?  
>
> It would surely help us test our own code (parser and exporters) and
> help others write parsers/exporters (outside of Emacs).

See https://orgmode.org/list/87fsqzi4tw.fsf@localhost

P.S. Note that I am trying to keep discussion relevant to this
emacs-devel thread. More serious discussion about switching back from
.org to .texinfo should probably happen at Org ML.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien Guerry]

Bastien <bzg@gnu.org> writes:

> What do you think of my proposal to produce an example document,
> demonstrating the Org syntax and its possibilities?  

PS: I see that you raised a similar idea on the Org-mode mailing list,
I will follow-up there.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Bastien <bzg@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 25 Sep 2022 08:35:54 +0200
> 
> Respecting the GNU standards about manuals ("The preferred document
> format for the GNU system is the Texinfo formatting language.") and
> the recent discussions provide good reasons for switching back to
> .texi, if all maintainers agree.

I definitely agree (it makes it easier for me to contribute to the Org
manual).  But eventually it is the decision of the Org project.

Re: Org mode and Emacs

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Bastien <bzg@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 25 Sep 2022 08:35:54 +0200
>> 
>> Respecting the GNU standards about manuals ("The preferred document
>> format for the GNU system is the Texinfo formatting language.") and
>> the recent discussions provide good reasons for switching back to
>> .texi, if all maintainers agree.
>
> I definitely agree (it makes it easier for me to contribute to the Org
> manual).  But eventually it is the decision of the Org project.

and I would be exactly the opposite. I know and use org-mode. I don't
know and have never used texinfo despite over 25 years of Emacs
use. Having to learn another formatting solution just to contribute to
the formatting solution I use isn't going to be encouraging. 

The question I wonder about is where are we most likely to get the
majority of our contributions from, those who use org mode and know it
or those who don't and for those who use org-mode, how many will know texinfo? 

I know one of the original justifications was to see if it improved
contributions towards documentation and it appears this has not been the
case. However, I do wonder about that - I have certainly seen numerous
manual patches on the list and I wonder how many of those would not
occur if the patcher also needed to know texinfo? There may also be
other impediments which slows down contributions that are unrelated to
the documentation format (I still find determining if something is a
known issue or not and the state of progress to resolving it difficult
to track - not a criticism of the core maintainers, who I believe do an
incredible job. Real problem is the challenge of realising a better
process given the very very few core contributors available - basically
a resourcing challenge).  

At the end of the day, I think the dog food argument is
important. Having the manual in org format has seen a number of
improvements and does provide a good and most importantly large and used
example. Having a sample document which developers could use to verify
parsers etc would be a good addition, but the problem with such
documents is they tend not to be maintained and are not actively
used. There is huge value in having a large and reasonably complex
document which is being actively updated/enhanced and which is used in
the real world to produce documents in various formats which are also
actively read and used. It tends to be in active use of generated
documents we find more subtle issues, things which tend to be
missed in cursory scans of test documents.

Re: Org mode and Emacs

[Bastien]

Hi Tim,

Tim Cross <theophilusx@gmail.com> writes:

> The question I wonder about is where are we most likely to get the
> majority of our contributions from, those who use org mode and know it
> or those who don't and for those who use org-mode, how many will
> know texinfo?

Recruiting contributors for Org is also a way to recruit contributors
for the GNU project in general, which uses Texinfo as its standard
format for manuals.

For occasional fixes, I don't think the difference between the .texi
and .org format makes that much of a difference.

For substantial contributions, it probably does: but contributors of
these important changes are probably those for which this difference
can easily be overcome -- and *should* be overcome, because they are
also potential contributors for the GNU project.

> (I still find determining if something is a
> known issue or not and the state of progress to resolving it difficult
> to track

(FWIW I agree, that's the motivation behind my work on Woof!.)

> Real problem is the challenge of realising a better
> process given the very very few core contributors available - basically
> a resourcing challenge).

What we don't see so far is the contributors we lose because we use
.org as the format for the manual: Eli is one and there are probably
others.

> At the end of the day, I think the dog food argument is
> important. Having the manual in org format has seen a number of
> improvements and does provide a good and most importantly large and used
> example. Having a sample document which developers could use to verify
> parsers etc would be a good addition, but the problem with such
> documents is they tend not to be maintained and are not actively
> used. There is huge value in having a large and reasonably complex
> document which is being actively updated/enhanced and which is used in
> the real world to produce documents in various formats which are also
> actively read and used. It tends to be in active use of generated
> documents we find more subtle issues, things which tend to be
> missed in cursory scans of test documents.

Full disclosure: the dog food argument never convinced me.  Dog
fooding /per se/ never makes any sense, unless you motivate it with
another good reason.

I suspect our (lispian?) brains is fascinated by recursive stuff (a
rose is a rose is a rose) but this is something we should resist.

Anyway, I won't insist on this anymore, the decision will be that
of all Org core maintainers, of course.

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

> Hi Tim,
>
> Tim Cross <theophilusx@gmail.com> writes:
>
>> The question I wonder about is where are we most likely to get the
>> majority of our contributions from, those who use org mode and know it
>> or those who don't and for those who use org-mode, how many will
>> know texinfo?
>
> Recruiting contributors for Org is also a way to recruit contributors
> for the GNU project in general, which uses Texinfo as its standard
> format for manuals.

I am now more concerned about Org contributors. I am not sure if we have
a luxury to push our contributors into GNU project by forcing them to
learn texinfo in addition to Org (note that I do not say that we should
not encourage people to contribute to GNU; just that we should not
discourage people from becoming Org contributors in the first place).

>> Real problem is the challenge of realising a better
>> process given the very very few core contributors available - basically
>> a resourcing challenge).
>
> What we don't see so far is the contributors we lose because we use
> .org as the format for the manual: Eli is one and there are probably
> others.

I am wondering how many Emacs developers contributed to Org in the past.
In particular, before we switched away from .texi manual.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> I am now more concerned about Org contributors. I am not sure if we have
> a luxury to push our contributors into GNU project by forcing them to
> learn texinfo in addition to Org (note that I do not say that we should
> not encourage people to contribute to GNU; just that we should not
> discourage people from becoming Org contributors in the first place).

I will write to you off-list this week to try to find a way to define
our strategy here (beyond the sole topic of the manual).

I suggest we have an online event after the EmacsConf (Dec. 11th would
be good) to include the community in our decisions about setting up a
strategy for Org future maintenance and contributions.

> I am wondering how many Emacs developers contributed to Org in the past.
> In particular, before we switched away from .texi manual.

My impression is that these contributions have been small in size of
text modified but very useful and regular, at least as much useful as
the occasional ones received from Org casual contributors.

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

>> I am wondering how many Emacs developers contributed to Org in the past.
>> In particular, before we switched away from .texi manual.
>
> My impression is that these contributions have been small in size of
> text modified but very useful and regular, at least as much useful as
> the occasional ones received from Org casual contributors.

Hmm. I tried (in Emacs git repo), skipping contributions from Org
maintainers:

git shortlog -- doc/misc/org.texi (see below for shortened output)

The overwhelming majority of the commits are (1) typos; (2) texinfo
markup changes (not needed as long as we follow the conventions in
ox-texinfo); (3) copyright year updates; (4) some common wording
conventions, like using "GNU" or https in urls.

This was since 2007.

git shortlog -- doc/misc/org.org is since Feb 2021 (3 years)

Eli Zaretskii (1):
      <copyright>

Glenn Morris (3):
      <moving from texi to org source>
      <copyright>

Hanno Perrey (1):
      ; * doc/misc/org.org: fix capture context example

Stefan Kangas (14):
      <typos>
      Avoid treating number as an enum in the org manual
      Drop support for the dead third-party w3 package
      <merges>
      Recommend NonGNU ELPA over MELPA
      Delete broken link to Network Theory Ltd.
      ; Prefer HTTPS to HTTP in many URLs

Total 19 commits from Emacs contributors.

For the same 3 year period org.texi saw
git shortlog --since 2018 -- doc/misc/org.texi
21 commits.

As expected, we are no longer seeing texinfo markup fixes, except a
single work around a bug in Emacs Info command. All other types of the
contributions from Emacs devs are not stopping.

I conclude that .org version of the manual did not cause reduced
contributions from Emacs devs compared to .texi version. And I believe
that automated export is useful to avoid dealing with texinfo issues in
addition to our own .org issues we would have to address anyway.

-------------------

git shortlog -- doc/misc/org.texi

Alan Mackenzie (1):
      Expunge "allow" + infinitive without direct object from source and doc.

Andreas Schwab (2):
      <2 commits fixing texinfo markup>

Chong Yidong (3):
      * org.texi (Org Plot): Fix tags (Bug#3507).
      * org.texi (Workflow states, Agenda commands): Fix tags (Bug#3508).
      <1 merge>      

Eli Zaretskii (4):
      <3 commits fixing texinfo markup>
      <1 commit fixing typo>

Glenn Morris (57):
      <several commits moving files around Emacs repo>
      Update Back-Cover Text as per maintain.info.
      Change to GFDL 1.2.  Refer to license in Emacs manual.
      <typos>
      <copyright year updates>
      Peter Tury  <tury.peter at gmail.com>  (tiny change)
      Peter Tury  <tury.peter at gmail.com>  (tiny change)
      Nuke arch-tags.
      Standardize possessive apostrophe usage in manuals, docs, and comments Ref: http://lists.gnu.org/archive/html/emacs-devel/2012-02/msg00649.html
      <texinfo markup fixes>
      Standardize case of "Front-Cover Texts" in texi file permissions notices.
      <merges>
      Replace doc references to load-hooks
      Distribute the real source for some doc/misc manuals (bug#45143)

Juanma Barranquero (7):
      <typos>

Karl Berry (1):
      <texinfo markup fixes>

Lars Ingebrigtsen (1):
      <texinfo markup fixes>

Mario Lang (2):
      <typos>

Martin Rudalics (1):
      <texinfo markup>

Mauro Aranda (1):
      <texinfo markup>

Michael Albinus (6):
      Document remote file name syntax change
      <texinfo markup>
      * doc/misc/org.texi (Installation): Fix clone commands.

Miles Bader (2):
      <merges>

Paul Eggert (54):
     <spelling>
     <copyright years>
     <texinfo markup>
      Simplify use of current-time and friends.
      Fix some 24-hour time stamps in documentation.
      Modernize usage of 'macOS' in doc and comments
      Prefer HTTPS to FTP and HTTP in documentation
      <merges>
      Update some URLs

Radon Rosborough (1):
      Add early init file, stop package-initialize insertion

Stefan Kangas (2):
      ; Prefer https to http in more URLs
      Avoid recommending Google

Stefan Monnier (3):
      <merges>
      * src/print.c (syms_of_print) <print_quoted>: Set default to true

Stephen Eglen (1):
      <typos>

Ville Skyttä (1):
      <typos>


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Ihor Radchenko <yantar92@gmail.com>, Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 17:09:42 +0000
> 
> I conclude that .org version of the manual did not cause reduced
> contributions from Emacs devs compared to .texi version. And I believe
> that automated export is useful to avoid dealing with texinfo issues in
> addition to our own .org issues we would have to address anyway.

You are comparing negligibly small numbers of contributions, so any
conclusions from such comparison are questionable at best.

My personal conclusion is that Emacs developers don't contribute too
much to Org's documentation because Emacs developers who are used to
work on documentation seldom if ever use Org for any significant work.
So they don't have significant changes to contribute, except fixing
typos and markup.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> You are comparing negligibly small numbers of contributions, so any
> conclusions from such comparison are questionable at best.

> My personal conclusion is that Emacs developers don't contribute too
> much to Org's documentation because Emacs developers who are used to
> work on documentation seldom if ever use Org for any significant work.
> So they don't have significant changes to contribute, except fixing
> typos and markup.

I think that I need to clarify about the purpose of my "analysis" (I
know that data is not very conclusive, but that all we got).

I was looking into the idea that we can get more help from Emacs
contributors if we switch the format back from .org to .texi.
(Mostly, out of curiosity; and because it was hanging in my TODO list)

According to the above, I do not see that we can achieve such goal:

1. Absolute number of contributions to Org manual was never large (not
   by Org-only contrubutors)

2. Relative numbers, although hard to compare, also do not indicate that
   the switch would be helpful in any way.

Having said that, RMS officially asked Org team to work towards allowing
Org to become the new GNU documentation standard.
https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
So, the above idea was rather theoretical. Unless we disregard RMS
request, Org manual will be a good reference template to explore
real-life caveats of using .org as the true documentation source.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 18:49:53 +0000
> 
> I was looking into the idea that we can get more help from Emacs
> contributors if we switch the format back from .org to .texi.

I'd welcome such a change, FWIW.

> Having said that, RMS officially asked Org team to work towards allowing
> Org to become the new GNU documentation standard.
> https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
> So, the above idea was rather theoretical. Unless we disregard RMS
> request, Org manual will be a good reference template to explore
> real-life caveats of using .org as the true documentation source.

I don't know why Richard made that request.  From my POV, switching
from Texinfo to _any_ significantly different markup has the
significant disadvantage that people who routinely improve our manuals
will have to learn a new language, and that will likely lower their
motivation.

For better and for worse, Texinfo is our documentation system.  It is
a good system, and is actively developed (version 7.1 will be released
soon), so abandoning it for something that is less familiar to those
who take care of the documentation is a net loss, IMO.  We have enough
real problems on our hands to afford inventing new ones.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I was looking into the idea that we can get more help from Emacs
>> contributors if we switch the format back from .org to .texi.
>
> I'd welcome such a change, FWIW.

But how would Org benefit from it?
For good reasons or not, we have already switched to .org format.
And switching back now would require extra effort that needs to be justified.
(Even without taking into account RMS' request).

>> Having said that, RMS officially asked Org team to work towards allowing
>> Org to become the new GNU documentation standard.
>> https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
> ...
> I don't know why Richard made that request.  From my POV, switching
> from Texinfo to _any_ significantly different markup has the
> significant disadvantage that people who routinely improve our manuals
> will have to learn a new language, and that will likely lower their
> motivation.
>
> For better and for worse, Texinfo is our documentation system.  It is
> a good system, and is actively developed (version 7.1 will be released
> soon), so abandoning it for something that is less familiar to those
> who take care of the documentation is a net loss, IMO.  We have enough
> real problems on our hands to afford inventing new ones.

Feel free to reply to RMS' email. I am sure that he had good reasons to
make the official request. I am not in position to argue on his behalf.

If you prefer, the same email is also on emacs-devel:
https://yhetil.org/emacs-devel/E1ocmvz-0002iB-2M@fencepost.gnu.org/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 19:31:31 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I was looking into the idea that we can get more help from Emacs
> >> contributors if we switch the format back from .org to .texi.
> >
> > I'd welcome such a change, FWIW.
> 
> But how would Org benefit from it?

I explained that below this part in my email.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I'd welcome such a change, FWIW.
>> 
>> But how would Org benefit from it?
>
> I explained that below this part in my email.

I thought that you were referring to RMS' request.

>  I don't know why Richard made that request.  From my POV, switching
>  from Texinfo to _any_ significantly different markup has the
>  significant disadvantage that people who routinely improve our manuals
>  will have to learn a new language, and that will likely lower their
>  motivation.

>  For better and for worse, Texinfo is our documentation system.  It is
>  a good system, and is actively developed (version 7.1 will be released
>  soon), so abandoning it for something that is less familiar to those
>  who take care of the documentation is a net loss, IMO.  We have enough
>  real problems on our hands to afford inventing new ones.

This may be true in general, but not for Org development.
People contributing to the Org manual are expected to understand Org
markup. So, it is Texinfo that becomes the new language to learn and
that creates an extra contribution barrier.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 09:04:38 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > I'd welcome such a change, FWIW.
> >> 
> >> But how would Org benefit from it?
> >
> > I explained that below this part in my email.
> 
> I thought that you were referring to RMS' request.

I was.  But Richard reads this list, so he will see what I wrote and
respond if he wants to.

> >  I don't know why Richard made that request.  From my POV, switching
> >  from Texinfo to _any_ significantly different markup has the
> >  significant disadvantage that people who routinely improve our manuals
> >  will have to learn a new language, and that will likely lower their
> >  motivation.
> 
> >  For better and for worse, Texinfo is our documentation system.  It is
> >  a good system, and is actively developed (version 7.1 will be released
> >  soon), so abandoning it for something that is less familiar to those
> >  who take care of the documentation is a net loss, IMO.  We have enough
> >  real problems on our hands to afford inventing new ones.
> 
> This may be true in general, but not for Org development.
> People contributing to the Org manual are expected to understand Org
> markup. So, it is Texinfo that becomes the new language to learn and
> that creates an extra contribution barrier.

If the idea is that the main contributors to the Org documentation are
Org users, and the rest of Emacs developers don't really count for
this purpose, then why did you count the contributions of the latter
to begin with? they should be immaterial for you, AFAIU.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> This may be true in general, but not for Org development.
>> People contributing to the Org manual are expected to understand Org
>> markup. So, it is Texinfo that becomes the new language to learn and
>> that creates an extra contribution barrier.
>
> If the idea is that the main contributors to the Org documentation are
> Org users, and the rest of Emacs developers don't really count for
> this purpose, then why did you count the contributions of the latter
> to begin with? they should be immaterial for you, AFAIU.

They are not immaterial. The work contributed to Org manual by Emacs
devs, even if small, is certainly appreciated. However, decision to
switch back to .texi is a major one and will affect all the
contributors, not just Emacs devs. Thus, I am trying to weight the
overall impact on the _Org_ development and maintenance.

I looked into git logs specifically to see the contribution pattern of
Emacs developers and whether it was influenced by switching from .texi
to .org. I only found that Texinfo markup-related contributions
disappeared (naturally). Other contributions remained at their levels:
typo fixes and enforcing Emacs-wide terminology and wording standards.

Neither typo fixes nor enforcing certain terminology uses are limited by
the fact that Org manual is not in .texi format. (Correct me if I am
wrong).

On the other hand, people contributing non-trivial patches for Org
manual should naturally be familiar with Org mode. So, using Org markup
should not be a handicap for them, while forcing texinfo may sometimes
be.

That's why, not disregarding Emacs dev contributions, I do not see what
positive change switching back to .texi would bring to Org development. 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> This may be true in general, but not for Org development.
>>> People contributing to the Org manual are expected to understand Org
>>> markup. So, it is Texinfo that becomes the new language to learn and
>>> that creates an extra contribution barrier.
>>
>> If the idea is that the main contributors to the Org documentation are
>> Org users, and the rest of Emacs developers don't really count for
>> this purpose, then why did you count the contributions of the latter
>> to begin with? they should be immaterial for you, AFAIU.
>
> They are not immaterial. The work contributed to Org manual by Emacs
> devs, even if small, is certainly appreciated. However, decision to
> switch back to .texi is a major one and will affect all the
> contributors, not just Emacs devs. Thus, I am trying to weight the
                    ^^^^^^^^^^^^^^^

Is Org not part of Emacs?

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bzg@gnu.org,  yantar92@gmail.com,
>   theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 18:19:45 +0800
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
> > They are not immaterial. The work contributed to Org manual by Emacs
> > devs, even if small, is certainly appreciated. However, decision to
> > switch back to .texi is a major one and will affect all the
> > contributors, not just Emacs devs. Thus, I am trying to weight the
>                     ^^^^^^^^^^^^^^^
> 
> Is Org not part of Emacs?

Its development is separate.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> They are not immaterial. The work contributed to Org manual by Emacs
>> devs, even if small, is certainly appreciated. However, decision to
>> switch back to .texi is a major one and will affect all the
>> contributors, not just Emacs devs. Thus, I am trying to weight the
>                     ^^^^^^^^^^^^^^^
>
> Is Org not part of Emacs?
 
Org is maintained separately from Emacs, but distributed together with
Emacs. 

That said, by "Emacs devs" I mostly meant experienced Elisp contributors
who are not familiar with Org markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 09:44:08 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If the idea is that the main contributors to the Org documentation are
> > Org users, and the rest of Emacs developers don't really count for
> > this purpose, then why did you count the contributions of the latter
> > to begin with? they should be immaterial for you, AFAIU.
> 
> They are not immaterial. The work contributed to Org manual by Emacs
> devs, even if small, is certainly appreciated. However, decision to
> switch back to .texi is a major one and will affect all the
> contributors, not just Emacs devs. Thus, I am trying to weight the
> overall impact on the _Org_ development and maintenance.
> 
> I looked into git logs specifically to see the contribution pattern of
> Emacs developers and whether it was influenced by switching from .texi
> to .org. I only found that Texinfo markup-related contributions
> disappeared (naturally). Other contributions remained at their levels:
> typo fixes and enforcing Emacs-wide terminology and wording standards.
> 
> Neither typo fixes nor enforcing certain terminology uses are limited by
> the fact that Org manual is not in .texi format. (Correct me if I am
> wrong).
> 
> On the other hand, people contributing non-trivial patches for Org
> manual should naturally be familiar with Org mode. So, using Org markup
> should not be a handicap for them, while forcing texinfo may sometimes
> be.
> 
> That's why, not disregarding Emacs dev contributions, I do not see what
> positive change switching back to .texi would bring to Org development. 

We are going in circles.  I already explained, or tried to explain,
what would be the positive change.  It's your call, obviously.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> That's why, not disregarding Emacs dev contributions, I do not see what
>> positive change switching back to .texi would bring to Org development. 
>
> We are going in circles.  I already explained, or tried to explain,
> what would be the positive change.  It's your call, obviously.

I understand that. However, I do not fully agree with the arguments you
used to support this statement. And I replied explaining why I do not
fully agree. Let me know if I missed something you wanted to point out.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > >  I don't know why Richard made that request.

I don't remember making that request.  I must have done so, but I
don't know why.

Can someone please show me that message?  Once i see its date,
I can find the pertinent messages in my saved old mail, and
think about this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > >  I don't know why Richard made that request.
>
> I don't remember making that request.  I must have done so, but I
> don't know why.
>
> Can someone please show me that message?  Once i see its date,
> I can find the pertinent messages in my saved old mail, and
> think about this.

These were the two messages referenced in the thread:

https://yhetil.org/emacs-devel/E1ocmvz-0002iB-2M@fencepost.gnu.org/
https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Having said that, RMS officially asked Org team to work towards allowing
  > Org to become the new GNU documentation standard.
  > https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/

The words quoted above can be interpreted in multiple ways, so when I
read it the first time, it didn't seem like anything I would have sad.
Now I see it does fit, when interpreted differently.

My suggestion was not that we adopt the current Org format for GNU
documentation.  The current Org format doesn't have all the features
that are needed -- it can't do the job that Texinfo does for us.

But I think the Org format could be _extended_ into a format that
would have all the features we need -- and then, with work, it could
become superior to Texinfo.

That would be an advance overall for documentation, and Org developers
will appreciate that Org plays a role in it.

The idea is based on a factual assumption that I think is true
but it couldn't hurt to check.  As I understand it, Org format is
fits into a family of formats, also including Markup and Sphinx,
which are similar in the basics.  Because of this, I think a lot
of people would find it easier to learn to use Org mode (including
the extensions that will be needed) than Texinfo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> The idea is based on a factual assumption that I think is true
> but it couldn't hurt to check.  As I understand it, Org format is
> fits into a family of formats, also including Markup and Sphinx,
> which are similar in the basics.

At least, Wikipedia classifies Org into "Lightweight Markup" family:
https://en.wikipedia.org/wiki/Lightweight_markup_language

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Ihor Radchenko <yantar92@posteo.net> writes:

    > Richard Stallman <rms@gnu.org> writes:
    >> The idea is based on a factual assumption that I think is true
    >> but it couldn't hurt to check.  As I understand it, Org format is
    >> fits into a family of formats, also including Markup and Sphinx,
    >> which are similar in the basics.

    > At least, Wikipedia classifies Org into "Lightweight Markup"
    > family: https://en.wikipedia.org/wiki/Lightweight_markup_language

"Lightweight Markup" appears to be the author's invention, with no
reference or citation given.

Colin Baxter.

Re: Org mode and Emacs

[Ihor Radchenko]

Colin Baxter <m43cap@yandex.com> writes:

>>>>>> Ihor Radchenko <yantar92@posteo.net> writes:
>
>     > Richard Stallman <rms@gnu.org> writes:
>     >> The idea is based on a factual assumption that I think is true
>     >> but it couldn't hurt to check.  As I understand it, Org format is
>     >> fits into a family of formats, also including Markup and Sphinx,
>     >> which are similar in the basics.
>
>     > At least, Wikipedia classifies Org into "Lightweight Markup"
>     > family: https://en.wikipedia.org/wiki/Lightweight_markup_language
>
> "Lightweight Markup" appears to be the author's invention, with no
> reference or citation given.

Why is it a problem? I have seen this term elsewhere as well.

In any case, the key point is not the terminology, but the fact that
other people independently grouped Org markup with multiple other
markups that are widely used.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Why is it a problem? I have seen this term elsewhere as well.
>
> In any case, the key point is not the terminology, but the fact that
> other people independently grouped Org markup with multiple other
> markups that are widely used.

I may not be the individual best qualified to speak apropos this
subject, but I can't imagine any reason learning Texinfo should be any
more challenging than learning a version of Org extended to sufficiently
address every one of Texinfo's applications.  Annulling any hypothetical
advantage -- should it exist -- is the significant body of GNU
developers who are already proficient in Texinfo, many of whom bear no
interest in Org.

Richard, what makes you believe that?  Can you provide any concrete
examples of Texinfo constructs that would be simpler in Org?

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Po Lu <luangruo@yahoo.com> writes:

    > Ihor Radchenko <yantar92@posteo.net> writes:
    >> Why is it a problem? I have seen this term elsewhere as well.
    >> 
    >> In any case, the key point is not the terminology, but the fact
    >> that other people independently grouped Org markup with multiple
    >> other markups that are widely used.

    > I may not be the individual best qualified to speak apropos this
    > subject, but I can't imagine any reason learning Texinfo should be
    > any more challenging than learning a version of Org extended to
    > sufficiently address every one of Texinfo's applications.
    > Annulling any hypothetical advantage -- should it exist -- is the
    > significant body of GNU developers who are already proficient in
    > Texinfo, many of whom bear no interest in Org.

    > Richard, what makes you believe that?  Can you provide any
    > concrete examples of Texinfo constructs that would be simpler in
    > Org?

Perhaps this ship sailed long ago but would removing org-mode from emacs
core go some way towards making the choice between using texinfo or
org-mode irrelevant? As an external package, org-mode's documentation
could then evolve independently, as its users think best.

Colin Baxter.

Re: Org mode and Emacs

[Ihor Radchenko]

Colin Baxter <m43cap@yandex.com> writes:

>     > Richard, what makes you believe that?  Can you provide any
>     > concrete examples of Texinfo constructs that would be simpler in
>     > Org?
>
> Perhaps this ship sailed long ago but would removing org-mode from emacs
> core go some way towards making the choice between using texinfo or
> org-mode irrelevant? As an external package, org-mode's documentation
> could then evolve independently, as its users think best.

IMHO, the problem with sources of Org manual is not that much of a big
deal. Eli favours all the manuals to be in .texi format because it is
standard. However, the current situation is certainly not _that_
unbearable to the level where we need to consider drastic measures like
removing the whole Org from Emacs. Not even close.

The far more important discussion here is about Org as potential
replacement to Texinfo (after Org is modified to support software
manual-specific markup). I am not an expert in Texinfo, so I cannot
judge if it is a good call or not.

On Texinfo side, there might be problems existing in Texinfo that will
not exist in Org, as I recall from previous discussions. But I cannot
tell any specifics.

On Org side, Org allows fine-tuned configuration of export for each
individual export target. And generating parts of the manuals
programmatically. AFAIK, this is not easily possible in Texinfo.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, rms@gnu.org, eliz@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 17:47:43 +0000
> 
> On Org side, Org allows fine-tuned configuration of export for each
> individual export target. And generating parts of the manuals
> programmatically. AFAIK, this is not easily possible in Texinfo.

Yes, it is.  At least the separate configuration part.  See the
section "Customization Variables" and its sub-sections in the Texinfo
manual.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On Org side, Org allows fine-tuned configuration of export for each
>> individual export target. And generating parts of the manuals
>> programmatically. AFAIK, this is not easily possible in Texinfo.
>
> Yes, it is.  At least the separate configuration part.  See the
> section "Customization Variables" and its sub-sections in the Texinfo
> manual.

I did not mean variables affecting all the output. I meant more
fine-grained staff, like direct html/latex/odt/... snippets only
embedded when exporting to html/latex/odt/...

For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
https://orgmode.org/manual/Quoting-HTML-tags.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: m43cap@yandex.com, luangruo@yahoo.com, rms@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 18:08:42 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> On Org side, Org allows fine-tuned configuration of export for each
> >> individual export target. And generating parts of the manuals
> >> programmatically. AFAIK, this is not easily possible in Texinfo.
> >
> > Yes, it is.  At least the separate configuration part.  See the
> > section "Customization Variables" and its sub-sections in the Texinfo
> > manual.
> 
> I did not mean variables affecting all the output. I meant more
> fine-grained staff, like direct html/latex/odt/... snippets only
> embedded when exporting to html/latex/odt/...
> 
> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
> https://orgmode.org/manual/Quoting-HTML-tags.html

Sorry, I don't understand what that means, even after looking at the
URLs.  "Direct snippets only embedded" I cannot parse, sorry.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I did not mean variables affecting all the output. I meant more
>> fine-grained staff, like direct html/latex/odt/... snippets only
>> embedded when exporting to html/latex/odt/...
>> 
>> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
>> https://orgmode.org/manual/Quoting-HTML-tags.html
>
> Sorry, I don't understand what that means, even after looking at the
> URLs.  "Direct snippets only embedded" I cannot parse, sorry.

For example, you can set a different color, for a single link, just when
exporting to html. Or put a page break at certain place just in odt
export. Or set image size differently in html and latex, for a single
image.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: m43cap@yandex.com, luangruo@yahoo.com, rms@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 18:36:36 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I did not mean variables affecting all the output. I meant more
> >> fine-grained staff, like direct html/latex/odt/... snippets only
> >> embedded when exporting to html/latex/odt/...
> >> 
> >> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
> >> https://orgmode.org/manual/Quoting-HTML-tags.html
> >
> > Sorry, I don't understand what that means, even after looking at the
> > URLs.  "Direct snippets only embedded" I cannot parse, sorry.
> 
> For example, you can set a different color, for a single link, just when
> exporting to html. Or put a page break at certain place just in odt
> export. Or set image size differently in html and latex, for a single
> image.

Texinfo has @ifhtml..@end ifhtml blocks, where you can do stuff for
HTML output only.  And similar for all other formats; see "Conditional
Commands" in the Texinfo manual.  There's also @html..@end html, where
you can write raw HTML (and similar for other formats; see "Raw
Formatter Commands" in the Texinfo manual.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> For example, you can set a different color, for a single link, just when
>> exporting to html. Or put a page break at certain place just in odt
>> export. Or set image size differently in html and latex, for a single
>> image.
>
> Texinfo has @ifhtml..@end ifhtml blocks, where you can do stuff for
> HTML output only.  And similar for all other formats; see "Conditional
> Commands" in the Texinfo manual.  There's also @html..@end html, where
> you can write raw HTML (and similar for other formats; see "Raw
> Formatter Commands" in the Texinfo manual.

This is indeed the same functionality.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, what makes you believe that?  Can you provide any concrete
  > examples of Texinfo constructs that would be simpler in Org?

That's not what the issue is.

Org format is similar to Markdown and I'd expect that millions of
users use Markdown.  Those millions of people would be willing to
learn all the other details if those details were in a framework
like Markdown.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Ihor Radchenko <yantar92@posteo.net> writes:

    > Colin Baxter <m43cap@yandex.com> writes:
    >>>>>>> Ihor Radchenko <yantar92@posteo.net> writes:
    >> 
    >> > Richard Stallman <rms@gnu.org> writes: >> The idea is based on
    >> a factual assumption that I think is true >> but it couldn't hurt
    >> to check.  As I understand it, Org format is >> fits into a
    >> family of formats, also including Markup and Sphinx, >> which are
    >> similar in the basics.
    >> 
    >> > At least, Wikipedia classifies Org into "Lightweight Markup" >
    >> family: https://en.wikipedia.org/wiki/Lightweight_markup_language
    >> 
    >> "Lightweight Markup" appears to be the author's invention, with
    >> no reference or citation given.

    > Why is it a problem? I have seen this term elsewhere as well.

Without a reference to consult, I do not understand what the author
means by the term "lightweight".

Colin Baxter.

Re: Org mode and Emacs

[Tassilo Horn]

Colin Baxter <m43cap@yandex.com> writes:

>     >> "Lightweight Markup" appears to be the author's invention, with
>     >> no reference or citation given.
>
>     > Why is it a problem? I have seen this term elsewhere as well.
>
> Without a reference to consult, I do not understand what the author
> means by the term "lightweight".

I think the common definition of lightweight markup is that it's still
easily readable by humans.  For example, HTML is pretty much unreadable
without a browser rendering it.  In contrast, Org and Markdown can be
easily read as plain text without a special renderer or editor, e.g.,
with just "less".

Bye,
Tassilo

Re: Org mode and Emacs

[Bastien Guerry]

FWIW, I think it is good to improve the .org format and Org tools
so that they are as good as .texi and Texinfo tools when it comes
to writing and publishing technical documentation.

Because most of Org syntax is easy to learn, this may help more
developers enjoy writing good documentation.

I don't think we need to consider replacing Texinfo by Org as the
GNU documentat format, though.

I understand such a possibility can be a motivation boost for Org
developers, but it could also be a pitfall, inducing focus on the
wrong priorities.

Let's simply try to improve Org in general, and see if more GNU
maintainers want to use it as their native documentat format (the
example of Org's documentation shows it's already possible.)

2 cts,

-- 
 Bastien Guerry

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 663 bytes --]

I agree. I think if we find gaps, we should first try to fix this in the
"export org to texi" code, and that will then reveal places where we
indeed need to evolve org as a format. In fact, if we discover that
org-to-texi cant express everything that texi can, we should first
consider adding styling options to org-to-texi flow, and only if that
fails, would we need to add new constructs to org.

Off the top of my head, the only things I can think of that org may not
presently cover is the ability to indicate index entries in the org
source


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> Off the top of my head, the only things I can think of that org may not
> presently cover is the ability to indicate index entries in the org
> source

For texi export, we allow #+cindex/#+findex/#+vindex.
Also, Timothy has been working on
https://github.com/tecosaur/org-glossary to bring it natively into Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[T.V Raman]

this is awesome to know -- glad we're already on it.



Ihor Radchenko writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > Off the top of my head, the only things I can think of that org may not
 > > presently cover is the ability to indicate index entries in the org
 > > source
 > 
 > For texi export, we allow #+cindex/#+findex/#+vindex.
 > Also, Timothy has been working on
 > https://github.com/tecosaur/org-glossary to bring it natively into Org.
 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Off the top of my head, the only things I can think of that org may not
  > presently cover is the ability to indicate index entries in the org
  > source

How does one express these constructs in Org format?

@code
@samo
@emph
@def
@var
@i
@url
@key
@kbd

  > In fact, if we discover that
  > org-to-texi cant express everything that texi can, we should first
  > consider adding styling options to org-to-texi flow, and only if that
  > fails, would we need to add new constructs to org.

I am skeptical that that would address these issues, but I can't be sure
since I don't know what that does.  Can you please show a hypotherical
example of solvin a problem this way?

For instance, how would you propose to use them to distinguish
between @var, @def and @emph?

They look alike in TeX output.  It is in OTHER formats that they
produce different output.

1. What would that approach look like in the document source?
I can't be sure 
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Oops!  I wrote

  > @code
  > @samo
  > @emph
  > @def
  > @var
  > @i
  > @url
  > @key
  > @kbd

but it is not @def, it is @dfn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW, I think it is good to improve the .org format and Org tools
  > so that they are as good as .texi and Texinfo tools when it comes
  > to writing and publishing technical documentation.

The term "as good as" may suggest, incorrectly, that this is a matter
of comparing the two formats over some sense of _quality_.  But that's
not what this is about.  The improvements I've proposed for Org format
are a matter of _supporting the range of necessary constructs_.

Texinfo supports them all because I implemented them all in Texinfo.
As a result, you can express in Texinfo everything that has proved
necessary (so far, at least) in GNU documentation.

At present, that is not the case with Org format.  There are
situations in which Org format provides no way to express what needs
to be said.

  > Let's simply try to improve Org in general, and see if more GNU
  > maintainers want to use it as their native documentat format (the
  > example of Org's documentation shows it's already possible.)

We need to be careful here.  What does the existence of Org mode
documentation written in Org format actually show -- given that the
format doesn't support all the constructs that are needed in general?

It might show that the Org mode documentation doesn't make all the
textual distinctions properly -- that it fails to follow our style
guide.  If so, then it is "possible" but only with flawed output.

But not necesarily.  Perhaps it shows that the Org mode documentation
needs only a limited subset of those constructs, and those are all
implemened in Ogr format.  If so, that could mean that Org format is
fine for the Org mode manual in prticular, but is not adequate for the
whole range of our documentation.

Either way, to make Org format adequate for that whole range of
constructs, in all the output formats, will require working
specifically towards that goal.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Bastien Guerry]

Richard Stallman <rms@gnu.org> writes:

> The term "as good as" may suggest, incorrectly, that this is a matter
> of comparing the two formats over some sense of _quality_.  But that's
> not what this is about.  The improvements I've proposed for Org format
> are a matter of _supporting the range of necessary constructs_.

I'm confident we can support the necessary constructs in Org.

>   > Let's simply try to improve Org in general, and see if more GNU
>   > maintainers want to use it as their native documentat format (the
>   > example of Org's documentation shows it's already possible.)
>
> We need to be careful here.  What does the existence of Org mode
> documentation written in Org format actually show -- given that the
> format doesn't support all the constructs that are needed in general?
>
> It might show that the Org mode documentation doesn't make all the
> textual distinctions properly -- that it fails to follow our style
> guide.  If so, then it is "possible" but only with flawed output.

If a .texi expert can report such flaws in the Org manual, we can then
fix them and, if needed, implement the necessary constructs.

> But not necesarily.  Perhaps it shows that the Org mode documentation
> needs only a limited subset of those constructs, and those are all
> implemened in Ogr format.  If so, that could mean that Org format is
> fine for the Org mode manual in prticular, but is not adequate for the
> whole range of our documentation.

I believe this is more plausible.

> Either way, to make Org format adequate for that whole range of
> constructs, in all the output formats, will require working
> specifically towards that goal.

Agreed, and this is what Org maintainers are working on.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Philip Kaludercic]

Bastien Guerry <bzg@gnu.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> The term "as good as" may suggest, incorrectly, that this is a matter
>> of comparing the two formats over some sense of _quality_.  But that's
>> not what this is about.  The improvements I've proposed for Org format
>> are a matter of _supporting the range of necessary constructs_.
>
> I'm confident we can support the necessary constructs in Org.
>
>>   > Let's simply try to improve Org in general, and see if more GNU
>>   > maintainers want to use it as their native documentat format (the
>>   > example of Org's documentation shows it's already possible.)
>>
>> We need to be careful here.  What does the existence of Org mode
>> documentation written in Org format actually show -- given that the
>> format doesn't support all the constructs that are needed in general?
>>
>> It might show that the Org mode documentation doesn't make all the
>> textual distinctions properly -- that it fails to follow our style
>> guide.  If so, then it is "possible" but only with flawed output.
>
> If a .texi expert can report such flaws in the Org manual, we can then
> fix them and, if needed, implement the necessary constructs.

I don't know if this is of any use, but the initial manual for Compat
(https://elpa.gnu.org/packages/compat.html) was written using Org and
ox-texinfo and I later switched to writing .texi directly.  This commit
here documents the switch that includes a partial rewrite.

https://git.sr.ht/~pkal/compat/commit/dd48603a136881a5321de4419be95ea873496172

Some things here might be difficult to map, such as the proper usage of
reference macros or the different kinds of markup from (texinfo) Indicating.

>
>> But not necesarily.  Perhaps it shows that the Org mode documentation
>> needs only a limited subset of those constructs, and those are all
>> implemened in Ogr format.  If so, that could mean that Org format is
>> fine for the Org mode manual in prticular, but is not adequate for the
>> whole range of our documentation.
>
> I believe this is more plausible.
>
>> Either way, to make Org format adequate for that whole range of
>> constructs, in all the output formats, will require working
>> specifically towards that goal.
>
> Agreed, and this is what Org maintainers are working on.

Re: Org mode and Emacs

[Ihor Radchenko]

Philip Kaludercic <philipk@posteo.net> writes:

>> If a .texi expert can report such flaws in the Org manual, we can then
>> fix them and, if needed, implement the necessary constructs.
>
> I don't know if this is of any use, but the initial manual for Compat
> (https://elpa.gnu.org/packages/compat.html) was written using Org and
> ox-texinfo and I later switched to writing .texi directly.  This commit
> here documents the switch that includes a partial rewrite.
>
> https://git.sr.ht/~pkal/compat/commit/dd48603a136881a5321de4419be95ea873496172
>
> Some things here might be difficult to map, such as the proper usage of
> reference macros or the different kinds of markup from (texinfo) Indicating.

ox-texinfo certainly misses some specialalized Texinfo syntax, like
@defn, @var, etc:

doc/Documentation_Standards.org:

   + Texinfo commands such as @var and @defoption are not used.  The
     preference for this type of thing is that the user browses the
     customize groups.  If you want or need to refer to, say, a
     variable then document it as "the variable
     @code{org-startup-folded}"

It is not export backend fault per se - Org markup simply does not
define specialized markups more granular than ~code~.
So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
direct texinfo export snippet @@texinfo:@kbd{@@C-c SPC@@texinfo:}@@

Though we do not provide similar macros for @var/@env/etc. We might.
Or we might allow custom inline special markup as I suggested in
https://orgmode.org/list/87bkqx4jyg.fsf@localhost

Another limitation, I can see in the commit is using @xref, which Org
does not do. ox-texinfo now transforms <info:#node> links into @ref
only.

On the other hand, Org provides a powerful citation syntax
[cite:see @citation-key], which may be also utilized if we support
info files as bibliography source. This will be a superset of what
@xref/@ref/@pxref does as cite/variant is an equivalent of \autocite in
LaTeX.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is not export backend fault per se - Org markup simply does not
  > define specialized markups more granular than ~code~.
  > So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
  > direct texinfo export snippet ...

That source construct might be acceptable, though it has 6 extra characters
and that would be rather inconvenient.  Could you manage to reduce the number
of them?

Could you explain what is implied/presumed by "direct texinfo export"?
Are you saying that Org would be a front-end for Texinfo, for all
output formats?  I'd hope it could generate Info files and HTML directly.

  > Though we do not provide similar macros for @var/@env/etc. We might.

They are very often used.  To make org adequate as a replacement for
Texinfo, they would have to be predefined.  (It is ok if the file
has to specify loading some extension to define them.)

  > Or we might allow custom inline special markup as I suggested in
  > https://orgmode.org/list/87bkqx4jyg.fsf@localhost

I skimmed that long message but I couldn't see any suggestion about
@var, @env, etc.  Could you please send me the actual proposal, by
itself, rather than an indirection to it?

  > On the other hand, Org provides a powerful citation syntax
  > [cite:see @citation-key], which may be also utilized if we support
  > info files as bibliography source. This will be a superset of what
  > @xref/@ref/@pxref does as cite/variant is an equivalent of \autocite in
  > LaTeX.

If that is smarter than the current Texinfo commands and can replace
them with no loss of flexibility and capability, that would be fine.
Provided it handles all output formats correctly.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > It is not export backend fault per se - Org markup simply does not
>   > define specialized markups more granular than ~code~.
>   > So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
>   > direct texinfo export snippet ...
>
> That source construct might be acceptable, though it has 6 extra characters
> and that would be rather inconvenient.  Could you manage to reduce the number
> of them?

> Could you explain what is implied/presumed by "direct texinfo export"?
> Are you saying that Org would be a front-end for Texinfo, for all
> output formats?  I'd hope it could generate Info files and HTML directly.

To clarify, I was describing what already exists in Org. We leverage
Texinfo itself as intermediate format to export to Info. The above "kbd"
macro is specifically designed for Texinfo export.

We may eventually have an option to export to Info directly, but not
yet. Though nothing stops people from implementing such functionality.
Adding new export Backends is not very hard:
https://orgmode.org/worg/dev/org-export-reference.html#back-end

>   > Though we do not provide similar macros for @var/@env/etc. We might.
>
> They are very often used.  To make org adequate as a replacement for
> Texinfo, they would have to be predefined.  (It is ok if the file
> has to specify loading some extension to define them.)
>
>   > Or we might allow custom inline special markup as I suggested in
>   > https://orgmode.org/list/87bkqx4jyg.fsf@localhost
>
> I skimmed that long message but I couldn't see any suggestion about
> @var, @env, etc.  Could you please send me the actual proposal, by
> itself, rather than an indirection to it?

For more complete support of software manual-specific markup, we plan to
introduce more versatile, hackable markup that can be extended by users
to fit their needs. See
https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/ and
https://list.orgmode.org/orgmode/87mtaez8do.fsf@localhost/ 

One of the ideas was to follow TeXinfo markup closely, extending with
some Org-specific needs:

Simple case:
@name{<contents>}

If we need to escape { or } inside:
@name{{<contents}}

Extra configuration:
@name[:key value ...]{<contents>}

@color[:name red]{[[file:image.png][description]]}

Then, users or packages can define the details of how to fontify or
export a given @name{...} markup. That way, we can add anything,
including @var/@env/...

We already do something similar for links:
https://orgmode.org/manual/Adding-Hyperlink-Types.html (org-man-export),
except that links have limitations that prevent them from being used as
markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It might show that the Org mode documentation doesn't make all the
  > > textual distinctions properly -- that it fails to follow our style
  > > guide.  If so, then it is "possible" but only with flawed output.

  > If a .texi expert can report such flaws in the Org manual, we can then
  > fix them and, if needed, implement the necessary constructs.

Please look at the list I sent yesterday.  (See below.)  Can Org mode
handle each of them, and generate from it the correct output for each
output format?

  > @code
  > @samo
  > @emph
  > @dfn
  > @var
  > @i
  > @url
  > @key
  > @kbd

If it can, the next thing to do is to look through
the node in the Texinfo manual which documents these,
and see what other constructs are in that node,
and see if Org mode has equivalent constructs for
all of those Texinfo constructs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 25 Aug 2023 22:04:13 -0400
> 
> Please look at the list I sent yesterday.  (See below.)  Can Org mode
> handle each of them, and generate from it the correct output for each
> output format?
> 
>   > @code
>   > @samo
>   > @emph
>   > @dfn
>   > @var
>   > @i
>   > @url
>   > @key
>   > @kbd

I think it can, but a typical Texinfo manual these days uses many more
commands, not just those.  So this list is insufficient to judge these
issues.  The actual list is much longer.

Re: Org mode and Emacs

[Jose E. Marchesi]

>> From: Richard Stallman <rms@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 25 Aug 2023 22:04:13 -0400
>> 
>> Please look at the list I sent yesterday.  (See below.)  Can Org mode
>> handle each of them, and generate from it the correct output for each
>> output format?
>> 
>>   > @code
>>   > @samo
>>   > @emph
>>   > @dfn
>>   > @var
>>   > @i
>>   > @url
>>   > @key
>>   > @kbd
>
> I think it can, but a typical Texinfo manual these days uses many more
> commands, not just those.  So this list is insufficient to judge these
> issues.  The actual list is much longer.

I think the point here is not just these particular marks, or some other
particular set.  It is that "markdown-style" formats, like org-mode, are
not very good supporting "inline" marking.

For inline marks, Texinfo uses a conventient generic and extensible
form: @NAME{CONTENT}.

"markdown-style" formats, on the contrary, generally support a closed
set of marks that use delimited based ad-hoc syntax, such as *foo*,
=foo=, "foo", `foo' and so on.

Block marks are usually supported well enough by these formats.  In the
case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

Re: Org mode and Emacs

[Ihor Radchenko]

"Jose E. Marchesi" <jemarch@gnu.org> writes:

> I think the point here is not just these particular marks, or some other
> particular set.  It is that "markdown-style" formats, like org-mode, are
> not very good supporting "inline" marking.
>
> For inline marks, Texinfo uses a conventient generic and extensible
> form: @NAME{CONTENT}.

That's what we plan to add to Org. Not just because of this discussion;
there are other reasons why we want such custom inline markup.

> Block marks are usually supported well enough by these formats.  In the
> case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

Yup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Philip Kaludercic]

Ihor Radchenko <yantar92@posteo.net> writes:

> "Jose E. Marchesi" <jemarch@gnu.org> writes:
>
>> I think the point here is not just these particular marks, or some other
>> particular set.  It is that "markdown-style" formats, like org-mode, are
>> not very good supporting "inline" marking.
>>
>> For inline marks, Texinfo uses a conventient generic and extensible
>> form: @NAME{CONTENT}.
>
> That's what we plan to add to Org. Not just because of this discussion;
> there are other reasons why we want such custom inline markup.

Do you have a reference where I could read up on how this is going?

>> Block marks are usually supported well enough by these formats.  In the
>> case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
>
> Yup.

Re: Org mode and Emacs

[Ihor Radchenko]

Philip Kaludercic <philipk@posteo.net> writes:

>>> For inline marks, Texinfo uses a conventient generic and extensible
>>> form: @NAME{CONTENT}.
>>
>> That's what we plan to add to Org. Not just because of this discussion;
>> there are other reasons why we want such custom inline markup.
>
> Do you have a reference where I could read up on how this is going?

https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

OK, forget my previous request, I'll read from here.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Block marks are usually supported well enough by these formats.  In the
  > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

That syntax would do the job, but they are more clumsy than the syntax
used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
Org mode syntax to be a step up, it should look as clean as Texinfo.

Might it be posisble to simplify the syntax that Org would use for
this?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > > Block marks are usually supported well enough by these formats.  In the
>   > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
>
> That syntax would do the job, but they are more clumsy than the syntax
> used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
> Org mode syntax to be a step up, it should look as clean as Texinfo.
>
> Might it be posisble to simplify the syntax that Org would use for
> this?

I am not convinced that it is necessary.
Doing so will either be backward-incompatible or introduce more syntax
constructs to Org, making the syntax more complicated.

And I do not find the current syntax "clumsy".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > > Block marks are usually supported well enough by these formats.  In the
  > >   > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
  > >
  > > That syntax would do the job, but they are more clumsy than the syntax
  > > used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
  > > Org mode syntax to be a step up, it should look as clean as Texinfo.
  > >
  > > Might it be posisble to simplify the syntax that Org would use for
  > > this?

  > I am not convinced that it is necessary.

I won't claim it is _necessary_, but it would sure make these constructs
more convenient.

  > Doing so will either be backward-incompatible or introduce more syntax
  > constructs to Org, making the syntax more complicated.

Maybe it is ok for it to be backward-incompatible.  I don't know for
certain, but it looks like these constructs were recently added.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > > That syntax would do the job, but they are more clumsy than the syntax
>   > > used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
>   > > Org mode syntax to be a step up, it should look as clean as Texinfo.
>   > >
>   > > Might it be posisble to simplify the syntax that Org would use for
>   > > this?
>
>   > I am not convinced that it is necessary.
>
> I won't claim it is _necessary_, but it would sure make these constructs
> more convenient.

Maybe. But also more confusing. In org, we can have

#+begin_src latex
...
#+end_src latex

or

#+begin_export latex
...
#+end_export

or

#+begin_example latex
...
#+end_example

or

#+begin: name
...
#+end:

If we add simplified syntax like

#+latex
...
#+end latex

or, less consistently with the rest of Org

@latex
...
@end latex

it is actually not very intuitive that such construct implies export,
but not something else.

In addition, we already have

#+latex: latex one-liner

as a simple form, limited to a single line that avoids

#+begin_export latex
latex one-liner
#+end_export

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > or, less consistently with the rest of Org

  > @latex
  > ...
  > @end latex

  > it is actually not very intuitive that such construct implies export,
  > but not something else.

I'd like to understand this issue.  Could you please explain what
"export" means here?  And what is the "something else", that it might
have been?  It appears to be a distinction that doesn't exist in
Texinfo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > @latex
>   > ...
>   > @end latex
>
>   > it is actually not very intuitive that such construct implies export,
>   > but not something else.
>
> I'd like to understand this issue.  Could you please explain what
> "export" means here?  And what is the "something else", that it might
> have been?  It appears to be a distinction that doesn't exist in
> Texinfo.

You are right, Texinfo has a very narrow focus - authoring manuals in
multiple output formats. Org does a lot more.

-----

One feature of Org we are discussing here is "exporting":
https://orgmode.org/manual/Exporting.html - an ability to convert Org
documents into other textual formats.
Similar to Texinfo's raw formatter commands
(https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Raw-Formatter-Commands.html),
Org provides a special markup that specifies text to be placed into
specific tex/html/odt/etc export output:

#+begin_export html
...
#+end_export
or
#+HTML: <single line>
or
@@html:<inline html>@@

See https://orgmode.org/manual/Quoting-HTML-tags.html, for example.

-----

However, Org has more features, served by other Org's markup.

-----

One of the popular features is implementing literal programming (babel).
https://orgmode.org/manual/Features-Overview.html:

    Org can manage the source code in the block delimited by
    ‘#+BEGIN_SRC’ … ‘#+END_SRC’ in several ways that can simplify
    housekeeping tasks essential to modern source code maintenance. Org
    can edit, format, extract, export, and publish source code blocks.

#+begin_src emacs-lisp :results value
  (+ 1 2)
#+end_src

#+RESULTS[b6b278abd782307b5eca9b1b761906bae5292bcc]:
: 3

Users can execute arbitrary pieces of source code in arbitrary
programming languages, get the results inline and even mix input/output
of code written in different languages:

#+name: bash-input
#+begin_src bash :results output table :var file="file.csv"
cat $file
#+end_src

#+begin_src elisp :var table=bash-input(file="table-input.csv")
(mapcar (lambda (line) (format "%S" line)) table)
#+end_src

The code evaluation results can even be used to generate exported text.
Here is a slightly edited snippet from Org manual that generates a table
of all export customizations:

#+begin_src emacs-lisp :exports results :results table :eval yes
(require 'ox)
(let ((alist org-export-options-alist))
 (let (result)
  (dolist (spec alist)
    (when (and (nth 3 spec) (symbolp (nth 3 spec))) ; has customization
      (push (list (format "~%s~" (car spec)) (format "~%s~" (nth 3 spec))) result)))
  (nreverse result)))
#+end_src

You can see the output in https://orgmode.org/manual/Publishing-options.html

Furthermore, users can "tangle" code from the Org document, extracting
only the appropriate source code pieces into actual code-only files.
https://orgmode.org/manual/Extracting-Source-Code.html
So, one can keep documentation and notes (with markup!) together with
the actual code, tangle the org file to produce the program, or export
the org file to produce readable notes - full literal programming
implementation.

-----

Apart from snippets of source code, Org has a markup for non-code
literal text that is not a subject of Org markup:

#+begin_example
  Some example from a text file.
#+end_example

Some people even prefer to keep snippets of source code as example
blocks as well (when a sole purpose of such source block is exporting):

#+begin_example bash
  ls $HOME
#+end_example

-----

Finally, there are dynamic blocks that can be used to generate parts of
Org file programatically from Elisp: https://orgmode.org/manual/Dynamic-Blocks.html

#+begin: blockname <parameters>
<automatically generated text>
#+end:

One example of built-in dynamic block is clocktable that is used by
people that record their work time using Org mode
(https://orgmode.org/manual/Clocking-Work-Time.html,
https://orgmode.org/manual/The-clock-table.html,
https://olddeuteronomy.github.io/post/org-export-clocktables/)

-----

To conclude, the common pattern for Org's block markup is

#+begin<something>
...
#+end<something>

What you proposed with @latex: ... @end is similar to dropping
<something> part. However, as you see in the above, there is a variety
of different block markups serving for entirely different purposes and
just saying @latex (or rather #+latex, if we follow the common Org
block convention) does make it intuitive whether the block is "export",
or "example" or "src" or dynamic. Furthermore, #+latex is already used
for export one-liners.

In theory, we might drop "begin" parts and get

#+src elisp
...
#+end

#+example
...
#+end

#+export html
#+end

but this is akin converting Pascal's begin/end into C's {/} - possible,
but an alternative convention is already established.

And, generally, we, in Org, try to refrain from changing syntax too much
- a number of external projects are relying upon the existing syntax.
So, making significant syntax changes should not be taken lightly.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > One feature of Org we are discussing here is "exporting":
  > https://orgmode.org/manual/Exporting.html - an ability to convert Org
  > documents into other textual formats.

I think I understand what you mean by exporting.  Maybe I understand
what you have in mind for "exporting" Org documents as Texinfo so as
to make GNU manuals.

But I think I see a disappointing difference between the two.  Texinfo
for manuals is semantic markup, but the proposed use of Org mode for
manuals is not.

With Texinfo, you specify what the manual contents should be, and in
each format you get that contents as output.  See what I mean?

I was hoping to see that same smoothness with Org mode -- that you
specify the contents and Org mode owuld give youa manual with that
contents.  But instead, it seems that Org mode for a manual would
specify Texinfo text to "export", and Org would generate Texinfo source
to export which Texinfo could then convert convert into manuals as output.

This exporting might _work_, but it not be as good, overall, as
Texinfo.  It would be disappointing.

Could it be it possible to make Org mode generate the manual output
formats using contructs that are semantic in meaning, that specify
semantic markup for the contents of the manual -- and have Org mode
generate the output formats to represent the semantics of the manual
contents?  That would be a good replacement for Texinfo.

I think this from Bastien Guerry

    Yes, that's the idea behind using one-char constructs for basic inline
    markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
    specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

Is a sort of thing that might be smooth in this sense.  It would specify
semantics of the contents, rather than "what to export".

It might bot be the best way (as Po Lu says), but at least it is
semantic markup.

  > For more complete support of software manual-specific markup, we plan to
  > introduce more versatile, hackable markup that can be extended by users
  > to fit their needs. See
  > https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/ and
  > https://list.orgmode.org/orgmode/87mtaez8do.fsf@localhost/

That coukd be the sort of thing that would be as smooth so Texinfo
format.

  > One of the ideas was to follow TeXinfo markup closely, extending with
  > some Org-specific needs:

  > Simple case:
  > @name{<contents>}

  > If we need to escape { or } inside:
  > @name{{<contents}}

  > Extra configuration:
  > @name[:key value ...]{<contents>}

This describes a syntax, but not the semantics.

For

  > Simple case:
  > @name{<contents>}

what's an example of what _name_ might be, and what would it mean?





-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Juergen Fenn]

Am 04.09.23 um 03:32 Uhr schrieb Richard Stallman:
> But I think I see a disappointing difference between the two. Texinfo
> for manuals is semantic markup, but the proposed use of Org mode for
> manuals is not. With Texinfo, you specify what the manual contents
> should be, and in each format you get that contents as output. See what
> I mean?


This is an interesting point indeed. I wonder, too, whether we could add
semantic markup to Org? It's not exactly what lightweight markup
languages such as Markdown or Org are meant to be. But we would not
necessarily need this in core, a semantic add-on would probably be
enough. We would need an interface that allows for more or less freely
defining semantic markup to Org text. A semantic org mode. Hm.

Best regards,
Jürgen.

Re: Org mode and Emacs

[Ihor Radchenko]

Juergen Fenn <jfenn@gmx.net> writes:

> Am 04.09.23 um 03:32 Uhr schrieb Richard Stallman:
>> But I think I see a disappointing difference between the two. Texinfo
>> for manuals is semantic markup, but the proposed use of Org mode for
>> manuals is not. With Texinfo, you specify what the manual contents
>> should be, and in each format you get that contents as output. See what
>> I mean?
>
>
> This is an interesting point indeed. I wonder, too, whether we could add
> semantic markup to Org? It's not exactly what lightweight markup
> languages such as Markdown or Org are meant to be. But we would not
> necessarily need this in core, a semantic add-on would probably be
> enough. We would need an interface that allows for more or less freely
> defining semantic markup to Org text. A semantic org mode. Hm.

This is exactly what I have in mind.
Similar idea is implemented in https://github.com/alhassy/org-special-block-extras

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is an interesting point indeed. I wonder, too, whether we could add
  > semantic markup to Org? It's not exactly what lightweight markup
  > languages such as Markdown or Org are meant to be. But we would not
  > necessarily need this in core, a semantic add-on would probably be
  > enough.

Implementing it in an extension would be ok -- it could still do the
job that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> I was hoping to see that same smoothness with Org mode -- that you
> specify the contents and Org mode owuld give youa manual with that
> contents.  But instead, it seems that Org mode for a manual would
> specify Texinfo text to "export", and Org would generate Texinfo source
> to export which Texinfo could then convert convert into manuals as output.

I think I need to clarify. This is how export to info works _now_.
However, nothing stops us from implementing direct export org->info
without intermediate org->texi step. Someone just needs to do this job.

> Could it be it possible to make Org mode generate the manual output
> formats using contructs that are semantic in meaning, that specify
> semantic markup for the contents of the manual -- and have Org mode
> generate the output formats to represent the semantics of the manual
> contents?  That would be a good replacement for Texinfo.

> I think this from Bastien Guerry
>
>     Yes, that's the idea behind using one-char constructs for basic inline
>     markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
>     specific inline markup (eg ~~defn~~, =~defvar~=, etc.).
>
> Is a sort of thing that might be smooth in this sense.  It would specify
> semantics of the contents, rather than "what to export".
>
> It might bot be the best way (as Po Lu says), but at least it is
> semantic markup.

Sorry, but I am lost here.

Currently, Org mode does not support all the markup supported by
texinfo. But we will change that.

Meanwhile, Org implements info/manual export by ad-hoc extension
allowing to add texinfo-specific markup. This is not ideal, and it does
not need to be done after we do add the missing markup (Org equivalents
of @kbd, @var, and other)

>   > One of the ideas was to follow TeXinfo markup closely, extending with
>   > some Org-specific needs:
>
>   > Simple case:
>   > @name{<contents>}
>
>   > If we need to escape { or } inside:
>   > @name{{<contents}}
>
>   > Extra configuration:
>   > @name[:key value ...]{<contents>}
>
> This describes a syntax, but not the semantics.
>
> For
>
>   > Simple case:
>   > @name{<contents>}
>
> what's an example of what _name_ might be, and what would it mean?

For example, @kbd{C-c C-x C-c} or @var{org-export-backends}.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 11:29:02 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > I was hoping to see that same smoothness with Org mode -- that you
> > specify the contents and Org mode owuld give youa manual with that
> > contents.  But instead, it seems that Org mode for a manual would
> > specify Texinfo text to "export", and Org would generate Texinfo source
> > to export which Texinfo could then convert convert into manuals as output.
> 
> I think I need to clarify. This is how export to info works _now_.
> However, nothing stops us from implementing direct export org->info
> without intermediate org->texi step. Someone just needs to do this job.

Unless the implementation will somehow invoke texi2any under the hood,
this would mean a separate implementation of texi2any in Emacs Lisp,
something that I don't recommend: it will need to play a constant
catchup game with the Texinfo development, and it will be probably
quite slow.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think I need to clarify. This is how export to info works _now_.
>> However, nothing stops us from implementing direct export org->info
>> without intermediate org->texi step. Someone just needs to do this job.
>
> Unless the implementation will somehow invoke texi2any under the hood,
> this would mean a separate implementation of texi2any in Emacs Lisp,

Yes. We already have org2pdf, org2ascii, org2html, but not org2info.

> something that I don't recommend: it will need to play a constant
> catchup game with the Texinfo development, and it will be probably
> quite slow.

I am not sure about catchup game. Unless Texinfo drastically changes the
output format, we can simply provide default settings to replicate the
Texinfo look of the produced html/pdf/plaintext/info files.

As for being slow, it is already not the case. Last time we discussed
Org export being slow, I have managed to speed things up on par with
Texinfo timings on Org manual.

In any case, the first step is extending Org markup to be more suitable
for software manuals. Whether we use texinfo under the hood or not might
be discussed afterwards.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> As for being slow, it is already not the case. Last time we discussed
> Org export being slow, I have managed to speed things up on par with
> Texinfo timings on Org manual.

Did you use Texinfo 4.13 as a baseline for the comparison, or a more
recent release written in Perl?

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> As for being slow, it is already not the case. Last time we discussed
>> Org export being slow, I have managed to speed things up on par with
>> Texinfo timings on Org manual.
>
> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
> recent release written in Perl?

More recent release.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Ihor Radchenko <yantar92@posteo.net> writes:
>>
>>> As for being slow, it is already not the case. Last time we discussed
>>> Org export being slow, I have managed to speed things up on par with
>>> Texinfo timings on Org manual.
>>
>> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
>> recent release written in Perl?
>
> More recent release.

makeinfo 7.0.2 requires approximately 18x more time to generate
emacs.info than 4.13, so an accurate performance comparison cannot be
made with the Perl versions as a basis.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>>>> As for being slow, it is already not the case. Last time we discussed
>>>> Org export being slow, I have managed to speed things up on par with
>>>> Texinfo timings on Org manual.
>>>
>>> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
>>> recent release written in Perl?
>>
>> More recent release.
>
> makeinfo 7.0.2 requires approximately 18x more time to generate
> emacs.info than 4.13, so an accurate performance comparison cannot be
> made with the Perl versions as a basis.

If makeinfo 7.0.2 deems its performance acceptable, I do not see why
similar Org's performance should not be.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> If makeinfo 7.0.2 deems its performance acceptable, I do not see why
> similar Org's performance should not be.

Many of us don't find makeinfo 7.0.2's performance tolerable, which is
why Emacs continues to support Texinfo 4.13.  It's not the only Texinfo
user to maintain such compatibility, BTW.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 21:33:50 +0800
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
> > If makeinfo 7.0.2 deems its performance acceptable, I do not see why
> > similar Org's performance should not be.
> 
> Many of us don't find makeinfo 7.0.2's performance tolerable, which is
> why Emacs continues to support Texinfo 4.13.

That's your personal view.  I find the time it takes to produce even
such large manuals as emacs.info and elisp.info with Texinfo 7
entirely reasonable, at least when Perl extensions are used.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 12:43:55 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I think I need to clarify. This is how export to info works _now_.
> >> However, nothing stops us from implementing direct export org->info
> >> without intermediate org->texi step. Someone just needs to do this job.
> >
> > Unless the implementation will somehow invoke texi2any under the hood,
> > this would mean a separate implementation of texi2any in Emacs Lisp,
> 
> Yes. We already have org2pdf, org2ascii, org2html, but not org2info.
> 
> > something that I don't recommend: it will need to play a constant
> > catchup game with the Texinfo development, and it will be probably
> > quite slow.
> 
> I am not sure about catchup game. Unless Texinfo drastically changes the
> output format, we can simply provide default settings to replicate the
> Texinfo look of the produced html/pdf/plaintext/info files.

Not the output format, but the language: new directives and sometimes
markup are added with every release of Texinfo.  Just read their NEWS
file.

> As for being slow, it is already not the case. Last time we discussed
> Org export being slow, I have managed to speed things up on par with
> Texinfo timings on Org manual.

That was for Org-to-Texinfo conversion.  Here we are talking about
Org-to-Info, which is much more work.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I am not sure about catchup game. Unless Texinfo drastically changes the
>> output format, we can simply provide default settings to replicate the
>> Texinfo look of the produced html/pdf/plaintext/info files.
>
> Not the output format, but the language: new directives and sometimes
> markup are added with every release of Texinfo.  Just read their NEWS
> file.

The way I see Org markup extension would make it easy to users add new
custom markup, as needed. Then, no frequent changes to the base markup
will be necessary to accommodate for less common use cases.

In other words, I do not see why Org should support every single Texinfo
markup. We just need to provide enough support to be on par in terms of
the needs of manual authors.

>> As for being slow, it is already not the case. Last time we discussed
>> Org export being slow, I have managed to speed things up on par with
>> Texinfo timings on Org manual.
>
> That was for Org-to-Texinfo conversion.  Here we are talking about
> Org-to-Info, which is much more work.

Why so?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 13:20:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I am not sure about catchup game. Unless Texinfo drastically changes the
> >> output format, we can simply provide default settings to replicate the
> >> Texinfo look of the produced html/pdf/plaintext/info files.
> >
> > Not the output format, but the language: new directives and sometimes
> > markup are added with every release of Texinfo.  Just read their NEWS
> > file.
> 
> The way I see Org markup extension would make it easy to users add new
> custom markup, as needed. Then, no frequent changes to the base markup
> will be necessary to accommodate for less common use cases.

You missed the "new directives" part.  I do recommend reading their
NEWS to get the idea of the rate at which new features are added to
Texinfo.

> In other words, I do not see why Org should support every single Texinfo
> markup. We just need to provide enough support to be on par in terms of
> the needs of manual authors.

GNU Manuals use a large portion of what Texinfo provides, and limiting
what they could use when they write in Org would mean some Texinfo
features cannot be used that way, which is a disadvantage.  People
will have to think twice before they switch to Org, because at some
point they might want to use a feature you decided not to support.

> > That was for Org-to-Texinfo conversion.  Here we are talking about
> > Org-to-Info, which is much more work.
> 
> Why so?

Because the production rules are much more complex.  I suggest to take
a look at the Perl source code of texi2any to see what it entails.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Not the output format, but the language: new directives and sometimes
>> > markup are added with every release of Texinfo.  Just read their NEWS
>> > file.
>> 
>> The way I see Org markup extension would make it easy to users add new
>> custom markup, as needed. Then, no frequent changes to the base markup
>> will be necessary to accommodate for less common use cases.
>
> You missed the "new directives" part.  I do recommend reading their
> NEWS to get the idea of the rate at which new features are added to
> Texinfo.

There is a single word "directives" in
https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:

    . #line directives are recognized.

>> In other words, I do not see why Org should support every single Texinfo
>> markup. We just need to provide enough support to be on par in terms of
>> the needs of manual authors.
>
> GNU Manuals use a large portion of what Texinfo provides, and limiting
> what they could use when they write in Org would mean some Texinfo
> features cannot be used that way, which is a disadvantage.  People
> will have to think twice before they switch to Org, because at some
> point they might want to use a feature you decided not to support.

I understand. However, new features are added to Texinfo for a reason.
If the same reason is valid for Org, equivalent features may be added.
Similarly, we may add features to Org that have no equivalent in
Texinfo. So, this is a normal development process with new features
being proposed, discussed, and maybe added.

From my point of view, equivalence with Texinfo is not something we care
as much about yet. It is just a general direction we can use for Org
development. I deem it useful even if we cannot reach all the Texinfo
features at the end. But if we can, we can _later_ see further about how
we want to incorporate (or not) new Texinfo features.

>> > That was for Org-to-Texinfo conversion.  Here we are talking about
>> > Org-to-Info, which is much more work.
>> 
>> Why so?
>
> Because the production rules are much more complex.  I suggest to take
> a look at the Perl source code of texi2any to see what it entails.

AFAIU, the conversion is done in
/usr/share/texinfo/Texinfo/Convert/Info.pm
I do not see anything extraordinary. Org uses similar approach - working
with parse tree.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 16:12:57 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Not the output format, but the language: new directives and sometimes
> >> > markup are added with every release of Texinfo.  Just read their NEWS
> >> > file.
> >> 
> >> The way I see Org markup extension would make it easy to users add new
> >> custom markup, as needed. Then, no frequent changes to the base markup
> >> will be necessary to accommodate for less common use cases.
> >
> > You missed the "new directives" part.  I do recommend reading their
> > NEWS to get the idea of the rate at which new features are added to
> > Texinfo.
> 
> There is a single word "directives" in
> https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:
> 
>     . #line directives are recognized.

Look for changes whose heading is "Language:".

> >> In other words, I do not see why Org should support every single Texinfo
> >> markup. We just need to provide enough support to be on par in terms of
> >> the needs of manual authors.
> >
> > GNU Manuals use a large portion of what Texinfo provides, and limiting
> > what they could use when they write in Org would mean some Texinfo
> > features cannot be used that way, which is a disadvantage.  People
> > will have to think twice before they switch to Org, because at some
> > point they might want to use a feature you decided not to support.
> 
> I understand. However, new features are added to Texinfo for a reason.
> If the same reason is valid for Org, equivalent features may be added.

The reason why features are added to Texinfo is that those features
are useful for writing software documentation.  After all, this is the
main purpose of Texinfo.  So I cannot see how the same reasons could
not be valid for Org -- assuming that Org wants to support writing
software documentation, not just the Emacs manuals in their current
form.

> Similarly, we may add features to Org that have no equivalent in
> Texinfo.

When you add features to Org, no one needs to catch up with them.  But
when Texinfo adds new features, users of Org who use Org for writing
software documentation will expect those features to be supported,
since Org will (AFAIU) claim that it can be used for that.

> >> Why so?
> >
> > Because the production rules are much more complex.  I suggest to take
> > a look at the Perl source code of texi2any to see what it entails.
> 
> AFAIU, the conversion is done in
> /usr/share/texinfo/Texinfo/Convert/Info.pm
> I do not see anything extraordinary. Org uses similar approach - working
> with parse tree.

Fine, then I guess you are all set and can simply forget what I said.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> There is a single word "directives" in
>> https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:
>> 
>>     . #line directives are recognized.
>
> Look for changes whose heading is "Language:".

I did, and in many cases Texinfo added things that are already available
in Org. Otherwise, these changes are not really that frequent. I do not
see much problem catching up.

>> I understand. However, new features are added to Texinfo for a reason.
>> If the same reason is valid for Org, equivalent features may be added.
>
> The reason why features are added to Texinfo is that those features
> are useful for writing software documentation.  After all, this is the
> main purpose of Texinfo.  So I cannot see how the same reasons could
> not be valid for Org -- assuming that Org wants to support writing
> software documentation, not just the Emacs manuals in their current
> form.

Org is different from Texinfo and certain things that are impossible
with Texinfo and require changing grammar are trivial in Org without
upstream changes. I really do not see the raised problem as real
problem.

To clarify, my stance on this idea with better support for authoring
software manuals is to be done in several stages:


   [ we are at this point now ]

1. Provide basic programmable constructs on Org syntax level. This is
   generally useful and should be done anyway for various reasons.
   
2. Use the programmable constructs as a basis for an optional library
   that will allow software manual-specific markup in Org, aiming for an
   equivalence of export output. This can be useful, at least, for
   people using Org to write README files.

   [ here, we may re-consider the whole idea and see if we should move
     further ]

3. Look into fine-tuning commands provided by Texinfo (like formatting
   and layout control) and see if they can (or should) be integrated
   into Org.

   [ the problem you describe is to be considered seriously here, and I
      do not see why it should be the show stopper ]

4. Look if people are interested to switch from Texinfo to Org.

5. Look about maintaining parity between Texinfo and Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Those proposed steps seem reasonable to me in principle.
It is just a matter of making them work.

I don't see any reason why any of them should turn out impossible,
except _maybe_ the task of finding good syntaxes to use for the
various constructs new in Org format.  The rest is just writing
software, and we know how to do that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I understand. However, new features are added to Texinfo for a reason.

That is correct.  We added each of these features to Texinfo because
some GNU manual needed it.  Indeed, the simple way to find out all the
features GNU manuals need is to look at the Texinfo manual and see
what they required us to support.

We're comsidering the idea of making Org format adequate for GNU
mamuals.  Currently it can't express the distinctions that are needed
in order to produce the proper output for each of the output formats.

If we want to achieve that, we come to this question:

* What specific extension syntax would we use for Org equivalents
of the Texinfo constructs that Org currently cannot express>

  > The way I see Org markup extension would make it easy to users add new
  > custom markup, as needed. Then, no frequent changes to the base markup
  > will be necessary to accommodate for less common use cases.

I see a possible ambiguity and point of confusion.  When you say,
"extension", do you mean "a package that gets loaded on top of
ordinary Org mode"?  That's what I thought it meant.

Implementing some of the Texinfo constructs in such a package, perhaps
called org-texinfo, is an implementation detail as far as I'm
concerned.

But now I think maybe you mean something else -- that you propose
to add some sort of limited macro definition facility and have the
missing Texinfo constructs be defined using that.  Is that it?

To be adequate for this job, the macro definition facility needs to be
more powerful than they usually are.  The expansion of one construct
needs to depend on the output format being generated, and sometimes
the expansion of construct A depends on whether it is inside construct
B.

If the facility can do that, I think it will suffice for nearly all of
the missing Texinfo constructs.  If you think of this as a method to
simplify part of the implementation of Texinfo in Org, it may work.

But be prepared for exeptions, constructs that need special handling!
If you think of this as a way to keep Org itself free of Texinfo
impurities, it won't work.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > The way I see Org markup extension would make it easy to users add new
>   > custom markup, as needed. Then, no frequent changes to the base markup
>   > will be necessary to accommodate for less common use cases.
>
> I see a possible ambiguity and point of confusion.  When you say,
> "extension", do you mean "a package that gets loaded on top of
> ordinary Org mode"?  That's what I thought it meant.

A package or user Elisp snippet.

For example, Org currently allows extending hyperlinks like

     (defun org-man-export (link description format _)
       "Export a man page link from Org files."
       (let ((path (format "http://man.he.net/?topic=%s&section=all" link))
             (desc (or description link)))
         (pcase format
           (`html (format "<a target=\"_blank\" href=\"%s\">%s</a>" path desc))
           (`latex (format "\\href{%s}{%s}" path desc))
           (`texinfo (format "@uref{%s,%s}" path desc))
           (`ascii (format "%s (%s)" desc path))
           (t path))))

          (org-link-set-parameters "man" :export #'org-man-export)

Then, <man:emacs> links will be formatted arbitrarily during export.

The same idea will be for markup syntax:

@var{variable-name} will, in future, be defined as

    (org-markup-set-parameters "var" :export #'my-export-function-for-var)

@var is probably something we will have within Org, but if one needs
some weird markup for a specific manual, it will equally be possible to
define

@my-special-markup{contents}

via (org-markup-set-parameters "my-special-markup" :export #'custom-export-formatter)

> Implementing some of the Texinfo constructs in such a package, perhaps
> called org-texinfo, is an implementation detail as far as I'm
> concerned.
>
> But now I think maybe you mean something else -- that you propose
> to add some sort of limited macro definition facility and have the
> missing Texinfo constructs be defined using that.  Is that it?

We might allow in-document definitions like:

#+markup[html]: my-special-markup <foo>%s</foo>
#+markup[latex]: my-special-markup \foo{%s}
...

but the `org-markup-set-parameters' is what we preliminarily agreed upon
for now. Further features are to be discussed later.

> To be adequate for this job, the macro definition facility needs to be
> more powerful than they usually are.  The expansion of one construct
> needs to depend on the output format being generated, and sometimes
> the expansion of construct A depends on whether it is inside construct
> B.

> If the facility can do that, I think it will suffice for nearly all of
> the missing Texinfo constructs.  If you think of this as a method to
> simplify part of the implementation of Texinfo in Org, it may work.

Org is already capable to provide access to the full parse tree when
expanding links with custom :export function. The same can be done for
markup constructs. Much more difficult if one wants in-document
definitions though.

> But be prepared for exeptions, constructs that need special handling!
> If you think of this as a way to keep Org itself free of Texinfo
> impurities, it won't work.

May you elaborate about special handling?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, Org currently allows extending hyperlinks like

  >      (defun org-man-export (link description format _)
  >        "Export a man page link from Org files."

This method like it should be able to handle the various output formats
for Texinfo constructs -- as long as once in a while we can define
a function that's a little ideosyncratic, for a construct that is
a little more irregular in its behavior.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Currently, Org mode does not support all the markup supported by
  > texinfo. But we will change that.

  > Meanwhile, Org implements info/manual export by ad-hoc extension
  > allowing to add texinfo-specific markup. This is not ideal, and it does
  > not need to be done after we do add the missing markup (Org equivalents
  > of @kbd, @var, and other)

That is good news -- it means we have the same goal in mind.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> That's what we plan to add to Org. Not just because of this
> discussion;

Why?  Where can I find this discussion?

If the main reasons for supporting inline markup like @NAME{CONTENT}
(or similar) are the current fontification shortcomings, I'm skeptic.

The .org falls in the category of lightweight markup formats and I
believe it should stick close to this definition.

That is, natively support *bold*, /emphasis/, etc. without adding
heavy more constructs like @NAME{CONTENT} -- even optionnally.

If we need to support @defn{FUNCTION} for .texi export, what about
considering ~~FUNCTION~~ ?  Using two chars at the beg and end is a
natural extension of the current markup syntax and should spare us
some ambiguity that we have with one-char constructs.

> there are other reasons why we want such custom inline markup.

Thanks for pointing at these discussions.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The .org falls in the category of lightweight markup formats and I
  > believe it should stick close to this definition.

If people keep this unchanged, Org mode will remain useful for the
tasks it can do today.  But it won't be able to do the job for which
we now use Texinfo.

There are many different Texinfo markup constructs, so to do the job
of Texinfo, Org mode would need to be able to distinguish them all.

So the basic question is: to extend Org mode to do that job, or not?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > The .org falls in the category of lightweight markup formats and I
>   > believe it should stick close to this definition.
>
> If people keep this unchanged, Org mode will remain useful for the
> tasks it can do today.  But it won't be able to do the job for which
> we now use Texinfo.
> There are many different Texinfo markup constructs, so to do the job
> of Texinfo, Org mode would need to be able to distinguish them all.
>
> So the basic question is: to extend Org mode to do that job, or not?

I don't think that we need to compromise here. We can keep Org
lightweight yet allowing arbitrary markup.

We already have arbitrary environment markup (special blocks):

#+begin_<block type>
...
#+end_<block type>

like

#+begin_center
#+begin_abstract
#+begin_myweirdlatexenvironment
etc

These are all valid Org markups, but only a small subset of them has
special handling by Org. Other free-form special blocks are simply
ignored (or handled generically), for example, on export, unless the
specific export backend implements special handling for a given block
type.

Supporting TeXinfo-specific inline markup can be done using the same
approach, without complicating core Org markup specifications.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Alfred M. Szmidt]

     > The .org falls in the category of lightweight markup formats and I
     > believe it should stick close to this definition.

   If people keep this unchanged, Org mode will remain useful for the
   tasks it can do today.  But it won't be able to do the job for which
   we now use Texinfo.

   There are many different Texinfo markup constructs, so to do the job
   of Texinfo, Org mode would need to be able to distinguish them all.

   So the basic question is: to extend Org mode to do that job, or not?

It would make more sense to extend Texinfo to allow for org-mode like
markup (or lighter markup), than try to stuff Texinfo into org-mode.
Texinfo is despite its issues, very nice and easy to write.

Handling the version chasm between existing Texinfo and "new" Texinfo
can easily be solved by having a small @texinfo-version thing at the
top.

But replacing Texinfo outright with some org-mode variant seems like yak
shaving.  The big issue with Texinfo is that it is hard to make your own
complicated constructs (like @defn ..), and the lack of macro support of
any kind.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For inline marks, Texinfo uses a conventient generic and extensible
  > form: @NAME{CONTENT}.

  > "markdown-style" formats, on the contrary, generally support a closed
  > set of marks that use delimited based ad-hoc syntax, such as *foo*,
  > =foo=, "foo", `foo' and so on.

The challenge is to extend Org format with an open-ended syntax that 
could handle many different inline markup opertions.

ISTR that Sphinx has an extension which is good for such markup.
Could that same syntax fit into Org mode too?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> ISTR that Sphinx has an extension which is good for such markup.
> Could that same syntax fit into Org mode too?

May you provide some link that describes the extension you are referring
to?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > ISTR that Sphinx has an extension which is good for such markup.
  > > Could that same syntax fit into Org mode too?

  > May you provide some link that describes the extension you are referring
  > to?

Sorry, I may have said that in an unclear way.  It is not an extension
in the sense of an add-on to Sphinx that you need to install
separately.  It is a standard part of Sphinx and described in the
manual,

It is an extension in the sense that they added it to the simple markup
language that they started with.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > May you provide some link that describes the extension you are referring
>   > to?
>
> Sorry, I may have said that in an unclear way.  It is not an extension
> in the sense of an add-on to Sphinx that you need to install
> separately.  It is a standard part of Sphinx and described in the
> manual,
>
> It is an extension in the sense that they added it to the simple markup
> language that they started with.

Sorry, but I can only see a closed list of inline emphasis constructs in
https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

I may be misunderstanding you.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Yuri Khan]

On Mon, 28 Aug 2023 at 17:05, Ihor Radchenko <yantar92@posteo.net> wrote:

> Sorry, but I can only see a closed list of inline emphasis constructs in
> https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

The `interpreted text`:role: construct is extensible, by defining new
role codes.

Re: Org mode and Emacs

[Ihor Radchenko]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Mon, 28 Aug 2023 at 17:05, Ihor Radchenko <yantar92@posteo.net> wrote:
>
>> Sorry, but I can only see a closed list of inline emphasis constructs in
>> https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup
>
> The `interpreted text`:role: construct is extensible, by defining new
> role codes.

Thanks! It is similar to what we already plan for:
https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

Except that we also plan to allow assigning optional attributes to the
custom markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Sorry, but I can only see a closed list of inline emphasis constructs in
  > https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

  > I may be misunderstanding you.

I don't remember it any more precisely, so I can't tell you more.

I notice that you're talking about restructuredtext and I am talking
about Sphinx.  Are the two related?  I don't remember.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > Sorry, but I can only see a closed list of inline emphasis constructs in
>   > https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup
>
>   > I may be misunderstanding you.
>
> I don't remember it any more precisely, so I can't tell you more.
>
> I notice that you're talking about restructuredtext and I am talking
> about Sphinx.  Are the two related?  I don't remember.

When I looked up Sphinx, the documentation directed me to
reStructuredtext for markup description:

https://www.sphinx-doc.org/en/master/
    Sphinx uses the reStructuredText markup language by default, and can
    read MyST markdown via third-party extensions. 

Also, Yuri pointed me to

>> The `interpreted text`:role: construct is extensible, by defining new
>> role codes.

which is similar to what we planned for Org: @role{interpreted text}

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I notice that you're talking about restructuredtext and I am talking
  > > about Sphinx.  Are the two related?  I don't remember.

  > When I looked up Sphinx, the documentation directed me to
  > reStructuredtext for markup description:

I guess you must be right.  Thanks.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Bastien Guerry]

Richard Stallman <rms@gnu.org> writes:

>   > "markdown-style" formats, on the contrary, generally support a closed
>   > set of marks that use delimited based ad-hoc syntax, such as *foo*,
>   > =foo=, "foo", `foo' and so on.
>
> The challenge is to extend Org format with an open-ended syntax that 
> could handle many different inline markup opertions.

Yes, that's the idea behind using one-char constructs for basic inline
markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Po Lu]

Bastien Guerry <bzg@gnu.org> writes:

> Yes, that's the idea behind using one-char constructs for basic inline
> markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
> specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

I forsee a problem with this approach: users will be compelled to study
and memorize dozens of two character combinations that closely resemble
one another.  Contrast this to Texinfo, where:

  The key sequence @kbd{C-x C-f}
  The string @samp{"sample text"}
  The function @code{defun}
  The formula @var{X}
  An @dfn{example}

leaves no latitude for visual ambiguity, whereas it will not take much
for one to mistake =~ for -~ or ~~.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think it can, but a typical Texinfo manual these days uses many more
  > commands, not just those.  So this list is insufficient to judge these
  > issues.  The actual list is much longer.

You're right.  But I think that if Org can conveniently handle the
ones I listed, giving each its Texinfo behavior in each output format,
that means it has the capacity to handle any additional inline markup
features too.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 14:35:00 +0800
> 
> I am wondering how many Emacs developers contributed to Org in the past.
> In particular, before we switched away from .texi manual.

You can easily see that with the following command:

  git log -- doc/misc/org.texi

Re: Org mode and Emacs

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2022-09-26 09:35]:
> I am now more concerned about Org contributors. I am not sure if we have
> a luxury to push our contributors into GNU project by forcing them to
> learn texinfo in addition to Org (note that I do not say that we should
> not encourage people to contribute to GNU; just that we should not
> discourage people from becoming Org contributors in the first
> place).

You may have contributors write it in Org and then convert Org to
texinfo by using pandoc:

pandoc -f org -t texinfo your-file.org


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Org mode and Emacs

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2022-09-26 09:35]:
>> I am now more concerned about Org contributors. I am not sure if we have
>> a luxury to push our contributors into GNU project by forcing them to
>> learn texinfo in addition to Org (note that I do not say that we should
>> not encourage people to contribute to GNU; just that we should not
>> discourage people from becoming Org contributors in the first
>> place).
>
> You may have contributors write it in Org and then convert Org to
> texinfo by using pandoc:
>
> pandoc -f org -t texinfo your-file.org

This is almost what we do now. Org contributors write in Org, and then we
use Org itself to convert to texinfo. Org files serve as the source
files.

What we are discussing here is whether Org should switch from Org
sources back to texinfo sources used in Emacs.

1. Switching to texinfo will improve Emacs coherence and potentially
   increase the pool of Emacs contributors familiar with texinfo (by
   pulling some of the Org contributors)
2. On the other hand, switching from Org to texinfo sources on Org side
   will require Org contributors to learn texinfo, which they do not
   really need now.
3. Yet, switching from Org to texinfo will allow Emacs contributors to
   send patches in familiar texinfo format to Org.

From the perspective of Emacs, using texinfo sources is a good thing.
From the perspective of Org, it is not that obvious.
Hence this discussion.

Note that we are now discussing this on emacs-devel and do not yet have
inputs from many Org users/contributors. However, we (Org devs) can get
inputs from Emacs devs, which is part of the information we need to see
what we want to do about Org vs. Texinfo sources.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Robert Pluim]

>>>>> On Mon, 26 Sep 2022 17:54:33 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> 1. Switching to texinfo will improve Emacs coherence and potentially
    Ihor>    increase the pool of Emacs contributors familiar with texinfo (by
    Ihor>    pulling some of the Org contributors)
    Ihor> 2. On the other hand, switching from Org to texinfo sources on Org side
    Ihor>    will require Org contributors to learn texinfo, which they do not
    Ihor>    really need now.
    Ihor> 3. Yet, switching from Org to texinfo will allow Emacs contributors to
    Ihor>    send patches in familiar texinfo format to Org.

    Ihor> From the perspective of Emacs, using texinfo sources is a good thing.
    Ihor> From the perspective of Org, it is not that obvious.
    Ihor> Hence this discussion.

As someone who has written the odd bit of Emacs documentation and
(less) org-mode documentation, I find the cognitive load of texinfo or
org to be about the same [1][2]; whichever one I wrote most recently is
relatively easy to use, and the other I need to relearn. I suspect the
same is true for most casual submitters of documentation patches.

Robert

Footnotes:
[1]  In a lot of cases itʼs "scroll around to see what the rest of the
     document is doing and just copy that", which is perhaps
     inefficent of my time, but works.

[2]  The load would be less if I didnʼt *also* have to write Markdown
     for work, but Emacs help a lot with that.  😀

-- 

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is almost what we do now. Org contributors write in Org, and then we
  > use Org itself to convert to texinfo. Org files serve as the source
  > files.

In order for that to produce proper Texinfo source as output, there
would need to be, in Org format, ways to express the many distinctions
that Texinfo source needs to make.  Yesterday I posted a list of
Texinfo constructs and suggested adding to Org mode ways to
distinguish them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> In order for that to produce proper Texinfo source as output, there
> would need to be, in Org format, ways to express the many distinctions
> that Texinfo source needs to make.  Yesterday I posted a list of
> Texinfo constructs and suggested adding to Org mode ways to
> distinguish them.

I have just forwarded that email to Org ML.
https://list.orgmode.org/87bkqx4jyg.fsf@localhost/T/#u

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Speaking as the Chief GNUisance, rssponsible for GNU Project
standards, I would be happy to adopt an upgraded Org format as a new
standard source format for GNU manuals, _provided_ Org format has been
extended with the capability to express all the constructions and
distinctions that Texinfo can express, generate all the output formats
Texinfo can generate, and use TeX to make beautiful printed output.

Texinfo can generate these output formats: Info files, HTML, ASCII
text, and DVI and PDF files via TeX.

Texinfo provides numerous subtle distinctions that show up clearly in
each of these output formats.  Compare, for example, @var, @dfn and
@emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I am sure people can extend Org software to handle these semantic
distinctions and generate these output formats.  Since it has been
done once, it can be done again.  But the work is not trivial.

The work has to start by designing what the extended Org format will look
like.  That part is the crucial part; once it has been specified,
people can work independently to implement various parts of handling
that format.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

[HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Dear List,

I am forwarding an official email from RMS changing subject line to more
descriptive. See below.

For some context, in order to support specialized syntax for manuals, we
may first need to implement the discussed special blocks export and
inline special blocks:
1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/

The above links aim to introduce export functionality that we now have
for links to special blocks and new custom markup elements. I am
referring to
1. Ability to create new custom element types programmatically 
2. Ability to define how to :export the custom element types

Similar to `org-link-set-parameters'.

Patches and more concrete ideas are welcome!

-------------------- Start of forwarded message --------------------
From: Richard Stallman <rms@gnu.org>
To: Tim Cross <theophilusx@gmail.com>
Cc: emacs-devel@gnu.org
Subject: Re: Org mode and Emacs
Date: Mon, 26 Sep 2022 08:10:03 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Speaking as the Chief GNUisance, rssponsible for GNU Project
standards, I would be happy to adopt an upgraded Org format as a new
standard source format for GNU manuals, _provided_ Org format has been
extended with the capability to express all the constructions and
distinctions that Texinfo can express, generate all the output formats
Texinfo can generate, and use TeX to make beautiful printed output.

Texinfo can generate these output formats: Info files, HTML, ASCII
text, and DVI and PDF files via TeX.

Texinfo provides numerous subtle distinctions that show up clearly in
each of these output formats.  Compare, for example, @var, @dfn and
@emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I am sure people can extend Org software to handle these semantic
distinctions and generate these output formats.  Since it has been
done once, it can be done again.  But the work is not trivial.

The work has to start by designing what the extended Org format will look
like.  That part is the crucial part; once it has been specified,
people can work independently to implement various parts of handling
that format.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



-------------------- End of forwarded message --------------------

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Samuel Wales]

i really wish i could engage or even post drafts of extensible syntax
stuff that i had replied to many years ago and recently.

On 9/29/22, Ihor Radchenko <yantar92@gmail.com> wrote:
> Dear List,
>
> I am forwarding an official email from RMS changing subject line to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for manuals, we
> may first need to implement the discussed special blocks export and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> -------------------- Start of forwarded message --------------------
> From: Richard Stallman <rms@gnu.org>
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs
> Date: Mon, 26 Sep 2022 08:10:03 -0400
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a new
> standard source format for GNU manuals, _provided_ Org format has been
> extended with the capability to express all the constructions and
> distinctions that Texinfo can express, generate all the output formats
> Texinfo can generate, and use TeX to make beautiful printed output.
>
> Texinfo can generate these output formats: Info files, HTML, ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I am sure people can extend Org software to handle these semantic
> distinctions and generate these output formats.  Since it has been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format will look
> like.  That part is the crucial part; once it has been specified,
> people can work independently to implement various parts of handling
> that format.
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> -------------------- End of forwarded message --------------------
>
> --
> Ihor Radchenko,
> Org mode contributor,
> Learn more about Org mode at https://orgmode.org/.
> Support Org development at https://liberapay.com/org-mode,
> or support my work at https://liberapay.com/yantar92
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Thomas S. Dye]

Good news.  Thanks Ihor.  And thanks to Nicolas for his work on 
ox.

Ihor Radchenko <yantar92@gmail.com> writes:

> Dear List,
>
> I am forwarding an official email from RMS changing subject line 
> to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for 
> manuals, we
> may first need to implement the discussed special blocks export 
> and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we 
> now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically 
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> From: Richard Stallman <rms@gnu.org>
> Subject: Re: Org mode and Emacs
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 08:10:03 -0400 (3 days, 17 hours, 36 
> minutes ago)
> Flags: seen, list
> Maildir: /TSD-ONLINE/INBOX
> List: emacs-orgmode.gnu.org
>
> [[[ To any NSA and FBI agents reading my email: please consider 
> ]]]
> [[[ whether defending the US Constitution against all enemies, 
> ]]]
> [[[ foreign or domestic, requires you to follow Snowden's 
> example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a 
> new
> standard source format for GNU manuals, _provided_ Org format 
> has been
> extended with the capability to express all the constructions 
> and
> distinctions that Texinfo can express, generate all the output 
> formats
> Texinfo can generate, and use TeX to make beautiful printed 
> output.
>
> Texinfo can generate these output formats: Info files, HTML, 
> ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up 
> clearly in
> each of these output formats.  Compare, for example, @var, @dfn 
> and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and 
> @key.
>
> I am sure people can extend Org software to handle these 
> semantic
> distinctions and generate these output formats.  Since it has 
> been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format 
> will look
> like.  That part is the crucial part; once it has been 
> specified,
> people can work independently to implement various parts of 
> handling
> that format.
>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> ----------
All the best,
Tom

-- 
Thomas S. Dye
https://tsdye.online/tsdye

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Christopher M. Miles]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]


Glad to hear this news which Org mode is used in more places.

-- 

[ stardiviner ]
I try to make every word tell the meaning that I want to express without misunderstanding.

Blog: https://stardiviner.github.io/
IRC(libera.chat, freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 30/09/2022 10:31, Ihor Radchenko wrote:
> 
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I have not read the emacs-devel thread, so I may ask about something 
that already has been discussed.

Are there cases when texinfo may use nested formatting commands of the 
same type, something like @samp{a @code{b @samp{c} d} e}? My concern is 
that current org-element parser may be a blocker.

Another point is that most of the mentioned commands a close to 
verbatim, but Org has much more special characters recognized as markup 
and no markup is allowed inside Org verbatim snippets. Escaping (by zero 
width spaces?) of code and samples may be prohibitively inconvenient in 
Org if markup should be recognized inside.

One more point is external tools like pandoc export from Org to other 
formats. When Org extensions are implemented in elisp, such tools become 
hardly usable. Unsure if some kind of declarative style sheets will be 
enough.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 30/09/2022 10:31, Ihor Radchenko wrote:
>> 
>> Texinfo provides numerous subtle distinctions that show up clearly in
>> each of these output formats.  Compare, for example, @var, @dfn and
>> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I have not read the emacs-devel thread, so I may ask about something 
> that already has been discussed.
>
> Are there cases when texinfo may use nested formatting commands of the 
> same type, something like @samp{a @code{b @samp{c} d} e}? My concern is 
> that current org-element parser may be a blocker.

AFAIK, the only nested construct in Texinfo spec is
@acronym{GNU, @acronym{GNU}'s Not Unix}
https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html#g_t_0040acronym

But it should not be a blocker. We can use two different elements for
this instead or employ the requirement of balanced parenthesis inside,
similar to what we do in the HEADERS for inline src blocks
https://orgmode.org/worg/dev/org-syntax.html#Source_Blocks

> Another point is that most of the mentioned commands a close to 
> verbatim, but Org has much more special characters recognized as markup 
> and no markup is allowed inside Org verbatim snippets. Escaping (by zero 
> width spaces?) of code and samples may be prohibitively inconvenient in 
> Org if markup should be recognized inside.

We need a new special object type for markup that does not suffer from
the limitations of our current single-char-style *markup* constructs.
(It is not even solely motivated by this request from RMS; we just need
something to allow whitespace in verbatim boundaries)

> One more point is external tools like pandoc export from Org to other 
> formats. When Org extensions are implemented in elisp, such tools become 
> hardly usable. Unsure if some kind of declarative style sheets will be 
> enough.

We already have Org extensions for links. I envision supporting GNU
manuals in a similar fashion. Only a subset of universally useful
extensions will become the actual part of Org syntax spec. At the end,
Org should remain generic export format without going too far into
manual-specific requirements. Specific exports should be supported
optionally, similar to our ol-*.el libraries.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 10:30, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Another point is that most of the mentioned commands a close to
>> verbatim, but Org has much more special characters recognized as markup
>> and no markup is allowed inside Org verbatim snippets. Escaping (by zero
>> width spaces?) of code and samples may be prohibitively inconvenient in
>> Org if markup should be recognized inside.
> 
> We need a new special object type for markup that does not suffer from
> the limitations of our current single-char-style *markup* constructs.
> (It is not even solely motivated by this request from RMS; we just need
> something to allow whitespace in verbatim boundaries)

I do not remember if the following idea has been discussed. What about 
extending source blocks and inline source snippets to allow :results ast 
header argument that caused executing during export only and expects 
s-expression with AST branch that is included into parsed tree without 
intermediate Org markup representation? Then a special org-babel backend 
may be created to support new markup type. I am unsure however at which 
stage of export source blocks are executed, maybe too early to implement 
the idea. The advantage is that no extension of Org syntax is necessary.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> I do not remember if the following idea has been discussed. What about 
> extending source blocks and inline source snippets to allow :results ast 
> header argument that caused executing during export only and expects 
> s-expression with AST branch that is included into parsed tree without 
> intermediate Org markup representation? Then a special org-babel backend 
> may be created to support new markup type. I am unsure however at which 
> stage of export source blocks are executed, maybe too early to implement 
> the idea. The advantage is that no extension of Org syntax is necessary.

This will not help with new extendable markup elements. No new org
element types can be created this way. We still need generalized inline
markup object.

This can help with escaping syntax and spaces in verbatim.

I do recall you mentioned this idea in one of the earlier threads.
https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
Max Nikulin <manikulin@gmail.com> (2021-12-06)
Subject: Raw Org AST snippets for "impossible" markup

This idea is ok by itself, although we should not use the real AST;
rather some AST-like representation that can be fixed and then
translated to real AST. (I'd rather keep the possibility to change our
real AST; for example, I have plans to change property storage into
vector for faster access---current property list is slower than querying
the binary tree with all elements in buffer).

The downside of this idea is that it will force users into the Elisp
escape hell src_elisp(verbatim "this is \\-n just slash-n and we still
cannot use unmatched } here"}.

I'd rather have something more friendly to users, if possible.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 18:01, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> I do not remember if the following idea has been discussed. What about
>> extending source blocks and inline source snippets to allow :results ast
>> header argument that caused executing during export only and expects
>> s-expression with AST branch that is included into parsed tree without
>> intermediate Org markup representation? Then a special org-babel backend
>> may be created to support new markup type. I am unsure however at which
>> stage of export source blocks are executed, maybe too early to implement
>> the idea. The advantage is that no extension of Org syntax is necessary.
> 
> This will not help with new extendable markup elements. No new org
> element types can be created this way. We still need generalized inline
> markup object.
> 
> This can help with escaping syntax and spaces in verbatim.

It should be enough to create nested "code" or "verbatim" inline objects 
with some attribute like :class file, :class var, :class dfn, etc. 
Export backend may interpret this attribute to create proper texinfo 
commands or more precisely choose HTML element.

> I do recall you mentioned this idea in one of the earlier threads.
> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
> Max Nikulin (2021-12-06)
> Subject: Raw Org AST snippets for "impossible" markup

It was using links, not source blocks. Typing AST directly is a kind of 
last resort. Now I am writing about some higher level markup with a 
parser that allows nested element and uses just a few special characters 
to denote commands. It is a kind of trade-off between brevity of 
lightweight markup and flexibility of more verbose language with ability 
to more precisely express intentions.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> This can help with escaping syntax and spaces in verbatim.
>
> It should be enough to create nested "code" or "verbatim" inline objects 
> with some attribute like :class file, :class var, :class dfn, etc. 
> Export backend may interpret this attribute to create proper texinfo 
> commands or more precisely choose HTML element.
>
>> I do recall you mentioned this idea in one of the earlier threads.
>> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
>> Max Nikulin (2021-12-06)
>> Subject: Raw Org AST snippets for "impossible" markup
>
> It was using links, not source blocks. Typing AST directly is a kind of 
> last resort. Now I am writing about some higher level markup with a 
> parser that allows nested element and uses just a few special characters 
> to denote commands. It is a kind of trade-off between brevity of 
> lightweight markup and flexibility of more verbose language with ability 
> to more precisely express intentions.

I am not sure if I understand the details of what you are referring to.

Can you please illustrate how to use the described AST markup for the
following Texinfo snippet:

    By convention, the dynamic library for @var{language} is
    @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
    system-specific extension for dynamic libraries.


-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Fraga, Eric]

Hello all,

I like the idea of inline special blocks.  Given that we have recently
introduced [cite:...] as new syntax, could we generalise this and allow
any xxx in [xxx:...]?

With this, the example Max gave from texinfo:

>     By convention, the dynamic library for @var{language} is
>     @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>     system-specific extension for dynamic libraries.

could look like

   By convention, the dynamic library for [var:language] is
   [code:libtree-{sitter}-[var:"language"].[var:ext]], where [var:ext]
   is the system-specific extension for dynamic libraries.

noting the recursive embedded syntax.  (and not knowing texinfo, I've
assumed that @{ and @} are escapes for the braces but could be something
else.)

Obviously, this would be a breaking change for any documents that
actually had anything along the lines of [xxx:...] in their text.

Just musing out loud. ;-) And procrastinating from preparing my lectures
for tomorrow... Feel free to ignore of course.

eric
-- 
: Eric S Fraga, with org release_9.5.5-851-ge9781f in Emacs 29.0.50

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> I like the idea of inline special blocks.  Given that we have recently
> introduced [cite:...] as new syntax, could we generalise this and allow
> any xxx in [xxx:...]?

This syntax will make it difficult to pass optional arguments, like in
@abbr{FSF, Free Software Foundation}. Also, escaping "]" will be tricky.

> With this, the example Max gave from texinfo:
>
>>     By convention, the dynamic library for @var{language} is
>>     @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>>     system-specific extension for dynamic libraries.
>
> could look like
>
>    By convention, the dynamic library for [var:language] is
>    [code:libtree-{sitter}-[var:"language"].[var:ext]], where [var:ext]
>    is the system-specific extension for dynamic libraries.

I am thinking about something like

_name{<contents>}
_name{{<contents>}}  <--- extra {} is added as needed for escaping
_name[:key value ...]{<contents>}

The syntax idea is to follow the relevance between [[links]] and [cite:citations]
but here we have src_name[...]{...} and _name[...]{<contents>} instead.

    By convention, the dynamic library for _var{language} is
    _code{{libtree-{sitter}-_var{"language"}._var{ext}}}, where _var{ext}
    is the system-specific extension for dynamic libraries.

We may even follow Max's idea about AST and make it so that
_bold{contents} will be parsed just like *contents*.

> noting the recursive embedded syntax.  (and not knowing texinfo, I've
> assumed that @{ and @} are escapes for the braces but could be something
> else.)

Yep, @{ and @} are the escaped { and } in Texinfo.

-- 
Ihor Radchenko aka yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Fraga, Eric]

On Sunday,  2 Oct 2022 at 21:02, Ihor Radchenko wrote:
>
> _name{<contents>}
> _name{{<contents>}}  <--- extra {} is added as needed for escaping
> _name[:key value ...]{<contents>}

I'd be okay with this as well.


-- 
: Eric S Fraga, with org release_9.5.5-851-ge9781f in Emacs 29.0.50

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Juan Manuel Macías]

Ihor Radchenko writes:

> _name{<contents>}
> _name{{<contents>}}  <--- extra {} is added as needed for escaping
> _name[:key value ...]{<contents>}
>
> The syntax idea is to follow the relevance between [[links]] and [cite:citations]
> but here we have src_name[...]{...} and _name[...]{<contents>} instead.

And how about this:

%_name{<contents>}
%_name{{<contents>}}  <--- extra {} is added as needed for escaping
%_name[:key value ...]{<contents>}

or any other symbol instead of "%" ?

N.B.: I must admit this is more for an "aesthetic" reason. Although
perhaps it can be useful to search in the documents.

Best regards,

Juan Manuel

--
--
------------------------------------------------------
Juan Manuel Macías

https://juanmanuelmacias.com

https://lunotipia.juanmanuelmacias.com

https://gnutas.juanmanuelmacias.com

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> And how about this:
>
> %_name{<contents>}
> %_name{{<contents>}}  <--- extra {} is added as needed for escaping
> %_name[:key value ...]{<contents>}
>
> or any other symbol instead of "%" ?
>
> N.B.: I must admit this is more for an "aesthetic" reason. Although
> perhaps it can be useful to search in the documents.

I have to admit that I am not a big fan of underscore in _name myself.
Just wanted to keep some resemblance of src_name{<contents>} yet not
making things too verbose (like block_name{<contents>}).

If I were to choose an alternative symbol other than "_", I'd choose
"@":

@name{<contents>}
@name{{<contents}}
@name[:key value ...]{<contents>}

1. It is similar to Texinfo
2. It does not clash with TeX
3. We already use @ in the inline export snippets.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Juan Manuel Macías]

Ihor Radchenko writes:

> If I were to choose an alternative symbol other than "_", I'd choose
> "@":
>
> @name{<contents>}
> @name{{<contents}}
> @name[:key value ...]{<contents>}
>
> 1. It is similar to Texinfo
> 2. It does not clash with TeX
> 3. We already use @ in the inline export snippets.

I like the "@" alternative a lot. And I agree with all three points. It is
also compact without losing clarity, and does not give the feeling of a
blank space before, as in the case of "_".

Best regards,

Juan Manuel 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Rick Lupton]

Hi Ihor and all,

I wonder if you have seen Pollen’s approach to this? https://docs.racket-lang.org/pollen/pollen-command-syntax.html

There are two separate ideas used there which seemed related to this discussion. I’m not sure if they are useful in the org context.  

1. The use of a special character (◊ by default) which introduces a command/inline special block. I don’t know if this would be worth considering as an alternative to @ (which also seems reasonable) to avoid ambiguity with other syntax. As the link above discusses it’s harder to type but there are solutions. 

2. Making it easy to define custom functions that produce org syntax. A bit like perhaps Max's suggestion to use source blocks, but instead of writing AST-like structure directly in the document where you want it, you can call a previously defined function to build it. This is similar to org macros but I’m not sure if they are so flexible as a lisp function. There is also the option to choose between passing arguments as lisp (in [ ]) or as markup (in { })

On Tue, 4 Oct 2022, at 9:28 PM, Juan Manuel Macías wrote:
> Ihor Radchenko writes:
>
>> If I were to choose an alternative symbol other than "_", I'd choose
>> "@":
>>
>> @name{<contents>}
>> @name{{<contents}}
>> @name[:key value ...]{<contents>}
>>
>> 1. It is similar to Texinfo
>> 2. It does not clash with TeX
>> 3. We already use @ in the inline export snippets.
>
> I like the "@" alternative a lot. And I agree with all three points. It is
> also compact without losing clarity, and does not give the feeling of a
> blank space before, as in the case of "_".
>
> Best regards,
>
> Juan Manuel

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

"Rick Lupton" <rclsub@fastmail.com> writes:

> I wonder if you have seen Pollen’s approach to this? https://docs.racket-lang.org/pollen/pollen-command-syntax.html

Thanks for the interesting reference!
After skimming through the syntax described in the link, I feel like
most of the features are already present in Org syntax or being
discussed here.

> There are two separate ideas used there which seemed related to this discussion. I’m not sure if they are useful in the org context.  
>
> 1. The use of a special character (◊ by default) which introduces a command/inline special block. I don’t know if this would be worth considering as an alternative to @ (which also seems reasonable) to avoid ambiguity with other syntax. As the link above discusses it’s harder to type but there are solutions. 

I do understand the idea behind ◊, but it is the design decision we
would need to make before creating Org syntax. At this point, we are not
going to change the Org syntax concepts just for having the nice
feature. There is no point to solve the escaping problem just for a
single new Org syntax element and not changing the rest of the syntax. 

So, I'd prefer to keep "@" or other symbol we already use in the
existing Org elements/objects.

For escaping @ inside, we already have zero-width, an alternative idea
with special \-- entity, and we can introduce \atsymbol entity that
exports and displays as "@".

> 2. Making it easy to define custom functions that produce org syntax. A bit like perhaps Max's suggestion to use source blocks, but instead of writing AST-like structure directly in the document where you want it, you can call a previously defined function to build it. This is similar to org macros but I’m not sure if they are so flexible as a lisp function. There is also the option to choose between passing arguments as lisp (in [ ]) or as markup (in { })

We already have named code blocks to call arbitrary code in arbitrary
programming language and produce arbitrary results, including Org
markup. On export. I do not think that we should allow runtime markup
calculations on-the-fly due to performance reasons.

Also, Org macros do allow arbitrary elisp:

       As a special case, Org parses any replacement text starting with
    ‘(eval’ as an Emacs Lisp expression and evaluates it accordingly.
    Within such templates, arguments become strings.  Thus, the following
    macro


         #+MACRO: gnustamp (eval (concat "GNU/" (capitalize $1)))
    
    turns ‘{{{gnustamp(linux)}}}’ into ‘GNU/Linux’ during export.

Max's idea about AST structure is just an extra fallback to produce
really tricky cases. And since it is going to be an src block result,
all the features for src blocks will apply, including calling arbitrary
code (not just Elisp). I do not see this idea as being a part of normal
usage, just really weird border cases. (At least, it is my
understanding. Max may chime in with more clarifications).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 02/10/2022 11:59, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>> This can help with escaping syntax and spaces in verbatim.
>>
>> It should be enough to create nested "code" or "verbatim" inline objects
>> with some attribute like :class file, :class var, :class dfn, etc.
>> Export backend may interpret this attribute to create proper texinfo
>> commands or more precisely choose HTML element.
>>
>>> I do recall you mentioned this idea in one of the earlier threads.
>>> https://orgmode.org/list/sokqa3$e8s$1@ciao.gmane.io
>>> Max Nikulin (2021-12-06)
>>> Subject: Raw Org AST snippets for "impossible" markup
>>
>> It was using links, not source blocks. Typing AST directly is a kind of
>> last resort. Now I am writing about some higher level markup with a
>> parser that allows nested element and uses just a few special characters
>> to denote commands. It is a kind of trade-off between brevity of
>> lightweight markup and flexibility of more verbose language with ability
>> to more precisely express intentions.
> 
> I am not sure if I understand the details of what you are referring to.
> 
> Can you please illustrate how to use the described AST markup for the
> following Texinfo snippet:
> 
>      By convention, the dynamic library for @var{language} is
>      @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>      system-specific extension for dynamic libraries.

If you are asking how to represent such construct without introducing 
custom elements then (it may be e.g. :type, not :class) parsed AST 
should be like

     (code nil ("libtree-{sitter}-"
                (code (:class var) "\"language\"")
                "."
                (code (:class var) "ext")))

If there was some syntax for object attributes then simple cases would 
be like

     [[attr:(:class var)]]~language~

I have no idea concerning particular markup that can be used inside 
source blocks. It might be LaTeX-like commands as discussed in the 
sibling subthread or HTML (XML) based syntax that is more verbose than 
TeX-like notation.

     By convention, the dynamic library
     for src_alt{\code[class=var]{language}} is
 
src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
     where src_alt{\code[class=var]{ext}} is the
     system-specific extension for dynamic libraries.

or

     By convention, the dynamic library for
     src_alt{<code class="var">language</code>} is
     src_alt{<code>libtree-{sitter}-<code 
class="var">"language"</code>.<code class="var">ext</code></code>},
     where src_alt{<code class="var">ext</code>} is the
     system-specific extension for dynamic libraries.

Hypothetical "alt" babel language has default :results ast :export 
results header arguments to inject AST bypassing Org markup stage.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 02/10/2022 11:59, Ihor Radchenko wrote:
>> Can you please illustrate how to use the described AST markup for the
>> following Texinfo snippet:
>> 
>>      By convention, the dynamic library for @var{language} is
>>      @code{libtree-@{sitter@}-@var{"language"}.@var{ext}}, where @var{ext} is the
>>      system-specific extension for dynamic libraries.
>
> If you are asking how to represent such construct without introducing 
> custom elements then (it may be e.g. :type, not :class) parsed AST 
> should be like
>
>      (code nil ("libtree-{sitter}-"
>                 (code (:class var) "\"language\"")
>                 "."
>                 (code (:class var) "ext")))

This is not much different from @name[nil]{<contents>} idea, but
more verbose.

Also, more importantly, I strongly dislike the need to wrap the text
into "". You will have to escape \". And it will force third-party
parsers to re-implement Elisp sexp reader.

> If there was some syntax for object attributes then simple cases would 
> be like
>
>      [[attr:(:class var)]]~language~

I do not like this idea. It will require non-trivial changes in Org
parser and fontification.

Using dedicated object properties or at least inheriting properties from
:parent is the style we employ more commonly across the code:

@var{language}
or
@code[:class var]{language}
or
@attr[:class var]{~language~}

> I have no idea concerning particular markup that can be used inside 
> source blocks. It might be LaTeX-like commands as discussed in the 
> sibling subthread or HTML (XML) based syntax that is more verbose than 
> TeX-like notation.
>
>      By convention, the dynamic library
>      for src_alt{\code[class=var]{language}} is
>  
> src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
>      where src_alt{\code[class=var]{ext}} is the
>      system-specific extension for dynamic libraries.

I am against the idea of LaTeX-like commands. It will clash with
latex-fragment object type.
https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Fragments

> or
>
>      By convention, the dynamic library for
>      src_alt{<code class="var">language</code>} is
>      src_alt{<code>libtree-{sitter}-<code 
> class="var">"language"</code>.<code class="var">ext</code></code>},
>      where src_alt{<code class="var">ext</code>} is the
>      system-specific extension for dynamic libraries.

This style will indeed make things easier for the parser. But I find it
too verbose for practical usage. This is why I instead proposed the idea
with variable number of brackets: @code{{can have } inside}}.

> Hypothetical "alt" babel language has default :results ast :export 
> results header arguments to inject AST bypassing Org markup stage.

The problem with src block emitting AST is clashing with the way src
blocks work during export. What `org-export-as' does is replacing/adding
src block output into the actual Org buffer text before the parsing is
done.

Handling direct AST sexps will require a rewrite on how babel
integration with export works.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

It seems I completely failed trying to express my idea.

Instead of extending Org grammar (syntax), I suggest to change behavior 
of source blocks during export. In addition to current :results options, 
"ast" may be added. Its effect is that instead of adding text to export 
buffer that is parsed as Org markup, it causes insertion of a branch of 
syntax tree into original parse results. I admit, during export it may 
be necessary to iterate over source blocks one more time at a later stage.

Such source blocks should return "Org Syntax Tree", a simplified variant 
of org-element. It allows to change implementation details and e.g. to 
use vectors instead of lists for attributes in org-element. A converter 
from Org Syntax Tree to org-element should be implemented.

Certainly such format may be used directly as src_ost{(code (:class var) 
"language")} inline snippets or as

#+begin_src ost
   (code nil ("libtree-{sitter}-"
              (code (:class var) "\"language\"")
              "."
              (code (:class var) "ext")))
#+end_src

block-level elements. However I expect that it is the last resort option 
when there is no way to express desired construct in some other way.

I think, more convenient org-babel backends may be created to parse 
TeX-like (texinfo-like) or SGML-like (XML-like) syntax into Org Syntax 
Tree hierarchy. The essential idea is that outside of source blocks 
usual lightweight markup is used. Source blocks however have just a few 
special characters ([\{}], [@{}], or [&<>], etc.) to reduce issues with 
escaping for regular text or verbatim-like commands.

Some comments are inline.

On 03/10/2022 11:36, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> On 02/10/2022 11:59, Ihor Radchenko wrote:
>>
>> If you are asking how to represent such construct without introducing
>> custom elements then (it may be e.g. :type, not :class) parsed AST
>> should be like
>>
>>       (code nil ("libtree-{sitter}-"
>>                  (code (:class var) "\"language\"")
>>                  "."
>>                  (code (:class var) "ext")))
> 
> This is not much different from @name[nil]{<contents>} idea, but
> more verbose.
>
 > Also, more importantly, I strongly dislike the need to wrap the text
 > into "". You will have to escape \". And it will force third-party
 > parsers to re-implement Elisp sexp reader.

By this example I was trying to show how to express @var, @samp, @file 
without introducing of new custom objects. I do not see any problem with 
verbosity of such format, it may be used for really special cases only, 
while some more convenient markup is used for more simple cases.

>> If there was some syntax for object attributes then simple cases would
>> be like
>>
>>       [[attr:(:class var)]]~language~
> 
> I do not like this idea. It will require non-trivial changes in Org
> parser and fontification.
> 
> Using dedicated object properties or at least inheriting properties from
> :parent is the style we employ more commonly across the code:
> 
> @var{language}
> or
> @code[:class var]{language}
> or
> @attr[:class var]{~language~}

I do not mind to have some "span" object to assign attributes to its 
direct children. I used link-like prefix object just because a proof of 
concept may be tried with no changes in Org. It does not require support 
of nested objects. There is no existing syntax for such "span" objects, 
but perhaps it is not necessary and source blocks should be used instead 
for special needs.

>> I have no idea concerning particular markup that can be used inside
>> source blocks. It might be LaTeX-like commands as discussed in the
>> sibling subthread or HTML (XML) based syntax that is more verbose than
>> TeX-like notation.
>>
>>       By convention, the dynamic library
>>       for src_alt{\code[class=var]{language}} is
>>   
>> src_alt{\code{libtree-\{sitter\}-\code[class=var]{"language"}.\code[class=var]{ext}}},
>>       where src_alt{\code[class=var]{ext}} is the
>>       system-specific extension for dynamic libraries.
> 
> I am against the idea of LaTeX-like commands. It will clash with
> latex-fragment object type.
> https://orgmode.org/worg/dev/org-syntax.html#LaTeX_Fragments
> 
>> or
>>
>>       By convention, the dynamic library for
>>       src_alt{<code class="var">language</code>} is
>>       src_alt{<code>libtree-{sitter}-<code
>> class="var">"language"</code>.<code class="var">ext</code></code>},
>>       where src_alt{<code class="var">ext</code>} is the
>>       system-specific extension for dynamic libraries.
> 
> This style will indeed make things easier for the parser. But I find it
> too verbose for practical usage. This is why I instead proposed the idea
> with variable number of brackets: @code{{can have } inside}}.

Texinfo is TeX with \ replaced by @. Just another character has the 
category starting command. The important point is that while Org markup 
uses a lot of special characters (*/_+[]...) this flexible markup should 
use just a few ones. I do not see any obstacles to try texinfo-like 
markup. Source blocks allow to have several languages.

>> Hypothetical "alt" babel language has default :results ast :export
>> results header arguments to inject AST bypassing Org markup stage.
> 
> The problem with src block emitting AST is clashing with the way src
> blocks work during export. What `org-export-as' does is replacing/adding
> src block output into the actual Org buffer text before the parsing is
> done.
> 
> Handling direct AST sexps will require a rewrite on how babel
> integration with export works.

Yes, it will. I am evaluating feasibility of such change instead of 
extending of Org syntax for custom elements.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> It seems I completely failed trying to express my idea.
>
> Instead of extending Org grammar (syntax), I suggest to change behavior 
> of source blocks during export. In addition to current :results options, 
> "ast" may be added. Its effect is that instead of adding text to export 
> buffer that is parsed as Org markup, it causes insertion of a branch of 
> syntax tree into original parse results. I admit, during export it may 
> be necessary to iterate over source blocks one more time at a later stage.

So, do I understand it correctly that you are _not_ suggesting the AST
format _instead_ of the discussed inline blocks?

> Such source blocks should return "Org Syntax Tree", a simplified variant 
> of org-element. It allows to change implementation details and e.g. to 
> use vectors instead of lists for attributes in org-element. A converter 
> from Org Syntax Tree to org-element should be implemented.

Doesn't it effectively mean that we need to introduce yet another Org
element syntax---"Simplified AST"?

> Certainly such format may be used directly as src_ost{(code (:class var) 
> "language")} inline snippets or as
>
> #+begin_src ost
>    (code nil ("libtree-{sitter}-"
>               (code (:class var) "\"language\"")
>               "."
>               (code (:class var) "ext")))
> #+end_src
>
> block-level elements. However I expect that it is the last resort option 
> when there is no way to express desired construct in some other way.

I can foresee issues when we insert, say, code object in place of
paragraph element. The consequences might be drastic. Unless we just
allow users to shoot their own leg.

> I think, more convenient org-babel backends may be created to parse 
> TeX-like (texinfo-like) or SGML-like (XML-like) syntax into Org Syntax 
> Tree hierarchy. The essential idea is that outside of source blocks 
> usual lightweight markup is used. Source blocks however have just a few 
> special characters ([\{}], [@{}], or [&<>], etc.) to reduce issues with 
> escaping for regular text or verbatim-like commands.

This is not a bad idea, but I feel like it is getting a bit too far from
the syntax discussion herein. You might open a new thread about
importing foreign syntax into Org.

>>>       (code nil ("libtree-{sitter}-"
>>>                  (code (:class var) "\"language\"")
>>>                  "."
>>>                  (code (:class var) "ext")))
>> 
>> This is not much different from @name[nil]{<contents>} idea, but
>> more verbose.
>>
>  > Also, more importantly, I strongly dislike the need to wrap the text
>  > into "". You will have to escape \". And it will force third-party
>  > parsers to re-implement Elisp sexp reader.
>
> By this example I was trying to show how to express @var, @samp, @file 
> without introducing of new custom objects. I do not see any problem with 
> verbosity of such format, it may be used for really special cases only, 
> while some more convenient markup is used for more simple cases.

I was comparing the inline block vs. your AST proposal. If the AST idea
is complementary, I see no issue.

>>> If there was some syntax for object attributes then simple cases would
>>> be like
>>>
>>>       [[attr:(:class var)]]~language~
>> 
>> I do not like this idea. It will require non-trivial changes in Org
>> parser and fontification.
>> 
>> Using dedicated object properties or at least inheriting properties from
>> :parent is the style we employ more commonly across the code:
>> 
>> @var{language}
>> or
>> @code[:class var]{language}
>> or
>> @attr[:class var]{~language~}
>
> I do not mind to have some "span" object to assign attributes to its 
> direct children. I used link-like prefix object just because a proof of 
> concept may be tried with no changes in Org. It does not require support 
> of nested objects. There is no existing syntax for such "span" objects, 
> but perhaps it is not necessary and source blocks should be used instead 
> for special needs.

The problem is that instead of supporting nested objects we will have to
support "previous object changing the meaning of subsequent", which is
more fundamental change.

I envision the new inline blocks to allow assigning attributes to
children. These new inline blocks must not have issues with nesting by
design.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Dear List,
>
> I am forwarding an official email from RMS changing subject line to more
> descriptive. See below.
>
> For some context, in order to support specialized syntax for manuals, we
> may first need to implement the discussed special blocks export and
> inline special blocks:
> 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
> 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
> The above links aim to introduce export functionality that we now have
> for links to special blocks and new custom markup elements. I am
> referring to
> 1. Ability to create new custom element types programmatically 
> 2. Ability to define how to :export the custom element types
>
> Similar to `org-link-set-parameters'.
>
> Patches and more concrete ideas are welcome!
>
> From: Richard Stallman <rms@gnu.org>
> Subject: Re: Org mode and Emacs
> To: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 08:10:03 -0400 (4 days, 8 hours, 26 minutes ago)
> Flags: seen, list
> Maildir: /gmail/10 - Tech/software/org
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Speaking as the Chief GNUisance, rssponsible for GNU Project
> standards, I would be happy to adopt an upgraded Org format as a new
> standard source format for GNU manuals, _provided_ Org format has been
> extended with the capability to express all the constructions and
> distinctions that Texinfo can express, generate all the output formats
> Texinfo can generate, and use TeX to make beautiful printed output.
>
> Texinfo can generate these output formats: Info files, HTML, ASCII
> text, and DVI and PDF files via TeX.
>
> Texinfo provides numerous subtle distinctions that show up clearly in
> each of these output formats.  Compare, for example, @var, @dfn and
> @emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.
>
> I am sure people can extend Org software to handle these semantic
> distinctions and generate these output formats.  Since it has been
> done once, it can be done again.  But the work is not trivial.
>
> The work has to start by designing what the extended Org format will look
> like.  That part is the crucial part; once it has been specified,
> people can work independently to implement various parts of handling
> that format.
>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
> ----------

I realise this will likely come across as another post from "Debbie
Downer", but I feel it is important to add a warning here and not get
too carried away with the excitement of seeing org mode accepted to the
point it becomes the official documentation format for Emacs. There are
some potential pitfalls here which need to be considered and which could
impact on how we satisfy the remaining 'blocker' to org mode taking on
this important role. A few questions we might want to consider....

What impact will adding all the additional formatting/markup primitives
have to the user experience?

One of the big benefits org has is simplicity in markup. This is one of
the driving themes in the 'markdown' movement. Will adding a lot of
additional syntax and markup tags add to cognitive load and complexity
and losing some of what makes org mode great to use. This could be one
of those situations where less is more.

Will adding a lot of additional markup entities have any impact on the
development of new and maintenance of existing export back ends? i

With all the additional entities, I suspect the demand for nesting of
entities will also increase. This has been an area org has struggled
with in the past. I suspect the big issue is that allowing nesting of
markup entities and maintaining simple syntax is very difficult.

Is there a risk of one aspect of org mode dominating all others and
potentially transforming it from a very flexible and general solution to
a technical documentation focus?

Texifno has a very specific focus. It aims to be an advanced formatting
system for writing software technical documents. As such, it is very
suitable for Emacs documentation. Org mode on the other hand is not a
documentation framework. While it does a fine job in this area, it is
not its prime focus. Org has many other unrelated roles, such as task
management, time tracking, simple spreadsheet and data
management/manipulation, literate programming and live document
generation, data capture etc etc. Will adopting org mode as the default
documentation format for Emacs run the risk of the technical
documentation aspects of org mode taking precedence over other aspects?
Will funcionality in different areas have to be modified in order to
better support technical documentation?

What is the underlying motivation for this very significant change? 

A big question which I've not seen answered is what is the motivation
for this very significant change? Are there problems with texinfo which
are driving this change? If so, are these problems ones we need to be
very careful not to import into org mode. When you look at it, texinfo
already exports to many different formats similar to what org mode
does. It is a system with a very specific and quite narrow focus -
software technical documentation and yet, its use sems to be
declining. If it wasn't the flagship for GNU documentation, would it
even still exist? So the question becomes, why is this system not more
popular within software documentation circles? If part of the reason is
to potentialy increase contributors, will there still be as many people
wanting to use org mode if the underlying syntax is extended and
modified to support all the main texinfo markup set?

What impact on maintenance and future development directions will
becoming the official documentation framework have for org mode?

Will this result in document formatting gaining additional focus over
other areas?  Will it result in interface changes which favor
documentation processes over other areas like babel, data capture etc?

I think it should also be noted that there are some core Emacs
developers who are not supportive of this change and who don't like or
use org mode. Despite what RMS states, this is not a sure thing. Once
org mode is able to meet all the stated requirements, there debate
regarding switching to use org mode will begin and I suspect it will be
pretty full on. We will need to have a very clear picture regarding what
our (the org mode community) motivation is here and know what we are
prepared to compromise and what is non-negotiable.

If we do decide to go down this road, one idea which I feel would be
worth exploring is the extent to which we could have these additional
markup elements as an optional component. In some ways, similar to org
export back ends. If we could add these additional makrujp elements as
an optional add on, perhaps we can maintain the markup simplicity and
simple syntax for those who don't need specialised markup and for when
we do, we could require the 'technical documentation' module? 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> What impact will adding all the additional formatting/markup primitives
> have to the user experience?

From my perspective, the first and main goal related to this request is
developing the minimally (!) necessary Org format extensions to allow
Texinfo constructs.

I do not think that we need to add all the variety of Texinfo-specific
constructs to the Org specification. Instead, we should add generic
configurable syntax elements to Org. (The Texinfo-specific part will
come as a separate library, similar to ol-*.el)

In particular, I suggest to (1) extend the existing special block elements
with Elisp-configurable :export settings; (2) add special block
equivalent object.

The latter will also serve as an alternative to our current awkward
approach to escaping the inline markup (bold, italics, verbatim, etc).
Especially wrt requirements to the spaces around the markup symbols.

For user experience, the new markup primitives will be unnecessary,
except specific export requirements or edge cases with our current
markup (e.g. spaces at the beginning/end of verbatim/code).

> One of the big benefits org has is simplicity in markup. This is one of
> the driving themes in the 'markdown' movement. Will adding a lot of
> additional syntax and markup tags add to cognitive load and complexity
> and losing some of what makes org mode great to use. This could be one
> of those situations where less is more.

I agree that adding yet another markup tagging will be an extra
cognitive burden. But I envision it to be used only when necessary. If
our current markup is sufficient for the user needs, there will be no
need to go into the more verbose tagging. But if there are special needs
(like exporting manual), there is no way to maintain Org simplicity yet
allowing the complex @dfn/@code/whatnot distinctions.

Also, see https://yhetil.org/emacs-devel/87v8t3wfgd.fsf@localhost

> Will adding a lot of additional markup entities have any impact on the
> development of new and maintenance of existing export back ends? i

I only suggest adding a single new markup object entity, similar to
special block elements. Just like #+BEGIN_FOO can spawn infinite number
of special block types, the new markup object will allow arbitrary new
markup entities loaded as necessary by the corresponding export
backends/libraries.

> With all the additional entities, I suspect the demand for nesting of
> entities will also increase. This has been an area org has struggled
> with in the past. I suspect the big issue is that allowing nesting of
> markup entities and maintaining simple syntax is very difficult.

Agree. However, it is the demand for nested constructs being one of the
motivators why we should have a new markup object syntax that will allow
nesting when absolutely necessary and when the existing simpler Org
markup does not cut it.

> Is there a risk of one aspect of org mode dominating all others and
> potentially transforming it from a very flexible and general solution to
> a technical documentation focus?

I do not think so. Org functionality only depends on the community
interest and contributions. I see no reason why technical documentation
is going to prevail over other Org uses.

> Texifno has a very specific focus. It aims to be an advanced formatting
> system for writing software technical documents. As such, it is very
> suitable for Emacs documentation. Org mode on the other hand is not a
> documentation framework. While it does a fine job in this area, it is
> not its prime focus. Org has many other unrelated roles, such as task
> management, time tracking, simple spreadsheet and data
> management/manipulation, literate programming and live document
> generation, data capture etc etc. Will adopting org mode as the default
> documentation format for Emacs run the risk of the technical
> documentation aspects of org mode taking precedence over other aspects?
> Will funcionality in different areas have to be modified in order to
> better support technical documentation?

> What impact on maintenance and future development directions will
> becoming the official documentation framework have for org mode?
>
> Will this result in document formatting gaining additional focus over
> other areas?  Will it result in interface changes which favor
> documentation processes over other areas like babel, data capture etc?


I do not think so. We are certainly not going to remove features from
Org just because they will improve support for technical documentation.

I mean, I do not really see why this should be needed. But if it is
needed, then we will not go this road and simply abandon the
documentation idea.
https://bzg.fr/en/the-software-maintainers-pledge/

> What is the underlying motivation for this very significant change? 
>
> A big question which I've not seen answered is what is the motivation
> for this very significant change? Are there problems with texinfo which
> are driving this change? If so, are these problems ones we need to be
> very careful not to import into org mode. When you look at it, texinfo
> already exports to many different formats similar to what org mode
> does. It is a system with a very specific and quite narrow focus -
> software technical documentation and yet, its use sems to be
> declining. If it wasn't the flagship for GNU documentation, would it
> even still exist? So the question becomes, why is this system not more
> popular within software documentation circles? If part of the reason is
> to potentialy increase contributors, will there still be as many people
> wanting to use org mode if the underlying syntax is extended and
> modified to support all the main texinfo markup set?

This is a good question. I'd like to hear from RMS or other people
familiar with Texinfo issues.

As a starter, there was a bit of related discussion in emacs-devel:
- RMS mentioned that one big gotcha with Texinfo is that it inherits
  from plain TeX. Supporting LaTeX features would require a major
  re-design. Org supports LaTeX out of the box.
  https://yhetil.org/emacs-devel/E1o0WEF-0004wJ-Iw@fencepost.gnu.org
- Also, from Christopher Dimech:
  "Because texinfo is just plain tex, and as consequence remained
   limited (e.g. no colouring, no bold mathematical expressions' no
   boxes for formal end of proof).
   https://yhetil.org/emacs-devel/trinity-5129f2bc-bc9d-4f40-a099-ba1c913178fe-1655084866514@3c-app-mailcom-bs16

> I think it should also be noted that there are some core Emacs
> developers who are not supportive of this change and who don't like or
> use org mode. Despite what RMS states, this is not a sure thing. Once
> org mode is able to meet all the stated requirements, there debate
> regarding switching to use org mode will begin and I suspect it will be
> pretty full on. We will need to have a very clear picture regarding what
> our (the org mode community) motivation is here and know what we are
> prepared to compromise and what is non-negotiable.
>
> If we do decide to go down this road, one idea which I feel would be
> worth exploring is the extent to which we could have these additional
> markup elements as an optional component. In some ways, similar to org
> export back ends. If we could add these additional makrujp elements as
> an optional add on, perhaps we can maintain the markup simplicity and
> simple syntax for those who don't need specialised markup and for when
> we do, we could require the 'technical documentation' module? 

Agree. Let's not go too far yet and focus on extending special blocks
and the inline special block element I propose. These topics (especially
for markup element with less edge cases) pop up on the Org ML from time
to time and worth looking into regardless whether Org is going to be
used as technical documentation format.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Agree. Let's not go too far yet and focus on extending special blocks
> and the inline special block element I propose. These topics (especially
> for markup element with less edge cases) pop up on the Org ML from time
> to time and worth looking into regardless whether Org is going to be
> used as technical documentation format.

+1

I'm confident your across the concerns I raised and if I understand your
proposal correctly, adding support for the markup entities RMS
requested should be possible without any significant adverse impact on
existing usability. Greater clarity regarding the underlying drive for
changing from texinfo for Emacs would be good, but that is essentially
and emacs devel question and as you point out, ability to support the
additional elements identified by RMS would potentially address other
issues, such as spaces, quoting and nesting within existing markup.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Max Nikulin]

On 01/10/2022 11:08, Ihor Radchenko wrote:
> Tim Cross writes:
> 
> I do not think that we need to add all the variety of Texinfo-specific
> constructs to the Org specification. Instead, we should add generic
> configurable syntax elements to Org. (The Texinfo-specific part will
> come as a separate library, similar to ol-*.el)
> 
> In particular, I suggest to (1) extend the existing special block elements
> with Elisp-configurable :export settings; (2) add special block
> equivalent object.

While I do not mind to have flexible generic inline objects, I feel some 
doubts.

Export is already customizable through creation of derived backend. For 
links :export property is merely an alternative way supposed to be more 
convenient. In some sense it is a way to dispatch proper handler, a kind 
of virtual function table, etc. I see a couple of limitations with link 
export implementation.

1. Interface is rather different from the derived backend property. 
Instead of org-element object only selected properties (and backend 
communication channel is available).

2. Unsure if there is a robust way to extend implementation of the 
backend handler without replacing it completely. I mean a function that 
modifies or sets some attributes and delegate real export to the default 
handler.

Mentioned in this thread texinfo commands are not convincing reason for 
special blocks from my point of view. They are different flavors of 
verbatim (or code) object. If they are verbatim objects with some 
additional property then they may be just exported by a backend that is 
unaware of their kinds. It can be considered as graceful degradation. 
For special blocks export handlers must be implemented for each backend 
unless type hierarchy is someway declared.

Earlier we discussed assigning attributes for inline objects. While 
nesting is not involved, it may be a way to implement necessary texinfo 
commands as verbatim with class or type attribute. I am unsure if 
different types of special blocks is the best way to resolve nesting 
problem. Verbatim custom objects require different rules of parsing.

Actually I simplified things when wrote that a backend may be unaware of 
verbatim type. When nesting is involved it should be ready at least to 
nested verbatim object. E.g. markdown backend can not blindly wrap text 
into backticks, it has to check if parent element is not a verbatim 
snippet or a verbatim block.

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 01/10/2022 11:08, Ihor Radchenko wrote:
>> In particular, I suggest to (1) extend the existing special block elements
>> with Elisp-configurable :export settings; (2) add special block
>> equivalent object.
>
> While I do not mind to have flexible generic inline objects, I feel some 
> doubts.
>
> Export is already customizable through creation of derived backend. For 
> links :export property is merely an alternative way supposed to be more 
> convenient. In some sense it is a way to dispatch proper handler, a kind 
> of virtual function table, etc. I see a couple of limitations with link 
> export implementation.

Creating derived backend will force users to use that non-standard
backend for export - inconvenient. Especially for third-party backends.

:export property, among other things, can also provide a reasonable
fallback for arbitrary backend not considered in advance.

> 1. Interface is rather different from the derived backend property. 
> Instead of org-element object only selected properties (and backend 
> communication channel is available).

Is it? :export function for links is taking similar parameters with the
other export transcoders.

> 2. Unsure if there is a robust way to extend implementation of the 
> backend handler without replacing it completely. I mean a function that 
> modifies or sets some attributes and delegate real export to the default 
> handler.

We may provide something like :export-filter-object that will act
similar to `:filter-parse-tree' and replace the original link object
with arbitrary Org AST.

> Mentioned in this thread texinfo commands are not convincing reason for 
> special blocks from my point of view. They are different flavors of 
> verbatim (or code) object. If they are verbatim objects with some 
> additional property then they may be just exported by a backend that is 
> unaware of their kinds. It can be considered as graceful degradation. 
> For special blocks export handlers must be implemented for each backend 
> unless type hierarchy is someway declared.

No. There is no need to consider every possible backend. There could be
an export handler that will provide a fallback for unknown backend, if
needed.

> Earlier we discussed assigning attributes for inline objects. While 
> nesting is not involved, it may be a way to implement necessary texinfo 
> commands as verbatim with class or type attribute. I am unsure if 
> different types of special blocks is the best way to resolve nesting 
> problem. Verbatim custom objects require different rules of parsing.

Please do remember that texinfo commands, are _not_ verbatim. They can
contain other markup inside. I'd rather look at them as extended
emphasis. Their contents must be parsed as well.

> Actually I simplified things when wrote that a backend may be unaware of 
> verbatim type. When nesting is involved it should be ready at least to 
> nested verbatim object. E.g. markdown backend can not blindly wrap text 
> into backticks, it has to check if parent element is not a verbatim 
> snippet or a verbatim block.

Agree. See export filter idea.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For some context, in order to support specialized syntax for manuals, we
  > may first need to implement the discussed special blocks export and
  > inline special blocks:
  > 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
  > 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/

  > The above links aim to introduce export functionality that we now have
  > for links to special blocks and new custom markup elements. I am
  > referring to
  > 1. Ability to create new custom element types programmatically 
  > 2. Ability to define how to :export the custom element types

I'm not sure of the meaning of some of those words.  Does this mean
that people are implementing those distinctions in Org mode?  Or does
it mean someone has proposed to implement them in Org mode but not yet
started actually writing it?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > For some context, in order to support specialized syntax for manuals, we
>   > may first need to implement the discussed special blocks export and
>   > inline special blocks:
>   > 1. https://list.orgmode.org/orgmode/87y1yr9gbl.fsf@gmail.com/
>   > 2. https://list.orgmode.org/orgmode/87edzqv4ha.fsf@localhost/
>
>   > The above links aim to introduce export functionality that we now have
>   > for links to special blocks and new custom markup elements. I am
>   > referring to
>   > 1. Ability to create new custom element types programmatically 
>   > 2. Ability to define how to :export the custom element types
>
> I'm not sure of the meaning of some of those words.  Does this mean
> that people are implementing those distinctions in Org mode?  Or does
> it mean someone has proposed to implement them in Org mode but not yet
> started actually writing it?

The latter. We currently need a new syntax element that will allow all
the things available in Texinfo markup syntax.

We have decided that we do need such new syntax. We are discussing how
the new syntax will look like. We will implement the syntax in future.
Once implemented, the new syntax will open the road to add Texinfo
markup structures into Org export backends.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The latter. We currently need a new syntax element that will allow all
  > the things available in Texinfo markup syntax.

  > We have decided that we do need such new syntax. We are discussing how
  > the new syntax will look like. We will implement the syntax in future.
  > Once implemented, the new syntax will open the road to add Texinfo
  > markup structures into Org export backends.

I'm glad you are working on this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Robert Weiner]

I would just like to point out that anyone familiar with writing a Texinfo-format manual who wants to combine this with Org mode would likely just want to embed Org constructs, like Org tables in the manual; not to use Org as a formatter that exports individual source blocks to form a Texinfo manual (literate programming style).  This is because their focus while writing will be on the manual and its formatting, not on Org markup that exports to another markup.

I hope this is considered in your strategy.

-- rsw

> On Oct 8, 2022, at 6:38 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> The latter. We currently need a new syntax element that will allow all
>> the things available in Texinfo markup syntax.
> 
>> We have decided that we do need such new syntax. We are discussing how
>> the new syntax will look like. We will implement the syntax in future.
>> Once implemented, the new syntax will open the road to add Texinfo
>> markup structures into Org export backends.
> 
> I'm glad you are working on this.
> 
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
> 
> 
> 

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Ihor Radchenko]

Robert Weiner <rswgnu@gmail.com> writes:

> I would just like to point out that anyone familiar with writing a Texinfo-format manual who wants to combine this with Org mode would likely just want to embed Org constructs, like Org tables in the manual; not to use Org as a formatter that exports individual source blocks to form a Texinfo manual (literate programming style).  This is because their focus while writing will be on the manual and its formatting, not on Org markup that exports to another markup.

Could you please elaborate?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [HELP] Fwd: Org format as a new standard source format for GNU manuals

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I would just like to point out that anyone familiar with writing a
  > Texinfo-format manual who wants to combine this with Org mode
  > would likely just want to embed Org constructs, like Org tables in
  > the manual; not to use Org as a formatter that exports individual
  > source blocks to form a Texinfo manual (literate programming
  > style).

The proposal I thought I was responding to is different from both of
those.  It was to make Org format an option for the source format for
a whole GNU manual.  One that could generate the same output formats
that we now generate from Texinfo, and would be able to produce a
well-formatted printed manual via TeX.

This does not inherently require imply generating Texinfo format from
Org format.  Texinfo format is a source format, not an output format.

However, generating Texinfo format as an intermediate format might be
advantageous as a way to feed the Org-format manual through TeX to
get a well-formatted printed manual.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

Bastien <bzg@gnu.org> writes:

>> - Let's see if we have more contributions to the manual and if
>>   we really solved a problem here.
>
> (=> 2) This didn't happen.

I played around with the git log data and your claim appears to be
incorrect. The number of commits affecting manual increases
substantially according to my analysis. And it is not just initial spike
of fixes after introducing the new format.

See the attached histogram.


[-- Attachment #2: manual-commits-stats.png --]
[-- Type: image/png, Size: 27488 bytes --]

[-- Attachment #3: Type: text/plain, Size: 916 bytes --]


Methodology:

The org manual has been introduced around d330eed7c (2017-12)

#+begin_src bash
git log main --grep manual --format=%ci > manual-commits.dat
cat manual-commits.dat | awk '{print $1}' | cut -d- -f1,2 > manual-commits-month.dat 
#+end_src

#+begin_src gnuplot
set timefmt '%Y-%m'
set xdata time
set title "Number of commits according to \"git log --grep manual\"" font ",20"
set tics out
set label 1 ".texi → .org" at "2017-12",155 center font ",18" textcolor rgb'red'
set arrow from "2017-12",0 to "2017-12",150 nohead front lw 10 lc rgb'red'
set style fill solid 0.5 border
set yrange [0:160]
plot 'manual-commits-month.dat' u 1:(1.0) bins=40 w boxes t''
#+end_src

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@gmail.com> writes:

> I played around with the git log data and your claim appears to be
> incorrect. The number of commits affecting manual increases
> substantially according to my analysis. And it is not just initial spike
> of fixes after introducing the new format.
>
> See the attached histogram.
>
>
>
>
> Methodology:
>
> The org manual has been introduced around d330eed7c (2017-12)
>
> #+begin_src bash
> git log main --grep manual --format=%ci > manual-commits.dat
> cat manual-commits.dat | awk '{print $1}' | cut -d- -f1,2 > manual-commits-month.dat 
> #+end_src
>
> #+begin_src gnuplot
> set timefmt '%Y-%m'
> set xdata time
> set title "Number of commits according to \"git log --grep manual\"" font ",20"
> set tics out
> set label 1 ".texi → .org" at "2017-12",155 center font ",18" textcolor rgb'red'
> set arrow from "2017-12",0 to "2017-12",150 nohead front lw 10 lc rgb'red'
> set style fill solid 0.5 border
> set yrange [0:160]
> plot 'manual-commits-month.dat' u 1:(1.0) bins=40 w boxes t''
> #+end_src

Maybe that points to Org being more maintainence-heavy than Texinfo?

Anyway, I've made the following offer and will make it again: if anyone
who does not know Texinfo wants to contribute documentation to Emacs, he
can write it in plain text, and I will manually convert it to Texinfo.

Re: Org mode and Emacs

[Emanuel Berg]

Po Lu wrote:

> Anyway, I've made the following offer and will make it
> again: if anyone who does not know Texinfo wants to
> contribute documentation to Emacs, he can write it in plain
> text, and I will manually convert it to Texinfo.

There are tools for that ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Org mode and Emacs

[Bastien Guerry]

Po Lu <luangruo@yahoo.com> writes:

> Anyway, I've made the following offer and will make it again: if anyone
> who does not know Texinfo wants to contribute documentation to Emacs, he
> can write it in plain text, and I will manually convert it to Texinfo.

That's a very good move, one that we should make for Org too.

I'll propose something on our guide for contributors.

-- 
 Bastien

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1206 bytes --]

I think there are two (perhaps more) aspects to writing good
documentation and accepting / making contributions:

1. The basics -- write a paragraph or two describing some feature.
Orthogonal to that and here is where each format bites in different
ways:

2. Integrating that contribution into the main corpus of the work --- at
   the write place, with the right surrounding context, generating the
   right table-of-contents and menus (a running nightmare in Texinfo if
   you dont set things up right at the beginning) and so on.

I think (2) might likely contribute negatively to  (1) happening more
often.

So rather than arguing about formats (and it's been pointed out already
that there are tools for automatic conversion) we should perhaps:

A. Create a simple content repository where contributors can check-in
   their small to medium contributions in the format of their choice
   (limited to a few open formats) along with the meta-data 
B. A small group of more experienced volunteers take on the task of
   incorporating those contributions into the main work.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Po Lu <luangruo@yahoo.com>,  Ihor Radchenko <yantar92@gmail.com>,  Payas
>  Relekar <relekarpayas@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 06:58:31 -0700
> 
> I think there are two (perhaps more) aspects to writing good
> documentation and accepting / making contributions:
> 
> 1. The basics -- write a paragraph or two describing some feature.
> Orthogonal to that and here is where each format bites in different
> ways:
> 
> 2. Integrating that contribution into the main corpus of the work --- at
>    the write place, with the right surrounding context, generating the
>    right table-of-contents and menus (a running nightmare in Texinfo if
>    you dont set things up right at the beginning) and so on.
> 
> I think (2) might likely contribute negatively to  (1) happening more
> often.

I think the division you suggest is somewhat artificial.  Unlike man
pages, Info manuals are not a collection of loosely-couples articles;
instead, the sections in a good Info manual are always connected and
context-aware.  Thus, writing the documentation for the manual should
_always_ start by studying the "neighborhood" in which that new stuff
will live.  IOW, putting the new text in context is not an
after-thought, it should be considered prior to writing.

I don't know what you allude to as "running nightmare in Texinfo".
Adding a new node to a manual is since long ago a very simple process:
just add it and make sure the @menu of the parent node (if there is
one) is updated to have the new node in its correct place.  That's it.

> So rather than arguing about formats (and it's been pointed out already
> that there are tools for automatic conversion) we should perhaps:
> 
> A. Create a simple content repository where contributors can check-in
>    their small to medium contributions in the format of their choice
>    (limited to a few open formats) along with the meta-data 
> B. A small group of more experienced volunteers take on the task of
>    incorporating those contributions into the main work.

Most contributors to Emacs already write pretty good Texinfo; adding
finishing touches to that and fixing a few style or markup mistakes is
already being done in the background by those who spot those blunders.
So I think we already have the above in place.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Maybe that points to Org being more maintainence-heavy than Texinfo?

While I am not familiar with Texinfo, I can only recall a single issue
with Org manual that is related to it being written in Org. I would not
call it maintainence-heavy.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> I played around with the git log data and your claim appears to be
> incorrect. 

Thanks for providing these numbers, I stand corrected then.

But I'm not *that* impressed as I still doubt we can infer a causal
connection between the texi->org switch and the increasing number of
commits (I've read the detailed logs.)

If you, Nicolas, Tim and other core contributors say that switching
back to .texi will definitely slow down your contributions to Org
(these has to be considered and said in all honesty, which is hard)
that I would consider a strong argument against the switch back to
.texi---but that's our collective decision anyway, so I'm fine with
whatever everyone decides (not now, around Org 10).

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Bastien <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> I played around with the git log data and your claim appears to be
>> incorrect. 
>
> Thanks for providing these numbers, I stand corrected then.

Actually, the number are not as accurate as I thought.
The old manual was in doc/org.texi and had no word "manual" in it.

Grepping the file names yields (different colours represent .texi and .org)


[-- Attachment #2: manual-commits-v2.png --]
[-- Type: image/png, Size: 16592 bytes --]

[-- Attachment #3: Type: text/plain, Size: 117 bytes --]


which is still an improvement because the total number of commits
is just a fraction of the early Org development.


[-- Attachment #4: total-commits-stats.png --]
[-- Type: image/png, Size: 27303 bytes --]

[-- Attachment #5: Type: text/plain, Size: 1036 bytes --]


> But I'm not *that* impressed as I still doubt we can infer a causal
> connection between the texi->org switch and the increasing number of
> commits (I've read the detailed logs.)

I am sure that you are a lot more familiar with the historic data.

> If you, Nicolas, Tim and other core contributors say that switching
> back to .texi will definitely slow down your contributions to Org
> (these has to be considered and said in all honesty, which is hard)
> that I would consider a strong argument against the switch back to
> .texi---but that's our collective decision anyway, so I'm fine with
> whatever everyone decides (not now, around Org 10).

Learning .texi will certainly be an extra burden for me. Not impossible,
but feels awkward that I'd need it for Org.

In any case, emacs-devel is not the place to decide this.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Monday, June 13, 2022 at 12:39 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Tim Cross" <theophilusx@gmail.com>
> Cc: rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Tim Cross <theophilusx@gmail.com> writes:
>
> > There are many ways org mode can be improved. Current work to clarify
> > and refine the syntax, provide a solid and efficient parser, improved
> > font-locking and formatting and more consistent and rich API for both
> > extensions and new export backends are far more critical than adding
> > additional markup/markdown syntax to enable org mode to replace texinfo.
> > IMO such changes are not only misguided, they have the real potential to
> > adversely impact org by making things more complicated and harder to
> > maintain, destroying the advantage of a very small and easy to use
> > markup and moving the focus towards becoming an authoring tool rather
> > than a personal data organisation aid.
>
> Tim, I afraid that we a slipping to overgeneralisation again.
> We may or may not need to make changes to Org syntax for the purpose of
> writing documentation using Org.
>
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?

Does org accept latex structures?  Because texinfo is just plain tex, and as
consequence remained limited (e.g. no colouring, no bold mathematical expressions'
no boxes for formal end of proof).

> If we have such an example and know the expected outputs, someone can
> then create an equivalent Org file. Then, the discussion will hopefuly
> become more constructive.
>
> Best,
> Ihor
>
>

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Christopher Dimech <dimech@gmx.com> writes:

> Does org accept latex structures?  Because texinfo is just plain tex, and as
> consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> no boxes for formal end of proof).

Yes, latex is supported natively. You can mix it into Org text. For
non-latex export, it will be either converted to image of left as-is.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Monday, June 13, 2022 at 2:47 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: "Tim Cross" <theophilusx@gmail.com>, rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Christopher Dimech <dimech@gmx.com> writes:
>
> > Does org accept latex structures?  Because texinfo is just plain tex, and as
> > consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> > no boxes for formal end of proof).
>
> Yes, latex is supported natively. You can mix it into Org text. For
> non-latex export, it will be either converted to image of left as-is.
>
> Best,
> Ihor

That would be very good development, using org for Official Gnu Documentation.
Had discussed with Gavin Smith about latex a couple of years ago, who explained
that texinfo would have to be rewritten completely if we are to have a latex engine
for it.  Lot of work  and unlikely to happen.

Re: Org mode and Emacs

[Werner LEMBERG]

> Had discussed with Gavin Smith about latex a couple of years ago,
> who explained that texinfo would have to be rewritten completely if
> we are to have a latex engine for it.  Lot of work and unlikely to
> happen.

By the way, has someone informed the Texinfo maintainers that people
are whetting knives here on the list, discussing quite fundamental
changes about the future of the project?


    Werner

Re: Org mode and Emacs

[Eli Zaretskii]

On June 13, 2022 8:22:57 AM GMT+03:00, Werner LEMBERG <wl@gnu.org> wrote:
> 
> > Had discussed with Gavin Smith about latex a couple of years ago,
> > who explained that texinfo would have to be rewritten completely if
> > we are to have a latex engine for it.  Lot of work and unlikely to
> > happen.
> 
> By the way, has someone informed the Texinfo maintainers that people
> are whetting knives here on the list, discussing quite fundamental
> changes about the future of the project?

Please don't over-dramatize.  No one is "wetting any knives" here, we are just talking.  If you followed the discussion, supplanting Texinfo is nowhere in sight yet.  This is Emacs, not GCC.

And yes, this discussion should move to the Texinfo list, as has been suggested already.

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jun 04, 2022 at 18:09:45 +0200, Stefan Kangas wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > No, no, no, no!  Org mode is a highly complicated, obscure mode which
> > is NOT part of core Emacs, and mustn't become so.

> AFAICT, Org-mode has been distributed with Emacs for more than five
> major releases by now (approx. 15 years).

That's a strawman.  Org mode is not part of the Emacs core.  It's not
something in the part of Emacs inevitably used by any normal proficient
user.  Neither is Gnus.  Neither is CC Mode, for that matter.  By
contrast *Help*, Info, font-lock, the package manager, a lot of other
things, and a vast selection of commands bound to keys in the global key
map, are the core.

> > Surely the solution has got to be to encourage package authors to
> > write plain text (or .texi) documentation, by pointing out the
> > difficulties the non-standard .org format creates.

> In the Emacs community, Org-mode is ubiquitous.

Only in the sense that it's distributed with each Emacs release.

> I think we should actively encourage using it for most types of
> documentation, certainly for README type files.

I disagree entirely with you.  Org mode is highly complicated, obscure
(I've never managed to get a feel for what it does), and difficult to
learn (I've never managed it).  A text file is far easier to read for
those not familiar with org.  We're talking about READMEs here.  They're
typically 20 to 100 lines long.  A text file is ideal for these.

> Texinfo has its strengths of course, but it is unfortunately perceived
> as arcane and hard to use by many.

It's moderately easy to use, somewhat difficult to learn.

> And Org-mode can be converted to be read in 'M-x info'.

You mean it _has_ to be converted to be read?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Michael Albinus]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Stefan.

Hi,

>> I think we should actively encourage using it for most types of
>> documentation, certainly for README type files.
>
> I disagree entirely with you.  Org mode is highly complicated, obscure
> (I've never managed to get a feel for what it does), and difficult to
> learn (I've never managed it).  A text file is far easier to read for
> those not familiar with org.  We're talking about READMEs here.  They're
> typically 20 to 100 lines long.  A text file is ideal for these.

1+. README is a text file which is consulted in order to get information
about the related software. There's no rule to visit it by Emacs only.

Best regards, Michael.

Re: Convert README.org to plain text README while installing package

[Po Lu]

Michael Albinus <michael.albinus@gmx.de> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Hello, Stefan.
>
> Hi,
>
>>> I think we should actively encourage using it for most types of
>>> documentation, certainly for README type files.
>>
>> I disagree entirely with you.  Org mode is highly complicated, obscure
>> (I've never managed to get a feel for what it does), and difficult to
>> learn (I've never managed it).  A text file is far easier to read for
>> those not familiar with org.  We're talking about READMEs here.  They're
>> typically 20 to 100 lines long.  A text file is ideal for these.
>
> 1+. README is a text file which is consulted in order to get information
> about the related software. There's no rule to visit it by Emacs only.
>
> Best regards, Michael.

Hear, hear!

It also takes a relatively long time to load Org mode, so opening a
README.org-type file for the first time will always cause an annoying
pause.

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Po Lu <luangruo@yahoo.com> writes:

>>>> I think we should actively encourage using it for most types of
>>>> documentation, certainly for README type files.
>>>
>>> I disagree entirely with you.  Org mode is highly complicated,
>>> obscure (I've never managed to get a feel for what it does), and
>>> difficult to learn (I've never managed it).  A text file is far
>>> easier to read for those not familiar with org.  We're talking about
>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>> is ideal for these.
>>
>> 1+. README is a text file which is consulted in order to get
>> information about the related software. There's no rule to visit it
>> by Emacs only.

People nowadays write README.org and README.md files instead of
plain-text READMEs because the packages' repositories are hosted on some
forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
those formats in a nice way.  So the choice of format is natural for
package authors.  And they frequently contain markup like headings,
bullet lists, bold, italics, and code snippets which simply look much
better than their text-only alternative (even in their
editing/non-rendered versions).

> Hear, hear!
>
> It also takes a relatively long time to load Org mode, so opening a
> README.org-type file for the first time will always cause an annoying
> pause.

That's true but wouldn't it be possible to extract the font-lock/faces
related parts in a separate file with no dependencies which can be
loaded quickly enough?  I mean, the purpose would only be to display org
and markdown READMEs with the highlighting intended by the package
author but there's no point in enabling the gazillon of other features
(from editing to time-tracking to spreadsheet computations) those
packages might provide in addition.

Bye,
Tassilo

Re: Convert README.org to plain text README while installing package

[Michael Albinus]

Tassilo Horn <tsdh@gnu.org> writes:

Hi Tassilo,

>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).

Not everybody uses Github/Gitlab/<you name it> infrastructure
exclusively. There are dinosaurs like me, who simply inspect tarballs
and alike.

I have no problem if there are structured README.org or README.md files
in parallel. But a README file should be plain text.

> Bye,
> Tassilo

Best regards, Michael.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Michael Albinus <michael.albinus@gmx.de> writes:

>
> I have no problem if there are structured README.org or README.md files
> in parallel. But a README file should be plain text.
>

I've seen this mentioned multiple times in this thread and it doesn't
make sense to me. 

Org files *are* plain text. This is one of (perhaps the biggest) selling
points for org mode. They don't use any form of 'binary' data and can be
read just fine in any text editor or just using cat/less/more whatever.
They may look a little *ugly*, especially with respect to URLs, but are
still quite readable - a lot more readable than other plain text formats
such as xml or html or json etc.

I also find arguments based around org being complex and difficult to
learn to be somewhat overstated. Org is powerful and very configurable,
which means there can be a lot to learn if you want to leverage off all
it has to offer. However, like emacs, the basics are very simple and
easy to learn. 

While I'm not arguing that org should be forced upon everyone and I
would agree we need to keep potential load time issues in mind, there
seems to be lots in this thread over stating the issues and jumping to
extremes. All that seems to really be under consideration is to enable
rendering of *org files in help buffers using org font locking and
perhaps enabling folding, which could be very beneficial for large
readme files and would not matter for small ones. I also suspect this is
something which could be disabled with a simple variable setting for
those who really don't like it. 

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:

> Michael Albinus <michael.albinus@gmx.de> writes:


> > I have no problem if there are structured README.org or README.md
> > files in parallel. But a README file should be plain text.

I agree with this.

> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me. 

> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.

We're reduced to arguing about the meaning of "plain text".  The way I
see it, plain text means to be read as is by the user.  In that sense,
the formats you mention below, xml, html, etc. are _not_ plain text -
they're designed for machine processing.  A typical spam email in HTML
has the human readable pieces swamped by the machine readable pieces.  I
think you're arguing that this isn't the case with org mode files, though
Philip Kaludercic pointed out an org file where this wasn't the case.

> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

And their performance in standard programs like grep, wc, etc. is that
much worse than plain text.

> I also find arguments based around org being complex and difficult to
> learn to be somewhat overstated.

There are 784 key bindings in org mode.  How can you say that this isn't
complex and difficult to learn?

> Org is powerful and very configurable, ....

That contradicts your previous statement to some extent.  Any program
which is very configurable _needs_ to be configured.

> .... which means there can be a lot to learn if you want to leverage
> off all it has to offer.

I don't want to do this.  I want to avoid org mode being loaded into my
Emacs; it fouls up my key bindings to a significant extent.  Particularly
if I just want to read a 50 line README.

> However, like emacs, the basics are very simple and easy to learn. 

I don't agree that the basics of Emacs are simple and easy to learn at
all.  That's a strong reason why other editors have become popular.  Why
should I have to spend time learning a super-complicated mode just to
read a 50 line README?

> While I'm not arguing that org should be forced upon everyone ....

If a README file is README.org, then org mode is forced upon anybody
wishing to read it in Emacs.  This is why README files should be plain
text.

> .... and I would agree we need to keep potential load time issues in
> mind, there seems to be lots in this thread over stating the issues and
> jumping to extremes.

I think org mode is an extreme, with its 784 key bindings which seem to
violate written and unwritten conventions.

> All that seems to really be under consideration is to enable rendering
> of *org files in help buffers using org font locking and perhaps
> enabling folding, which could be very beneficial for large readme files
> and would not matter for small ones. I also suspect this is something
> which could be disabled with a simple variable setting for those who
> really don't like it. 

"It" could best be avoided by having plain text README files.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tim.
>
> On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>
>> > I have no problem if there are structured README.org or README.md
>> > files in parallel. But a README file should be plain text.
>
> I agree with this.
>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>
> We're reduced to arguing about the meaning of "plain text".  The way I
> see it, plain text means to be read as is by the user.  In that sense,
> the formats you mention below, xml, html, etc. are _not_ plain text -
> they're designed for machine processing.  A typical spam email in HTML
> has the human readable pieces swamped by the machine readable pieces.  I
> think you're arguing that this isn't the case with org mode files, though
> Philip Kaludercic pointed out an org file where this wasn't the case.
>

Philip's example is an extreme case and not representative of normal
README.org files. 

>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> And their performance in standard programs like grep, wc, etc. is that
> much worse than plain text.


No, I disagree with this completely. Org mode markup is very simple and
rarely has any impact with regards to grep, ag or any other 'shell' tool
It is precisely why I consider it to be a plain text format  - all of
these standard plain trext processing tools work in the main just as
well as they do on any ASCII file. Again, org mode is not like XML or
HTML or other formats which do make such processing difficult. 
>
>> I also find arguments based around org being complex and difficult to
>> learn to be somewhat overstated.
>
> There are 784 key bindings in org mode.  How can you say that this isn't
> complex and difficult to learn?
>
Very simply because the vast majority of  those key bindings only come
into play when yhou use advanced features of org mode, such as the
agenda or table editing or noweb mode. If your just using basic org mode
features, very very few of those key bindings even come into play. Just
go throgh that list and remove any bindings relating to agenda mode,
table editing mode, source block modes, clock table mode, list editing
mode, etc and you end up with very few key bindings (all of which are
udner the C-c prefix, unlike your previous claim). 

>> Org is powerful and very configurable, ....
>
> That contradicts your previous statement to some extent.  Any program
> which is very configurable _needs_ to be configured.
>

ABsolute nonsense. Just because you can configure somethig doesn't
automatically mean you have to configure it. Emacs is extremely
configurable, but you can use it jsut fine without doing any
configuration at all. 

>> .... which means there can be a lot to learn if you want to leverage
>> off all it has to offer.
>
> I don't want to do this.  I want to avoid org mode being loaded into my
> Emacs; it fouls up my key bindings to a significant extent.  Particularly
> if I just want to read a 50 line README.
>

Other posts have already explained this is NOT what is being proposed
and that it would not affect your key bindings in the slightest. All
that is being propsoed here is that a read only buffer will use org mode
to format/display the content. Very simple and not the monster you
insist. 


>> However, like emacs, the basics are very simple and easy to learn. 
>
> I don't agree that the basics of Emacs are simple and easy to learn at
> all.  That's a strong reason why other editors have become popular.  Why
> should I have to spend time learning a super-complicated mode just to
> read a 50 line README?
>

Well, basically, you don't. That has already bene explained and does not
need to be reitterated here. However, to be clear, all that is being
proposed is that org formatting is applied to the contgents of this
read-only buffer. There will not be huge numbers of
unwanted/unexptrected key bindings and there won't be a huge number of
other changes. You will likely have some additional key bindings which
may make navigating larger README files easier and that is about it. All
that is really being proposed here is org font locking and outline mode
navigation. All very simple really. 

>> While I'm not arguing that org should be forced upon everyone ....
>
> If a README file is README.org, then org mode is forced upon anybody
> wishing to read it in Emacs.  This is why README files should be plain
> text.
>
>> .... and I would agree we need to keep potential load time issues in
>> mind, there seems to be lots in this thread over stating the issues and
>> jumping to extremes.
>
> I think org mode is an extreme, with its 784 key bindings which seem to
> violate written and unwritten conventions.


That 750 key bindings is an extreme over statement and not what is being
proposed. Yet again, people jumping to extremes based on ignorance and
paranoia. Spend the time to go through the key bindings listed in the
org manual and you will soon realise that the majority of those bindings
only come into effect in specialised mdoes - none of which are relevant
to a READM.org ile (for example, all the key bindings relating to agenda
mode). 

I find it odd that someone can on one hand argue so hard against using a
mode based on the complexity and vast nubmer of key bindings and at the
same time admit they have never spent the time to learn the mode. This
seems like an arguument based on fear and uncertainty about something
you don't know rather thaa one based on fact. Perhaps look more closely
at what was actually being proposed (which was not a full org mode
situation) and spend enough time to at least udnerstand the basics
before condemnation. 
>
>> All that seems to really be under consideration is to enable rendering
>> of *org files in help buffers using org font locking and perhaps
>> enabling folding, which could be very beneficial for large readme files
>> and would not matter for small ones. I also suspect this is something
>> which could be disabled with a simple variable setting for those who
>> really don't like it. 
>
> "It" could best be avoided by having plain text README files.

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Jun 2022 23:57:55 +1000
> 
> > There are 784 key bindings in org mode.  How can you say that this isn't
> > complex and difficult to learn?
> >
> Very simply because the vast majority of  those key bindings only come
> into play when yhou use advanced features of org mode, such as the
> agenda or table editing or noweb mode. If your just using basic org mode
> features, very very few of those key bindings even come into play. Just
> go throgh that list and remove any bindings relating to agenda mode,
> table editing mode, source block modes, clock table mode, list editing
> mode, etc and you end up with very few key bindings (all of which are
> udner the C-c prefix, unlike your previous claim). 

How about if you do this exercise and walk us through the results?
Visit an Org file, type "C-h b", then show us the list, and tell which
bindings you think are "basic" and which are "advanced", and why you
think so.

I can tell you what I see with my very simple 1570-line Org file,
which just has some todo items organized in 4-level hierarchy:

  . ~100 keys that look "general Org" to me
  . ~30 keys that Org remaps to its s own commands, like 'yank' and
    'backward-sentence'
  . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
    bound by default, but they are there
  . ~60 more keys that I'm not sure whether they are "basic" or not,
    but they are all bound by default, just because I visited an Org
    file 

So "just" 230 key bindings.  And this is without any minor/add-on Org
mode/feature enabled, at least according to "C-h m".

Do you see something different?  Are you still saying that it's not a
lot, or that it's "based on ignorance and paranoia"?  If so, please
point out where I'm ignorant and/or paranoiac, because I'd really like
to know.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 06 Jun 2022 23:57:55 +1000
>> 
>> > There are 784 key bindings in org mode.  How can you say that this isn't
>> > complex and difficult to learn?
>> >
>> Very simply because the vast majority of  those key bindings only come
>> into play when yhou use advanced features of org mode, such as the
>> agenda or table editing or noweb mode. If your just using basic org mode
>> features, very very few of those key bindings even come into play. Just
>> go throgh that list and remove any bindings relating to agenda mode,
>> table editing mode, source block modes, clock table mode, list editing
>> mode, etc and you end up with very few key bindings (all of which are
>> udner the C-c prefix, unlike your previous claim). 
>
> How about if you do this exercise and walk us through the results?
> Visit an Org file, type "C-h b", then show us the list, and tell which
> bindings you think are "basic" and which are "advanced", and why you
> think so.
>
> I can tell you what I see with my very simple 1570-line Org file,
> which just has some todo items organized in 4-level hierarchy:
>
>   . ~100 keys that look "general Org" to me
>   . ~30 keys that Org remaps to its s own commands, like 'yank' and
>     'backward-sentence'
>   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
>     bound by default, but they are there
>   . ~60 more keys that I'm not sure whether they are "basic" or not,
>     but they are all bound by default, just because I visited an Org
>     file 
>
> So "just" 230 key bindings.  And this is without any minor/add-on Org
> mode/feature enabled, at least according to "C-h m".
>
> Do you see something different?  Are you still saying that it's not a
> lot, or that it's "based on ignorance and paranoia"?  If so, please
> point out where I'm ignorant and/or paranoiac, because I'd really like
> to know.

I think Alan's response has pretty much confirmed what I was saying. Alan has
made it fairly clear that his real underlying concern is about a stealthy
strategy to get org mode more intrinsically tied into Emacs and in the end,
making it so critical to normal operation that everyone will be forced to learn
org mode regardless. I don't necessarily agree with such an argument, but think
having a debate around that would be more up-front and in some ways honest
compared to what appear to be somewhat contrived alternative arguments about
excessive key bindings and added complexity. My characterisation of this all
being fear and paranoia may have been overstating things, but then again I'm not
sure. There certainly does seem to be considerable fear involved. I don't know
about paranoia, but I have not seen any evil plan to have org mode assimilate
Emacs as if it was the Borg and I've seen no discussions on the org list about
ways to get org more ingrained in Emacs or make Emacs more dependent on org.

With respect to ignorance, Alan made it quite clear he has not spent
the time or effort to learn org mode, so he is ignorant about its
operation and lacks any real understanding of its operation. He has made
assertions about the number of key bindings which are wildly over
inflated and about required levels of complexity which simply would not
be necessary with the org mode rendering of a read only buffer. In
short, he has made a lot of assumptions about both the key bindings and
complexity based on only a fairly cursory scan of the manual and very
limited experience. 

All that appears to have occurred here is that someone noted that packages often
had a readme.org file and wondered if this should be translated into just a
plain readme file. Someone else suggested we could have package descriptions
render the readme.org file using org and not require export into plain readme
file and mentioned some possible advantages this could have wrt navigation,
rendering of tables, links and images or inclusion of source code snippets,
examples etc.

From this point, some somewhat extreme claims and arguments have been thrown in.
One was Alan's claim that org would introduce 785 new key bindings, many of
which not bounmd to C-c and which would presumably either pollute the key space
or conflict with existing bindings. All I have done is point out this claim was
inaccurate and overstated due to an overly simplistic analysis of what is in the
key index of the org manual. 

it should be noted that if we were to add 'native' rendering of readme.org
files, this would likely be done with an org minor mode rather than a full blown
org mode as very little of the overall org functionality would be required in
this situation. All that would be required is org rendering support and org
support for navigation (folding/unfolding, following links and possibly
list/table navigation). All the advanced org functionality relating to babel,
noweb, todo and time management, agendas, etc have no relevance in this
scenario. LIkewise, if this functionality was provided via a minor mode, it
would also be possible for people to disable that mode and be left with just the
readme.org file, which is just plain text and quite readable.

Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Running Emacs wiht -q and opeing up an org file, I find the following 

- 236 org related key bindings - far less than 785 Of these, a number are
- actually remapping of existing bindings, so not new ones. This leaves 208.

I said that for a read only buffer, many of the org key bindings are not
relevant as they relate to features which are not pertinent to a read only org
buffer. Any bindings relating to babel, todo management, time management,
agendas etc have no relevance when reading a readme.org file. Really, the only
key bindings of any relevance are navigation related bindings (including
opening/closing folds, following links, etc). Removing all unnecessary org
bindings leave us with just 77 new bindings. Of these remaining bindings, quite
a few could also be removed, but I've left them in as I only wanted to remove
those which would clearly be of no benefit in a read only buffer rendering of a
readme.org file. In an org minor mode setup for rendering of readme.org (and
possibly other read-only rendering of org files), I estimate at least another 30
bindings could be removed because they have no real benefit or are of only
marginal benefit. Creating an org minor mode for this purpose which only
introduced 20 or so new bindings would easily be possible. The key point is that
claims of 785 new bindings arfe incorrect and misleading. Even being very
conservative, we are really talking about 10% of the number being incorrectly claimed.  
if we introduced a minor mode for this, we are talking likely less than 20 or so.

Arguments regarding this being a 'slippery slope' or 'edge of the wedge' in an
org takeover are a completely different argument and there may well be some
concerns which may need to be considered. However, if that is the real concern,
then argue based on that rather than exaggerated claims about excessive key
bindings, key binding polution/conflicts and excessive complexity.

In a similar manner, lets not jump to conclusions or throw in wild speculation
about things which have not been proposed. For example, nobody has suggested
everyone would be required to provide readme.org files, nobody has suggested you
could not just have a readme or a readme.txt file, nobody has suggested making
anyone provide any specific format. All that has been rasied is whether, when a
package has a readme.org file, should it be translated into a 'plain text'
(ASCII/UTF-8) format or whether it might be useful to have the file formatted
and nicely displayed using existing org-mode formatting capabilities. It could
even be that the ability to have 'native' rendering of readme.org files could be
n optional feature which those who want it can enable.

All of this now seems to be irrelevant as Stefan has indicated he will
just automatically convert the readme.org to a plain readme. I hope that
the original readme.org file will still be included for those who do
prefer to use org to render and navigate a well formatted readme file
which includes working and concise links, source code syntax
highlighting, well formatted examples and fast navigation. 

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Tue, Jun 07, 2022 at 16:14:52 +1000, Tim Cross wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

> >> From: Tim Cross <theophilusx@gmail.com>
> >> Cc: emacs-devel@gnu.org
> >> Date: Mon, 06 Jun 2022 23:57:55 +1000

> >> > There are 784 key bindings in org mode.  How can you say that this isn't
> >> > complex and difficult to learn?

> >> Very simply because the vast majority of  those key bindings only come
> >> into play when you use advanced features of org mode, such as the
> >> agenda or table editing or noweb mode. If your just using basic org mode
> >> features, very very few of those key bindings even come into play. Just
> >> go throgh that list and remove any bindings relating to agenda mode,
> >> table editing mode, source block modes, clock table mode, list editing
> >> mode, etc and you end up with very few key bindings (all of which are
> >> udner the C-c prefix, unlike your previous claim). 

> > How about if you do this exercise and walk us through the results?
> > Visit an Org file, type "C-h b", then show us the list, and tell which
> > bindings you think are "basic" and which are "advanced", and why you
> > think so.

> > I can tell you what I see with my very simple 1570-line Org file,
> > which just has some todo items organized in 4-level hierarchy:

> >   . ~100 keys that look "general Org" to me
> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
> >     'backward-sentence'
> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
> >     bound by default, but they are there
> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
> >     but they are all bound by default, just because I visited an Org
> >     file 

> > So "just" 230 key bindings.  And this is without any minor/add-on Org
> > mode/feature enabled, at least according to "C-h m".

> > Do you see something different?  Are you still saying that it's not a
> > lot, or that it's "based on ignorance and paranoia"?  If so, please
> > point out where I'm ignorant and/or paranoiac, because I'd really like
> > to know.

> I think Alan's response has pretty much confirmed what I was saying. Alan has
> made it fairly clear that his real underlying concern is about a stealthy
> strategy to get org mode more intrinsically tied into Emacs and in the end,
> making it so critical to normal operation that everyone will be forced to learn
> org mode regardless. I don't necessarily agree with such an argument, but think
> having a debate around that would be more up-front and in some ways honest
> compared to what appear to be somewhat contrived alternative arguments about
> excessive key bindings and added complexity.

You've misread the situation.  My objections to org mode are almost
entirely due to its excessive complexity as measured by the number of key
bindings.

[ .... ]

> With respect to ignorance, Alan made it quite clear he has not spent
> the time or effort to learn org mode, so he is ignorant about its
> operation and lacks any real understanding of its operation.

This is indeed the case.

> He has made assertions about the number of key bindings which are
> wildly over inflated .....

You have said this several times in your post without any justification.
There are 784 entries in the key index in org.info.  You're saying it's
"wildly over inflated" to use this number, at least as a first estimate.
Why are you making such a counter intuitive claim without any
justification? 

Eli counted around 230 bindings from C-h m.  It thus seems likely that on
average, an org mode key sequence is used for 3½ distinct functions.
This is not a good thing.

> .... and about required levels of complexity which simply would not be
> necessary with the org mode rendering of a read only buffer.

You're not saying that loading a RO readme.org would somehow only load a
small part of org mode, surely?

> In short, he has made a lot of assumptions about both the key bindings
> and complexity based on only a fairly cursory scan of the manual and
> very limited experience. 

And now you're trying to be patronising, but failing.  ;-)

Why do you make no attempt to explain why my assumptions are invalid?

> All that appears to have occurred here is that someone noted that
> packages often had a readme.org file and wondered if this should be
> translated into just a plain readme file. Someone else suggested we
> could have package descriptions render the readme.org file using org
> and not require export into plain readme file and mentioned some
> possible advantages this could have wrt navigation, rendering of
> tables, links and images or inclusion of source code snippets, examples
> etc.

> From this point, some somewhat extreme claims and arguments have been
> thrown in.  One was Alan's claim that org would introduce 785 new key
> bindings, many of which not bounmd to C-c and which would presumably
> either pollute the key space or conflict with existing bindings. All I
> have done is point out this claim was inaccurate and overstated due to
> an overly simplistic analysis of what is in the key index of the org
> manual.

You have made an unfounded assertion of my inaccuracy.  Are you
suggesting, perhaps, that of these 784 entries in org.info's key index,
the vast bulk are dummies?  Some of these bindings (I don't think I used
the word "many", but may have done) are not in the standard major mode
space C-c C-<letter>.  Some of these clash with existing bindings,
examples of which I gave last night.

> it should be noted that if we were to add 'native' rendering of readme.org
> files, this would likely be done with an org minor mode rather than a full blown
> org mode as very little of the overall org functionality would be required in
> this situation.

In other words, an attempt would be made to fix the problems which I am
drawing to people's attention.

> All that would be required is org rendering support and org support for
> navigation (folding/unfolding, following links and possibly list/table
> navigation). All the advanced org functionality relating to babel,
> noweb, todo and time management, agendas, etc have no relevance in this
> scenario. LIkewise, if this functionality was provided via a minor
> mode, it would also be possible for people to disable that mode and be
> left with just the readme.org file, which is just plain text and quite
> readable.

> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Actually it was 784, 28².  Again, why is this number invalid?

> Running Emacs wiht -q and opeing up an org file, I find the following 

> - 236 org related key bindings - far less than 785 Of these, a number are
> - actually remapping of existing bindings, so not new ones. This leaves 208.

> I said that for a read only buffer, many of the org key bindings are not
> relevant as they relate to features which are not pertinent to a read only org
> buffer. Any bindings relating to babel, todo management, time management,
> agendas etc have no relevance when reading a readme.org file. Really, the only
> key bindings of any relevance are navigation related bindings (including
> opening/closing folds, following links, etc). Removing all unnecessary org
> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
> a few could also be removed, but I've left them in as I only wanted to remove
> those which would clearly be of no benefit in a read only buffer rendering of a
> readme.org file. In an org minor mode setup for rendering of readme.org (and
> possibly other read-only rendering of org files), I estimate at least another 30
> bindings could be removed because they have no real benefit or are of only
> marginal benefit. Creating an org minor mode for this purpose which only
> introduced 20 or so new bindings would easily be possible.

You're describing what might be rather than what is.

> The key point is that claims of 785 new bindings are incorrect and
> misleading.

I resent you continally repeating this unfounded assertion.  There are
784 entries in org.info's key index.  That is the current state.  That is
the truth.  How can that be "incorrect"?  Even if 750 of the 784 bindings
somehow "don't come into play", they still exist, and are still
problematic.  Are you saying that only every tenth entry actually
signifies an actual binding, or something like that?

> Even being very conservative, we are really talking about 10% of the
> number being incorrectly claimed.  if we introduced a minor mode for
> this, we are talking likely less than 20 or so.

> Arguments regarding this being a 'slippery slope' or 'edge of the
> wedge' in an org takeover are a completely different argument and there
> may well be some concerns which may need to be considered. However, if
> that is the real concern, then argue based on that rather than
> exaggerated claims about excessive key bindings, key binding
> polution/conflicts and excessive complexity.

If it weren't for org mode's excessive key bindings and complexity, the
"takeover scenario" would scarcely be a worry.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Stefan Monnier]

Rather than try and figure out who's right or wrong, I hope we can take
advantage of this discussion to get some code out that try to solve
the problem.

AFAICT the problem seen from Emacs, is that Org is large (even for
a basic uses, which occasionally leads to high load times) and that it
doesn't follow all the usual Emacs conventions, such as
overriding/remapping standard key-bindings (the resulting behaviors may
sometimes be similar to the standard one, but even when that's the case
the redefinition can easily bite the user).

The problem seen from Org is that Emacs doesn't use Org enough :-)
[ This paragraph's shorter length probably reflects the fact that I'm
  less familiar with Org than with Emacs.  ]

I think the way forward is to define a "basic org-mode".  This one could
be used at many more places where there's currently an occasional desire
to use Org that's resisted because of the above problems.
Ideally, org-mode would then be defined as an extension of this "basic
org-mode".  Also ideally some of those extensions would be reworked so
they can be used "Emacs-wide" rather than only in Org (obviously, that
can only make sense for some of them).

We could start this "basic org-mode" as a trivial copycat of
`outline-mode` and then start adding Org features to it.  The driving
constraint is to keep it lightweight and rules-abiding enough that there
won't be any resistance to using it, while at the same time making sure
that the features added to it can be removed from the "org-mode
extension", so that org-mode and "basic org-mode" don't diverge.


        Stefan

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.

I think that people that use Org mode wouldn't be very interested in
that, and people that don't use Org can just continue editing those
files with `M-x find-file-literally', really.  Doing so is fine.

But I don't understand the discussion this thread warped out of.  When
we display a README from Package, we shouldn't be showing the raw text
of README.org or README.md or README.html, lightly fontified -- we
should be showing a rendering of it.  People that read the README aren't
supposed to edit it, after all.

Which would make the 40K key bindings that Org apparently has completely
irrelevant for non-Org users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs

[Stefan Monnier]

> I think that people that use Org mode wouldn't be very interested in
> that, and people that don't use Org can just continue editing those
> files with `M-x find-file-literally', really.  Doing so is fine.

I was thinking of files like etc/NEWS where having "a bit more than
outline-mode" would be welcome.

> But I don't understand the discussion this thread warped out of.  When
> we display a README from Package, we shouldn't be showing the raw text
> of README.org or README.md

Why not?  If it can be prettified on the fly by font-lock?

Separating "viewing" from "editing" is un-Emacsish, because it puts
a barrier between the "consumers" and the "producers" whereas Emacs
likes to try and get end-users exposed to the source as mush as possible
so it takes little effort for them to change role.


        Stefan

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 14:48:31 -0400
> 
> > But I don't understand the discussion this thread warped out of.  When
> > we display a README from Package, we shouldn't be showing the raw text
> > of README.org or README.md
> 
> Why not?

Because it's butt-ugly!

Re: Org mode and Emacs

[Stefan Monnier]

>> > But I don't understand the discussion this thread warped out of.  When
>> > we display a README from Package, we shouldn't be showing the raw text
>> > of README.org or README.md
>> Why not?
> Because it's butt-ugly!

I don't think it has to be.


        Stefan

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But I don't understand the discussion this thread warped out of.  When
>> we display a README from Package, we shouldn't be showing the raw text
>> of README.org or README.md
>
> Why not?  If it can be prettified on the fly by font-lock?
>
> Separating "viewing" from "editing" is un-Emacsish, because it puts
> a barrier between the "consumers" and the "producers" whereas Emacs
> likes to try and get end-users exposed to the source as mush as possible
> so it takes little effort for them to change role.

That's true, but on the other hand, we don't show texinfo files to
people for a good reason.  That's about as "plain text" as normal Org
files, or as some Markdown files get.  (Most Markdown files are kinda
pleasant, but there's plenty of scope for unreadability -- it drops down
into HTML for tables, for instance.)

Or take doc strings as an example -- it's always been a markup language,
too (\\ and \\[] and \\<>), but it's moving more clearly in that
direction -- because we want to make the documentation we present the
user pretty and easy to read.  And we have one-key commands to take us
to the source code, which we could have in the Package README rendering,
too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Rather than try and figure out who's right or wrong, I hope we can take
> advantage of this discussion to get some code out that try to solve
> the problem.
>
> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).
>
> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.
>
>

Thanks Stefan and I agree. I also think this is in general exactly the
direction current org maintainers/developers are taking. 

At this point, considerable work is being done (by Ihor and others) to 

- Implement a concise and efficient parser. This has also involved
  refining and clarifying org syntax, which has had some holes and
  ambiguities. 

- Replace org's largely regexp based font locking mechanism with one
  based on the parsing and tagging made possible by the org parser. This
  should improve both performance and accuracy in parsing. 

- Improving efficiency of folding etc

- Improving efficiency with respect to larger org files through the use
  of caching etc. 

- Increasing org's use of built-in Emacs capabilities rather than using
  org specific implementations. For example, adopting transient instead
  of an org implemented module which replicates similar functionality. 

- Increase modularity an enable loading of the functionality that you
  want. 

In particular, the important work being done by Ihor wrt folding,
parsing and caching will provide a core of functionality which can be
used to define something like an org minor mode which could be used in
those situations where we want some basic org functionality, like
folding, navigation and formatting/presentation, but we don't need all
the additional advanced features, such as babel, todo and time
management, spreadsheets, tables and table formulas etc. 

Of course, to what extent Emacs will/should leverage off org mode is a
completely different debate. 

Re: Org mode and Emacs

[Visuwesh]

[புதன் ஜூன் 08, 2022] Tim Cross wrote:

> [...]
> - Increasing org's use of built-in Emacs capabilities rather than using
>   org specific implementations. For example, adopting transient instead
>   of an org implemented module which replicates similar functionality. 

Where can I read more about this?  I see it being mentioned a few times
in the org-mode mailing list, and in the matrix room.  Is the plan to
completely remove the org-mks interface and replace it with transient
without having an option to use the former?  I find org-mks perfectly
fine to use and would be sad if it was replaced with transient.
Transient needs time to get used to, and the default settings is quite
un-Emacsy; I'm not too excited about configuring yet another package
that has a hard-to-understand manual TBH.

Re: Org mode and Emacs

[Tim Cross]

Visuwesh <visuweshm@gmail.com> writes:

> [புதன் ஜூன் 08, 2022] Tim Cross wrote:
>
>> [...]
>> - Increasing org's use of built-in Emacs capabilities rather than using
>>   org specific implementations. For example, adopting transient instead
>>   of an org implemented module which replicates similar functionality. 
>
> Where can I read more about this?  I see it being mentioned a few times
> in the org-mode mailing list, and in the matrix room.  Is the plan to
> completely remove the org-mks interface and replace it with transient
> without having an option to use the former?  I find org-mks perfectly
> fine to use and would be sad if it was replaced with transient.
> Transient needs time to get used to, and the default settings is quite
> un-Emacsy; I'm not too excited about configuring yet another package
> that has a hard-to-understand manual TBH.


The specifics of what is planned are still being worked out. Initially,
the likely initial candidate for change will be to the export menu.
Unless you have extensive low level customisation, it should not be a
change which has significant impact on users. In fact, maintaining
backwards compatibility and consistency for end users is important to
the org developers. It is also quite possible that after an initial
investigation, it may be decided transient is not a good option for org
mode or perhaps it will be a good option once additonal functionality or
enhancements are added. Right now, all that has been agreed is that it
would be worthwhile looking at it to see if it can be of benefit in
helping to reduce org maintenance overheads and/or increase org's
consistency with other emacs packages. 

What will determine what remains and what choices are available will
depend on what involvement people have in the development and what will
be maintainable. There are a number of areas in org mode where
functionality has been implemented that is 80+% equivalent to
functionality which exists or has been added to core emacs. Having this
duplication of effort is adding to the burden of maintenance, which is
already significant. In general, org maintainers keep an eye on what is
being added/expanded in Emacs core and when things are added, like
transient mode, they are assessed to see if adopting that functionality
would reduce the org maintenance load and improve org's consistency
withi the rest of Emacs. At this stage, transient has been added as
something to look at in the backlog. When this task makes if off the
backlog, the typical process would be for it to be discussed on the org
devel list, for initial implementations to be done either in its own
branch (if considered a significantly large enough change) or on the
development branch otherwise and people will be asked to try it out. 

So, if your interested in this area, the first thing would be to get on
the org devel mail list. Anyone who is particularly keen to see such
things added might also initiate discussions and development in their
own branch, which could then be added in as a PR. However, like Emacs,
any significant development work on org mode also requires FSF copyright
assignment. 

At this specific point in time, the main focus with org has been on
stability, performance improvements, especially for large org files with
many babel blocks and sections and improvements to the org syntax and provision of an
org parser,  

In particular, I think the work being done by Ihor on folding and
the org parser will have huge benefits. In addition to making the code
base easier to maintain and improving performance, it will help reduce
org's current dependency on complicated and difficult to maintain
regular expressions in the font-locking layer. This should improve
performance and reduce errors or unexpected results that sometimes occur
because of regexp errors etc and reduce time spent by maintainers in
tracking down such errors and maintenance of those complex regexps.   

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we are going to redesign Org mode syntax,
can we please add room for extensibility
so that an extension of Org mode syntax
could have all the features of Texinfo?

If the new syntax can express all the distinctions that Texinfo
can already express, that would make it possible to switch to it
for our documentation sources.

This needs to include conversion into some variant of TeX syntax so
that we can generate high-quality printed manuals through TeX.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?
>
> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.


Nobody has said that org mode syntax was getting redesigned. What I said
was that we were working to clarify the existing syntax and remove
ambiguities and provide a standard parser. 

Making org mode syntax equivalent to texinfo syntax seems like a
mistake to me. The original idea was to have a light weight syntax which
is easyh to learn, not create a clone of texinfo. Besides, I suspect it
would be very difficult to maintain backwards compatibility. 

I"m also not convinced that org mode would be the right solution for
Emacs documentation. That is a completely different conversation and I'm
not convinced that org mode is a better answer wrt documentation than
texinfo. While there are certainly some areas where texinfo is showing
its age, it still works well for providing Emacs documentation within
the editor. There may be some role for org mode to augment what is
provided by tgexinfo, but again, that is a whole different conversation.
Note also that the previous thread was about handling of existing org
files within Emacs package ecosystem, not about replacing org
documentation with org mode. 

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 09:10:50 +1000
> 
> I"m also not convinced that org mode would be the right solution for
> Emacs documentation.

Then why Org's own manual is written in Org instead of Texinfo?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I"m also not convinced that org mode would be the right solution for
>> Emacs documentation.
>
> Then why Org's own manual is written in Org instead of Texinfo?

I do not know if Org mode would be a good solution for Emacs
documentation. However, compared to texinfo, our manual is lacking index
entries in all the exported versions but texinfo.

The indexes in pdf version of the manual are actually generated by
org -> texinfo -> tex -> pdf.

AFAIK.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 14:20:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I"m also not convinced that org mode would be the right solution for
> >> Emacs documentation.
> >
> > Then why Org's own manual is written in Org instead of Texinfo?
> 
> I do not know if Org mode would be a good solution for Emacs
> documentation. However, compared to texinfo, our manual is lacking index
> entries in all the exported versions but texinfo.

Lack of an index makes for a less useful manual, IME.

Anyway, is the lack of the index the _only_ missing feature from what
the Org markup provides as compared to Texinfo?  Tim's message sounded
like there are many more omissions.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Fri, 10 Jun 2022 14:20:37 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I"m also not convinced that org mode would be the right solution for
>> >> Emacs documentation.
>> >
>> > Then why Org's own manual is written in Org instead of Texinfo?
>> 
>> I do not know if Org mode would be a good solution for Emacs
>> documentation. However, compared to texinfo, our manual is lacking index
>> entries in all the exported versions but texinfo.
>
> Lack of an index makes for a less useful manual, IME.
>
> Anyway, is the lack of the index the _only_ missing feature from what
> the Org markup provides as compared to Texinfo?  Tim's message sounded
> like there are many more omissions.

My statement is simply waht it says. I am not convinced org mode would
be the right solution for Emacs documentation. This is for many reasons,
some relating to org, some relating to the fact I'm not convinced there
is a need to replace texinfo, especially given that in some areas, I
think texinfo is more feature rich and consistent and some relating to the feeling I have
that the result is hard to justify given the amount of work it would
require. At the end of the day, proposing org mode as a replacement for
texinfo feels too much like a solution looking for a problem and one
that doesn't appear to have clear enough advantages to justify the work
needed to make such a transition, especially given work being done in
the texinfo area to improve its support for new formats, like epub. 

Org mode is a fantastic tool and I use it every day. There are ways org
mode could be used to augment the Emacs environment, but I don't think
it is an all or nothing situation. I'm also not convinced that embedding
org into Emacs more and more is necessarily a good thing. Tighter
coupling comes with both pros and cons.  

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: Ihor Radchenko <yantar92@gmail.com>, rms@gnu.org,
>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 14:49:21 +1000
> 
> >> >> I"m also not convinced that org mode would be the right solution for
> >> >> Emacs documentation.
> >> >
> >> > Then why Org's own manual is written in Org instead of Texinfo?
> >> 
> >> I do not know if Org mode would be a good solution for Emacs
> >> documentation. However, compared to texinfo, our manual is lacking index
> >> entries in all the exported versions but texinfo.
> >
> > Lack of an index makes for a less useful manual, IME.
> >
> > Anyway, is the lack of the index the _only_ missing feature from what
> > the Org markup provides as compared to Texinfo?  Tim's message sounded
> > like there are many more omissions.
> 
> My statement is simply waht it says. I am not convinced org mode would
> be the right solution for Emacs documentation. This is for many reasons,
> some relating to org, some relating to the fact I'm not convinced there
> is a need to replace texinfo, especially given that in some areas, I
> think texinfo is more feature rich and consistent and some relating to the feeling I have
> that the result is hard to justify given the amount of work it would
> require.

Well, it is hard to say anything intelligent without knowing those
"many reasons".

And if your POV is shared by the Org developers, then I ask again: why
is the Org manual, which weighs in at 23,000 lines, is being written
in Org?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> And if your POV is shared by the Org developers, then I ask again: why
> is the Org manual, which weighs in at 23,000 lines, is being written
> in Org?

The original reason is simply

https://list.orgmode.org/AANLkTin=gUscE4znd4pYY85X9d2wLkC+JADVcDN8AeoJ@mail.gmail.com/
>>> I was just wondering... Is the manual written in texinfo markup, or is there
>>> some obscure .org file behind the manual still?
>>> 
>>> If it really is written in texinfo, is this not a shortcoming? Org mode is
>>> capable of generating html and pdf etc. Why not use it for the manual then
>>> to set the example and show its powers!?

and then

https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
>>> It's a real pleasure to know that Org can now eat its own dog food (and
>>> to see Carsten's memorable phrase on the Org mode list again). Many
>>> thanks for picking up this orphaned project. I look forward to the day
>>> it serves as the source for org.texi.

So, now we have our manual written in Org mode and we never had reasons
to come back to texi.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 17:05:02 +0800
> 
> https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
> >>> It's a real pleasure to know that Org can now eat its own dog food (and
> >>> to see Carsten's memorable phrase on the Org mode list again). Many
> >>> thanks for picking up this orphaned project. I look forward to the day
> >>> it serves as the source for org.texi.
> 
> So, now we have our manual written in Org mode and we never had reasons
> to come back to texi.

And I get to wait for 2 minutes for the build to finish each time
org.org is touched.  Right.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> So, now we have our manual written in Org mode and we never had reasons
>> to come back to texi.
>
> And I get to wait for 2 minutes for the build to finish each time
> org.org is touched.  Right.

This is not very polite. The change did not aim to make your life
harder.

You are free to open discussion in Org ML about turning back to texi
format, but it will, for example, make things harder for me and other
people who are not familiar with texinfo format. I am not sure if build
time justifies extra maintenance burden.

I just pushed several improvements to ox.el. They reduce manual
generation time 2x on my system (using main branch). Feel free to try it
on your side. AFAIU, the effect should be more noticeable on slower
systems.

It may also help if you try to profile org-make-manuals from
mk/org-fixup.el and share the results.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:04:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So, now we have our manual written in Org mode and we never had reasons
> >> to come back to texi.
> >
> > And I get to wait for 2 minutes for the build to finish each time
> > org.org is touched.  Right.
> 
> This is not very polite. The change did not aim to make your life
> harder.

It is an expression of frustration I feel each time the build needs to
rebuild the Org manual.  I wrote it to contrast that frustration with
what I perceived as satisfaction with this solution expressed in your
description of the history of this.

I really wonder how come no one on the Org list paid any attention to
the 10-fold to 40-fold slowdown in the time it takes to build the
manual, as result of that change.  But that's water under the bridge.

> You are free to open discussion in Org ML about turning back to texi
> format, but it will, for example, make things harder for me and other
> people who are not familiar with texinfo format. I am not sure if build
> time justifies extra maintenance burden.

I don't see any chance I will convince Org developers to go back to
Texinfo at this point.

> I just pushed several improvements to ox.el. They reduce manual
> generation time 2x on my system (using main branch). Feel free to try it
> on your side. AFAIU, the effect should be more noticeable on slower
> systems.

Thanks, I hope to see this soon in the Emacs repository.

> It may also help if you try to profile org-make-manuals from
> mk/org-fixup.el and share the results.

If profiling can help, wouldn't it be simpler to invoke the same
commands from an interactive Emacs session, then show the profile?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> It is an expression of frustration I feel each time the build needs to
> rebuild the Org manual.  I wrote it to contrast that frustration with
> what I perceived as satisfaction with this solution expressed in your
> description of the history of this.

That's not my satisfaction. I quoted the emails.
In any case, while I understand your frustration, you managed to spread
it to me at least. It did not make parsing this thread any easier.

> I really wonder how come no one on the Org list paid any attention to
> the 10-fold to 40-fold slowdown in the time it takes to build the
> manual, as result of that change.  But that's water under the bridge.

We rarely have bugs related to manual builds. I recall two in many
years.

Usually documentation is built automatically on ELPA and by our
publishing scripts on orgmode.org.

>> I just pushed several improvements to ox.el. They reduce manual
>> generation time 2x on my system (using main branch). Feel free to try it
>> on your side. AFAIU, the effect should be more noticeable on slower
>> systems.
>
> Thanks, I hope to see this soon in the Emacs repository.

Not soon. Unless you want major changes for Emacs 28.2. We restricted
stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

>> It may also help if you try to profile org-make-manuals from
>> mk/org-fixup.el and share the results.
>
> If profiling can help, wouldn't it be simpler to invoke the same
> commands from an interactive Emacs session, then show the profile?

This is exactly what I meant. To run org-make-manuals from interactive
Emacs session.

org-make-manuals takes about 20 seconds on my system (for combined Org
export and texinfo invocation). Your system is clearly different and
might have different bottlenecks.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I really wonder how come no one on the Org list paid any attention to
> > the 10-fold to 40-fold slowdown in the time it takes to build the
> > manual, as result of that change.  But that's water under the bridge.
> 
> We rarely have bugs related to manual builds. I recall two in many
> years.
> 
> Usually documentation is built automatically on ELPA and by our
> publishing scripts on orgmode.org.

So basically no one builds Org, including the manual, on their own
system?  Even not the Org developers?

> >> I just pushed several improvements to ox.el. They reduce manual
> >> generation time 2x on my system (using main branch). Feel free to try it
> >> on your side. AFAIU, the effect should be more noticeable on slower
> >> systems.
> >
> > Thanks, I hope to see this soon in the Emacs repository.
> 
> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

This is not needed for the emacs-28 branch, so I meant master.

> >> It may also help if you try to profile org-make-manuals from
> >> mk/org-fixup.el and share the results.
> >
> > If profiling can help, wouldn't it be simpler to invoke the same
> > commands from an interactive Emacs session, then show the profile?
> 
> This is exactly what I meant. To run org-make-manuals from interactive
> Emacs session.

Then why would I need org-fixup.el?

> org-make-manuals takes about 20 seconds on my system (for combined Org
> export and texinfo invocation). Your system is clearly different and
> might have different bottlenecks.

Your Emacs is probably built with optimizations, whereas mine isn't.
The optimized version build org.info in about 30 sec, as I said, which
is not very different from your timing.

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Ihor Radchenko <yantar92@gmail.com> Cc:
    >> theophilusx@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca,
    >> acm@muc.de, emacs-devel@gnu.org Date: Sun, 12 Jun 2022 18:38:45
    >> +0800
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > I really wonder how come no one on the Org list paid any
    >> attention to > the 10-fold to 40-fold slowdown in the time it
    >> takes to build the > manual, as result of that change.  But
    >> that's water under the bridge.
    >> 
    >> We rarely have bugs related to manual builds. I recall two in
    >> many years.
    >> 
    >> Usually documentation is built automatically on ELPA and by our
    >> publishing scripts on orgmode.org.

    > So basically no one builds Org, including the manual, on their own
    > system?  Even not the Org developers?

I build org-mode from git and have noticed the slowness in building the
documentation.

Weekly building from git is my way of avoiding having to deal
immediately with a myriad of changes in past Versions of org-mode.

Best wishes.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> We rarely have bugs related to manual builds. I recall two in many
>> years.
>> 
>> Usually documentation is built automatically on ELPA and by our
>> publishing scripts on orgmode.org.
>
> So basically no one builds Org, including the manual, on their own
> system?  Even not the Org developers?

Org releases are not frequent for most users and occasional build time
should not matter too much. Nobody complained on the list.

Org developers (at least, me) usually run make test. It does not touch
documentation. Full re-build is rarely needed because the documentation
does not change often.

>> > Thanks, I hope to see this soon in the Emacs repository.
>> 
>> Not soon. Unless you want major changes for Emacs 28.2. We restricted
>> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
>
> This is not needed for the emacs-28 branch, so I meant master.

I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
master only tracks our stable tagged releases. Not the working branches
(both bugfix and main). I do not know the motivation behind it and I do
not know if it is a good idea to change it. Do you think that Emacs
master should instead track our Org dev branch?

>> >> It may also help if you try to profile org-make-manuals from
>> >> mk/org-fixup.el and share the results.
>> >
>> > If profiling can help, wouldn't it be simpler to invoke the same
>> > commands from an interactive Emacs session, then show the profile?
>> 
>> This is exactly what I meant. To run org-make-manuals from interactive
>> Emacs session.
>
> Then why would I need org-fixup.el?

I assume that you figured that org-fixup.el contains functions to build
manual.

Best,
Ihor

Re: Org mode and Emacs

[David Engster]

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We rarely have bugs related to manual builds. I recall two in many
>>> years.
>>> 
>>> Usually documentation is built automatically on ELPA and by our
>>> publishing scripts on orgmode.org.
>>
>> So basically no one builds Org, including the manual, on their own
>> system?  Even not the Org developers?
>
> Org releases are not frequent for most users and occasional build time
> should not matter too much. Nobody complained on the list.

Well, for the record, I did complain, but that was over 11 years ago. :-)

https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

I still think Org is way too slow to be a viable documentation system
for a project like Emacs.

-David

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>>> So basically no one builds Org, including the manual, on their own
>>> system?  Even not the Org developers?
>>
>> Org releases are not frequent for most users and occasional build time
>> should not matter too much. Nobody complained on the list.
>
> Well, for the record, I did complain, but that was over 11 years ago. :-)
>
> https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

Point taken.

> I still think Org is way too slow to be a viable documentation system
> for a project like Emacs.

Things have changed since 2014.
On my system, using Emacs 28 and latest Org, exporting org-manual +
org-guide takes 18.8 sec. This time includes genering the .texi files
and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

org export takes: 15.4 sec, which is 4.5 x texinfo time.
Considering that Org export is more sofisticated compared to texinfo and
that Org export is written in Elisp, I see this performance as
acceptable for a documentation system.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 15:36:16 +0800
> 
> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
> 
> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

I don't know about "acceptable", sorry.  It is definitely "endurable",
but 18.8 seconds is annoyingly long for an optimized build of Emacs to
do anything this simple.

Can you find any other single command that's part of building Emacs
which takes a comparable amount of time in an optimized build?

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>> 
>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> I don't know about "acceptable", sorry.  It is definitely "endurable",
> but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> do anything this simple.

Ok. I pushed things a bit further. The latest Org main takes ~8sec to
generate Org manual. This _includes_ makeinfo call. Now, Org export
takes the same time with texinfo - ~4sec on my system.

Further improvements are not trivial, given my familiarity with Org
babel and Org export code.

> Can you find any other single command that's part of building Emacs
> which takes a comparable amount of time in an optimized build?

Optimized Emacs build takes >30 minutes on my system. Mainly because of
native compilation. 40, 20, or 10 seconds spent generating Org
documentation are negligible for me.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: deng@randomsample.de,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 13:36:35 +0800
> 
> > I don't know about "acceptable", sorry.  It is definitely "endurable",
> > but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> > do anything this simple.
> 
> Ok. I pushed things a bit further. The latest Org main takes ~8sec to
> generate Org manual. This _includes_ makeinfo call. Now, Org export
> takes the same time with texinfo - ~4sec on my system.

Thanks, this is awesome.  Hope to see this speedup in the Emacs
repository soon.

> > Can you find any other single command that's part of building Emacs
> > which takes a comparable amount of time in an optimized build?
> 
> Optimized Emacs build takes >30 minutes on my system. Mainly because of
> native compilation.

The slowness of native-compilation is a long-standing bug report, so
I'd rather not use that as a reference for acceptable speed.  I meant
to compare with other commands run by the build.

> 40, 20, or 10 seconds spent generating Org documentation are
> negligible for me.

They aren't negligible, even in a native-compilation build, because
they happen at the end, when all the rest was already done, and no
large N in "-j N" of the Make command can help with that last long
wait.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Optimized Emacs build takes >30 minutes on my system. Mainly because of
>> native compilation.
>
> The slowness of native-compilation is a long-standing bug report, so
> I'd rather not use that as a reference for acceptable speed.  I meant
> to compare with other commands run by the build.

The other relatively slow part of the built process without native-comp
is Scraping autoloads.

Re: Org mode and Emacs

[David Engster]

> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

On my system, generating org.texi from org.org takes 21s, makeinfo
org.texi takes 2.7s. But it's nice to see that it has become much faster
compared to 2014.

> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

Why does the implementation language matter whether a documentation
system is acceptable? And while sophisticated, Org is actually still
missing features (like a proper index) to be a suitable replacement.

Org is about 1/16th of the whole Emacs documentation, so we're looking
at over 5min if everything was written in Org, give or take. People were
already up in arms when the switch from Texinfo v4 to v5 was done (which
switched to a Perl implementation). When documentation generation takes
a long time, writing it becomes more painful, as you cannot quickly
check the resulting output. And I think you underestimate how important
a quick build process is. Apart from developer annoyance, you need less
resources for CI, for instance.

-David

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>
> On my system, generating org.texi from org.org takes 21s, makeinfo
> org.texi takes 2.7s. But it's nice to see that it has become much faster
> compared to 2014.

Can you test the latest Org main?

>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> Why does the implementation language matter whether a documentation
> system is acceptable? And while sophisticated, Org is actually still
> missing features (like a proper index) to be a suitable replacement.

Well. It was a rather hand-waving argument: Elisp is not as fast of many
other programming languages. So software written in Elisp may be slow
despite the best effort. etc etc

In any case, that argument is futile now. I managed to get Org export
work as fast as texinfo.

As for proper index, we do support index for texinfo export
specifically and for publishing websites. A more general-purpose index
support is in the works. See https://github.com/tecosaur/org-glossary

> Org is about 1/16th of the whole Emacs documentation, so we're looking
> at over 5min if everything was written in Org, give or take. People were
> already up in arms when the switch from Texinfo v4 to v5 was done (which
> switched to a Perl implementation). When documentation generation takes
> a long time, writing it becomes more painful, as you cannot quickly
> check the resulting output. And I think you underestimate how important
> a quick build process is. Apart from developer annoyance, you need less
> resources for CI, for instance.

Fair point.

Best,
Ihor

Re: Org mode and Emacs

[David Engster]

> David Engster <deng@randomsample.de> writes:
>
>>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>>> org-guide takes 18.8 sec. This time includes genering the .texi files
>>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>>
>> On my system, generating org.texi from org.org takes 21s, makeinfo
>> org.texi takes 2.7s. But it's nice to see that it has become much faster
>> compared to 2014.
>
> Can you test the latest Org main?

I tested with latest Emacs master and with increased gc-cons-threshold
it takes ~5 seconds to generate org.texi. Great job!

-David

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:19:10 +0800
> 
> >> > Thanks, I hope to see this soon in the Emacs repository.
> >> 
> >> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> >> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
> >
> > This is not needed for the emacs-28 branch, so I meant master.
> 
> I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
> master only tracks our stable tagged releases. Not the working branches
> (both bugfix and main). I do not know the motivation behind it and I do
> not know if it is a good idea to change it. Do you think that Emacs
> master should instead track our Org dev branch?

I see no need for such significant changes in our merging practices.
Rather, I hoped that the speedup you achieved will be important enough
to make an exception and install just it in the Emacs master branch.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> I see no need for such significant changes in our merging practices.
> Rather, I hoped that the speedup you achieved will be important enough
> to make an exception and install just it in the Emacs master branch.

See the attached patches.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-export-get-footnote-definition-Pre-cache-referen.patch --]
[-- Type: text/x-patch, Size: 3096 bytes --]

From 3342137359f122ed7168dc75096c6a5d3839a0c2 Mon Sep 17 00:00:00 2001
Message-Id: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:05:16 +0800
Subject: [PATCH 1/8] org-export-get-footnote-definition: Pre-cache references
 in parse tree

* lisp/ox.el (org-export-get-footnote-definition): Pre-process parse
tree once to filter out all non-footnote elements.  This speeds up
subsequent footnote definition searches.
---
 lisp/ox.el | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 2a3edaa50..7f90dc36f 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3748,28 +3748,33 @@ (defun org-export-get-footnote-definition (footnote-reference info)
     (if (not label) (org-element-contents footnote-reference)
       (let ((cache (or (plist-get info :footnote-definition-cache)
 		       (let ((hash (make-hash-table :test #'equal)))
+                         ;; Cache all the footnotes in document for
+                         ;; later search.
+                         (org-element-map (plist-get info :parse-tree)
+                             '(footnote-definition footnote-reference)
+                           (lambda (f)
+		             ;; Skip any standard footnote reference
+		             ;; since those cannot contain a
+		             ;; definition.
+                             (unless (eq (org-element-property :type f) 'standard)
+                               (puthash
+                                (cons :element (org-element-property :label f))
+                                f
+                                hash)))
+                           info)
 			 (plist-put info :footnote-definition-cache hash)
 			 hash))))
 	(or
 	 (gethash label cache)
 	 (puthash label
-		  (org-element-map (plist-get info :parse-tree)
-		      '(footnote-definition footnote-reference)
-		    (lambda (f)
-		      (cond
-		       ;; Skip any footnote with a different label.
-		       ;; Also skip any standard footnote reference
-		       ;; with the same label since those cannot
-		       ;; contain a definition.
-		       ((not (equal (org-element-property :label f) label)) nil)
-		       ((eq (org-element-property :type f) 'standard) nil)
-		       ((org-element-contents f))
-		       ;; Even if the contents are empty, we can not
-		       ;; return nil since that would eventually raise
-		       ;; the error.  Instead, return the equivalent
-		       ;; empty string.
-		       (t "")))
-		    info t)
+                  (let ((hashed (gethash (cons :element label) cache)))
+                    (when hashed
+                      (or (org-element-contents hashed)
+		          ;; Even if the contents are empty, we can not
+		          ;; return nil since that would eventually raise
+		          ;; the error.  Instead, return the equivalent
+		          ;; empty string.
+                          "")))
 		  cache)
 	 (error "Definition not found for footnote %s" label))))))
 
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-export-resolve-fuzyy-link-Pre-cache-all-possible.patch --]
[-- Type: text/x-patch, Size: 2598 bytes --]

From cff9ed2f7660abaa5bd67e5de87416cb393b2cb8 Mon Sep 17 00:00:00 2001
Message-Id: <cff9ed2f7660abaa5bd67e5de87416cb393b2cb8.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:06:47 +0800
Subject: [PATCH 2/8] org-export-resolve-fuzyy-link: Pre-cache all possible
 search cells

* lisp/ox.el (org-export-resolve-fuzzy-link): Before matching LINK,
pre-process and cache all the non-nil search cells in the parse tree.
When matching, use the pre-processed info.  Fix the :test function for
the cache hash table.
---
 lisp/ox.el | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 7f90dc36f..4a9387519 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4346,17 +4346,27 @@ (defun org-export-resolve-fuzzy-link (link info &rest pseudo-types)
   (let* ((search-cells (org-export-string-to-search-cell
 			(org-element-property :path link)))
 	 (link-cache (or (plist-get info :resolve-fuzzy-link-cache)
-			 (let ((table (make-hash-table :test #'eq)))
+			 (let ((table (make-hash-table :test #'equal)))
+                           ;; Cache all the element search cells.
+                           (org-element-map (plist-get info :parse-tree)
+		               (append pseudo-types '(target) org-element-all-elements)
+	                     (lambda (datum)
+		               (dolist (cell (org-export-search-cells datum))
+		                 (if (gethash cell table)
+                                     (push datum (gethash cell table))
+                                   (puthash cell (list datum) table)))))
 			   (plist-put info :resolve-fuzzy-link-cache table)
 			   table)))
 	 (cached (gethash search-cells link-cache 'not-found)))
     (if (not (eq cached 'not-found)) cached
       (let ((matches
-	     (org-element-map (plist-get info :parse-tree)
-		 (append pseudo-types '(target) org-element-all-elements)
-	       (lambda (datum)
-		 (and (org-export-match-search-cell-p datum search-cells)
-		      datum)))))
+             (let (result)
+               (dolist (search-cell search-cells)
+                 (setq result
+                       (nconc
+                        result
+	                (gethash search-cell link-cache))))
+               (delq nil result))))
 	(unless matches
 	  (signal 'org-link-broken (list (org-element-property :path link))))
 	(puthash
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-org-export-resolve-id-link-Pre-cache-all-the-ids-in-.patch --]
[-- Type: text/x-patch, Size: 2832 bytes --]

From eca7a1d4dea1e8980aee6a02967d4c596114f759 Mon Sep 17 00:00:00 2001
Message-Id: <eca7a1d4dea1e8980aee6a02967d4c596114f759.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:32:35 +0800
Subject: [PATCH 3/8] org-export-resolve-id-link: Pre-cache all the ids in the
 parse tree

* lisp/ox.el (org-export-resolve-id-link): Pre-cache all the ids in
the parse tree for faster lookup.
---
 lisp/ox.el | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 4a9387519..b431d7119 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4393,15 +4393,27 @@ (defun org-export-resolve-id-link (link info)
 \"custom-id\".  Throw an error if no match is found."
   (let ((id (org-element-property :path link)))
     ;; First check if id is within the current parse tree.
-    (or (org-element-map (plist-get info :parse-tree) 'headline
-	  (lambda (headline)
-	    (when (or (equal (org-element-property :ID headline) id)
-		      (equal (org-element-property :CUSTOM_ID headline) id))
-	      headline))
-	  info 'first-match)
-	;; Otherwise, look for external files.
-	(cdr (assoc id (plist-get info :id-alist)))
-	(signal 'org-link-broken (list id)))))
+    (or (let ((local-ids (or (plist-get info :id-local-cache)
+                             (let ((table (make-hash-table :test #'equal)))
+                               (org-element-map
+                                   (plist-get info :parse-tree)
+                                   'headline
+                                 (lambda (headline)
+                                   (let ((id (org-element-property :ID headline))
+                                         (custom-id (org-element-property :CUSTOM_ID headline)))
+                                     (when id
+                                       (unless (gethash id table)
+                                         (puthash id headline table)))
+                                     (when custom-id
+                                       (unless (gethash custom-id table)
+                                         (puthash custom-id headline table)))))
+                                 info)
+                               (plist-put info :id-local-cache table)
+                               table))))
+          (gethash id local-ids))
+        ;; Otherwise, look for external files.
+        (cdr (assoc id (plist-get info :id-alist)))
+        (signal 'org-link-broken (list id)))))
 
 (defun org-export-resolve-radio-link (link info)
   "Return radio-target object referenced as LINK destination.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-org-export-as-Do-not-update-buffer-settings-when-not.patch --]
[-- Type: text/x-patch, Size: 3174 bytes --]

From fb8b921066ca84aaa7034c7f13e2cc62da40d7ad Mon Sep 17 00:00:00 2001
Message-Id: <fb8b921066ca84aaa7034c7f13e2cc62da40d7ad.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:03:18 +0800
Subject: [PATCH 4/8] org-export-as: Do not update buffer settings when not
 modified

* lisp/ox.el (org-export-as): Use `buffer-chars-modified-tick' and
avoid extra invocations of `org-set-regexps-and-options' and
`org-update-radio-target-regexp' when the buffer is not changed.
Also, disable folding checks.  Folding is irrelevant inside export
buffer.
---
 lisp/ox.el | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index b431d7119..a4512270c 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2956,11 +2956,12 @@ (defun org-export-as
 		    (mapcar (lambda (o) (and (eq (nth 4 o) 'parse) (nth 1 o)))
 			    (append (org-export-get-all-options backend)
 				    org-export-options-alist))))
-	     tree)
+	     tree modified-tick)
 	;; Update communication channel and get parse tree.  Buffer
 	;; isn't parsed directly.  Instead, all buffer modifications
 	;; and consequent parsing are undertaken in a temporary copy.
 	(org-export-with-buffer-copy
+         (font-lock-mode -1)
 	 ;; Run first hook with current back-end's name as argument.
 	 (run-hook-with-args 'org-export-before-processing-hook
 			     (org-export-backend-name backend))
@@ -2972,6 +2973,7 @@ (defun org-export-as
 	 ;; potentially invasive changes.
 	 (org-set-regexps-and-options)
 	 (org-update-radio-target-regexp)
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;;  Possibly execute Babel code.  Re-run a macro expansion
 	 ;;  specifically for {{{results}}} since inline source blocks
 	 ;;  may have generated some more.  Refresh buffer properties
@@ -2979,8 +2981,10 @@ (defun org-export-as
 	 (when org-export-use-babel
 	   (org-babel-exp-process-buffer)
 	   (org-macro-replace-all '(("results" . "$1")) parsed-keywords)
-	   (org-set-regexps-and-options)
-	   (org-update-radio-target-regexp))
+           (unless (eq modified-tick (buffer-chars-modified-tick))
+	     (org-set-regexps-and-options)
+	     (org-update-radio-target-regexp))
+           (setq modified-tick (buffer-chars-modified-tick)))
 	 ;; Run last hook with current back-end's name as argument.
 	 ;; Update buffer properties and radio targets one last time
 	 ;; before parsing.
@@ -2988,8 +2992,10 @@ (defun org-export-as
 	 (save-excursion
 	   (run-hook-with-args 'org-export-before-parsing-hook
 			       (org-export-backend-name backend)))
-	 (org-set-regexps-and-options)
-	 (org-update-radio-target-regexp)
+         (unless (eq modified-tick (buffer-chars-modified-tick))
+	   (org-set-regexps-and-options)
+	   (org-update-radio-target-regexp))
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;; Update communication channel with environment.
 	 (setq info
 	       (org-combine-plists
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-doc-Makefile-Disable-GC-during-export.patch --]
[-- Type: text/x-patch, Size: 1396 bytes --]

From 7f3ee582c1105c66a66fb61786c4e086e30460b4 Mon Sep 17 00:00:00 2001
Message-Id: <7f3ee582c1105c66a66fb61786c4e086e30460b4.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 07:29:06 +0800
Subject: [PATCH 5/8] doc/Makefile: Disable GC during export

* doc/Makefile (org.texi):
(orgguide.texi): Set `gc-cons-threshold` to `most-positive=fixnum' and
thus disable garbage collection while exporting manuals.  This reduces
the manual generation time.
---
 doc/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/Makefile b/doc/Makefile
index cb6d72bdc..5911bd08a 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -31,12 +31,14 @@ org.texi:	org-manual.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-manual)'
 
 orgguide.texi:	org-guide.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-guide)'
 
 org-version.inc:	org.texi
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-org-export-data-Concatenate-strings-in-temporary-buf.patch --]
[-- Type: text/x-patch, Size: 3492 bytes --]

From 4a466e28999bbaa5169b6fad414cf08c6d23cd7b Mon Sep 17 00:00:00 2001
Message-Id: <4a466e28999bbaa5169b6fad414cf08c6d23cd7b.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:01:53 +0800
Subject: [PATCH 6/8] org-export-data: Concatenate strings in temporary buffer
 for performance

* lisp/ox.el (org-export-data): Use temporary buffer to collect export
data instead of `mapconcat'.  Using buffer puts less load on garbage
collector.
---
 lisp/ox.el | 50 ++++++++++++++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 22 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index a4512270c..ae7e41e57 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1923,28 +1923,34 @@ (defun org-export-data (data info)
 			      (and (not greaterp)
 				   (memq type org-element-recursive-objects)))
 			     (contents
-			      (mapconcat
-			       (lambda (element) (org-export-data element info))
-			       (org-element-contents
-				(if (or greaterp objectp) data
-				  ;; Elements directly containing
-				  ;; objects must have their indentation
-				  ;; normalized first.
-				  (org-element-normalize-contents
-				   data
-				   ;; When normalizing first paragraph
-				   ;; of an item or
-				   ;; a footnote-definition, ignore
-				   ;; first line's indentation.
-				   (and
-				    (eq type 'paragraph)
-				    (memq (org-element-type parent)
-					  '(footnote-definition item))
-				    (eq (car (org-element-contents parent))
-					data)
-				    (eq (org-element-property :pre-blank parent)
-					0)))))
-			       "")))
+                              (let ((export-buffer (current-buffer)))
+                                (with-temp-buffer
+                                  (dolist (element (org-element-contents
+				                    (if (or greaterp objectp) data
+				                      ;; Elements directly containing
+				                      ;; objects must have their indentation
+				                      ;; normalized first.
+				                      (org-element-normalize-contents
+				                       data
+				                       ;; When normalizing first paragraph
+				                       ;; of an item or
+				                       ;; a footnote-definition, ignore
+				                       ;; first line's indentation.
+				                       (and
+				                        (eq type 'paragraph)
+				                        (memq (org-element-type parent)
+					                      '(footnote-definition item))
+				                        (eq (car (org-element-contents parent))
+					                    data)
+				                        (eq (org-element-property :pre-blank parent)
+					                    0))))))
+                                    (insert
+                                     ;; Use right local variable
+                                     ;; environment if there are, for
+                                     ;; example, #+BIND variables.
+                                     (with-current-buffer export-buffer
+                                       (org-export-data element info))))
+                                  (buffer-string)))))
 			(broken-link-handler
 			 (funcall transcoder data
 				  (if (not greaterp) contents
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-org-element-map-Avoid-repetitive-plist-get-call.patch --]
[-- Type: text/x-patch, Size: 1486 bytes --]

From efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc Mon Sep 17 00:00:00 2001
Message-Id: <efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 09:28:27 +0800
Subject: [PATCH 7/8] org-element-map: Avoid repetitive `plist-get' call

* lisp/org-element.el (org-element-map): Do not call `(plist-get info
:ignore-list)' on every iteration.
---
 lisp/org-element.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index 9db1406b3..20b5b0303 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4391,6 +4391,7 @@ (defun org-element-map
 		       ;; every element it encounters.
 		       (and (not (eq category 'elements))
 			    (setq category 'elements))))))))
+         (--ignore-list (plist-get info :ignore-list))
 	 --acc)
     (letrec ((--walk-tree
 	      (lambda (--data)
@@ -4400,7 +4401,7 @@ (defun org-element-map
 		  (cond
 		   ((not --data))
 		   ;; Ignored element in an export context.
-		   ((and info (memq --data (plist-get info :ignore-list))))
+		   ((and info (memq --data --ignore-list)))
 		   ;; List of elements or objects.
 		   ((not --type) (mapc --walk-tree --data))
 		   ;; Unconditionally enter parse trees.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-org-cite-list-citations-Cache-footnote-definition-se.patch --]
[-- Type: text/x-patch, Size: 3309 bytes --]

From 56830c1792b548e9157cdd95f4bb7c093e390a05 Mon Sep 17 00:00:00 2001
Message-Id: <56830c1792b548e9157cdd95f4bb7c093e390a05.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 10:43:29 +0800
Subject: [PATCH 8/8] org-cite-list-citations: Cache footnote-definition
 searches

* lisp/oc.el (org-cite-list-citations): Avoid quadratic complexity.
Pre-calculate list of all footnote definitions and cache the footnote
label search hits.  Do not make `org-element-map' accumulate unused
result.
---
 lisp/oc.el | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/lisp/oc.el b/lisp/oc.el
index eb5f519cb..c4cd0268c 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -808,6 +808,8 @@ (defun org-cite-list-citations (info)
   (or (plist-get info :citations)
       (letrec ((cites nil)
                (tree (plist-get info :parse-tree))
+               (definition-cache (make-hash-table :test #'equal))
+               (definition-list nil)
                (find-definition
                 ;; Find definition for standard reference LABEL.  At
                 ;; this point, it is impossible to rely on
@@ -816,11 +818,21 @@ (defun org-cite-list-citations (info)
                 ;; un-processed citation objects.  So we use
                 ;; a simplified version of the function above.
                 (lambda (label)
-                  (org-element-map tree 'footnote-definition
-                    (lambda (d)
-                      (and (equal label (org-element-property :label d))
-                           (or (org-element-contents d) "")))
-                    info t)))
+                  (or (gethash label definition-cache)
+                      (org-element-map
+                          (or definition-list
+                              (setq definition-list
+                                    (org-element-map
+                                        tree
+                                        'footnote-definition
+                                      #'identity info)))
+                          'footnote-definition
+                        (lambda (d)
+                          (and (equal label (org-element-property :label d))
+                               (puthash label
+                                        (or (org-element-contents d) "")
+                                        definition-cache)))
+                        info t))))
                (search-cites
                 (lambda (data)
                   (org-element-map data '(citation footnote-reference)
@@ -834,7 +846,8 @@ (defun org-cite-list-citations (info)
                         (_
                          (let ((label (org-element-property :label datum)))
                            (funcall search-cites
-                                    (funcall find-definition label))))))
+                                    (funcall find-definition label)))))
+                      nil)
                     info nil 'footnote-definition t))))
         (funcall search-cites tree)
         (let ((result (nreverse cites)))
-- 
2.35.1

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 15:03:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see no need for such significant changes in our merging practices.
> > Rather, I hoped that the speedup you achieved will be important enough
> > to make an exception and install just it in the Emacs master branch.
> 
> See the attached patches.

Thanks, I installed these, with one exception: I limited
gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
most-positive-fixnum.  The slowdown in producing the Org manual as
result of that is negligible, and I didn't feel comfortable with
disabling GC altogether, as it caused the Emacs memory footprint go up
to 0.5GB.

These changes slash the conversion from 2.5 min to just 35 sec on my
system, with an unoptimized build of Emacs, which is a terrific
speedup.  Thanks!

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Mattias Engdegård]

16 juni 2022 kl. 10.13 skrev Eli Zaretskii <eliz@gnu.org>:

> I limited
> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
> most-positive-fixnum.

Some timings for the export to .texi (old machine, optimised build, bytecode only):

|    gc-cons-threshold | time |
|----------------------+------|
|               800000 | 14.1 | (default value)
|             25000000 |  6.2 |
|             50000000 |  5.7 |
|            100000000 |  5.7 |
| most-positive-fixnum |  5.1 |

thus Eli's choice is very good, and we really should do something about that GC of ours.

> a terrific
> speedup.  Thanks!

Echoed!

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Mattias Engdegård <mattiase@acm.org> writes:

>> I limited
>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>> most-positive-fixnum.
>
> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>
> |    gc-cons-threshold | time |
> |----------------------+------|
> |               800000 | 14.1 | (default value)
> |             25000000 |  6.2 |
> |             50000000 |  5.7 |
> |            100000000 |  5.7 |
> | most-positive-fixnum |  5.1 |
>
> thus Eli's choice is very good, and we really should do something about that GC of ours.

I am wondering if there is a "universal" value suitable for one-off Emacs
batch invocations.

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-16 20:58:07] wrote:
> Mattias Engdegård <mattiase@acm.org> writes:
>>> I limited
>>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>>> most-positive-fixnum.
>>
>> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>>
>> |    gc-cons-threshold | time |
>> |----------------------+------|
>> |               800000 | 14.1 | (default value)
>> |             25000000 |  6.2 |
>> |             50000000 |  5.7 |
>> |            100000000 |  5.7 |
>> | most-positive-fixnum |  5.1 |
>>
>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>
> I am wondering if there is a "universal" value suitable for one-off Emacs
> batch invocations.

I doubt that's the case, but of course we should try and use values that
work "overall best" on average.  Maybe we should revisit the current
knobs and their default values.

Currently the two important knobs we have are:

    gc-cons-threshold
    gc-cons-percentage

Maybe it's time for some benchmarks with various values for these knobs
to see if the values we're using now are "good enough" or might need to
be changed.

We could also try and be bit more discerning.  E.g. currently when the
program is in a phase where it builds a lot of new data-structures, we
run the GC everytime some amount of new memory has been allocated, but
presumably almost none of those objects have died yet (we're still
building the overall datastructure) so the GC will be a pure waste of
time.  OTOH in other phases the code allocates objects which are used
only temporarily, in which case it's beneficial to run the GC somewhat
frequently to avoid growing our heap unnecessarily (and suffering
fragmentation as a result).

It's hard to know beforehand whether a GC will be useful or not, tho.
But maybe we can find good heuristics.  E.g. have something like
a `gc-cons-percentage` which depends on how much garbage we collected in
the last GC: if a GC doesn't collect any garbage (or very little of it),
it's a sign that we're in a phase where running the GC is not very
useful so we should wait a bit more until the next GC, whereas if the GC
collected a fair bit of garbage, it's a sign that we're in a phase where
running the GC is beneficial and we can run it a bit more often.


        Stefan

Re: Org mode and Emacs

[Pankaj Jangid]

Ihor Radchenko <yantar92@gmail.com> writes:

>> This is not needed for the emacs-28 branch,
>> so I meant master.
>
> I am not too familiar with how Emacs handles
> Org updates. AFAIK, Emacs master only tracks
> our stable tagged releases. Not the working
> branches (both bugfix and main). I do not
> know the motivation behind it and I do not
> know if it is a good idea to change it. Do
> you think that Emacs master should instead
> track our Org dev branch?
>

Emacs master is the development branch of
Emacs. I have always wanted Org dev branch to
be part of that. But may be that Org’s dev
branch is not as stable as Emacs’. Hence it is
kept that way.

I have complained about this branch confusion
once on the emacs-devel mailing list. Only to
learn that there is a separate mailing list for
org development related things.

Even for reporting bugs there was a different
mailing list for Org. I am not sure what is the
situation now. I use whichever version comes
with master branch of Emacs. And I report bugs
using ‘M-x report-emacs-bug’ only.

Re: Org mode and Emacs

[Visuwesh]

[வியாழன் ஜூன் 16, 2022 08:49] Pankaj Jangid wrote:

> Even for reporting bugs there was a different
> mailing list for Org. I am not sure what is the
> situation now. I use whichever version comes
> with master branch of Emacs. And I report bugs
> using ‘M-x report-emacs-bug’ only.

The situation is still the same and you're supposed to report org-mode
bugs with M-x org-submit-bug-report.

Re: Org mode and Emacs

[Bastien]

Here is the email I sent when I accepted to switch to using .org as
the native format for Org documentation:

https://list.orgmode.org/orgmode/87371gfas7.fsf@bzg.fr/

As you can read, I wanted to make the switch as an experiment to see
if we were really solving a problem here.

I believe we didn't get more contributions to the manual by switching
to .org, so I'd be in favor of switching back to using .texi as the
native format for Org's documentation.  (Not for 9.6, obviously, more
probably for 10.0 -- I'll discuss this with other Org maintainers.)

The other topics in this thread (make Org's Texinfo exporter provide
good .texi manuals, make Org more modular, etc.) are interesting, but
they are really separate questions IMHO.

-- 
 Bastien

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > some relating to org, some relating to the fact I'm not convinced there
  > > is a need to replace texinfo, especially given that in some areas, I
  > > think texinfo is more feature rich and consistent and some relating to the feeling I have
  > > that the result is hard to justify given the amount of work it would
  > > require.

Why replace Texinfo?  Because it has some tricky syntax which it
inherits directly from TeX.  And some syntax which is cumbersome,
again imposed directly by TeX.

Nowadays nobody can maintain the TeX-based part which is used to
generate printed manuals.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Werner LEMBERG]

> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.

Well, the thing is backward compatibility – for the sake of the GNU
project.  If Texinfo was allowed to drop that (in particular, the many
unfortunate restrictions of the `.info` file format), everything would
be much easier.

> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.

How do you come to this conclusion?  `texinfo.tex` is actively
maintained, and bug reports (which I submitted a lot) are usually
fixed within days.

Of course, there are some areas where more development would be
desirable (in particular, a better font interface based on ideas from
LaTeX).  On the other hand, the next Texinfo version will contain a
converter from Texinfo source to LaTeX source, which should help
overcome (at least some of) the font problems.

> (In the Info version of the file, the title of the node for @code is
> actually formatted wrong.  It has single quotes around `@code' in
> the title itself.  Texinfo does have some problems that need fixing,
> and it is hard to fix anything in it.)

I strongly suggest that you write to bug-texinfo, discussing this
issue and your other concerns.


    Werner

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> One of Texinfo's crucial features is a wide range of semantic markup
> constructs, each of which can generate different output depending on
> the output format.  For instance, Texinfo has @var, @emph and @dfn,
> all of which generate italics in printed output, but they differ in
> what they generate for other output formats.  There are probably 15
> other such constructs.

Could you please point to the place where I can read about the different
generated output?

> Does Org format have the ability to make all these distinctions?
> If not, I suspect that the Org mode documentation isn't following
> all our style conventions for documentation.

Org manual does not AFAIK, other than various index entries.
However, I am not sure what exactly you are referring to.
Could you please provide a link describing the conventions you are
referring to?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   For instance, Texinfo has @var, @emph and @dfn,
  > > all of which generate italics in printed output, but they differ in
  > > what they generate for other output formats.  There are probably 15
  > > other such constructs.

  > Could you please point to the place where I can read about the different
  > generated output?

They are in the Texinfo manual.  Look at node "Indicating", which is
section 7.1 in the version I have in Info.

(In the Info version of the file, the title of the node for @code is
actually formatted wrong.  It has single quotes around `@code' in the
title itself.  Texinfo does have some problems that need fixing, and
it is hard to fix anything in it.)

  > One key reason I worry about going down that road is that I suspect it
  > would complicate org's syntax. Two key benefits of org mode is that the
  > basic syntax is simple and it maps reasonably consistently acorss
  > different output formats. However, this flexibility does come at a cost.
  > To provide consistency across export formats, the basic formatting
  > 'concepts' need to be somewhat 'generalised', which means at times you
  > will loose some of the more advanced or sophisticated formatting power
  > of some export back-ends. 

I suspect we are slightly miscommunicating, because Texinfo already
generates several different formats of output, and each markup construct
is carefully defined about how it should appear in each output format.

So I'm sure it is possible to define additional markup constucts and
make each one do, in each output format, what Texinfo would (or does)
do with it.

The only hard part is finding syntax for them.

  > As pointed out elswhere in this thread, org could support missing
  > texinfo syntax using texinfo specific blocks. However, that isn't a
  > great experience from an authoring perspective. It is fine if you only
  > need to do it occasionally, but if you have to constantly add such
  > blocks in order to get really well formatted texinfo output,

This is rather vague.  Why do you thik this would be difficult to use
Can we find a solution to make it easy to use?

We don't have to be limited to whichever escape syntaxes Org already
satisfies.  This use is _important_.  For this, it is worth making new
ones.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> They are in the Texinfo manual.  Look at node "Indicating", which is
> section 7.1 in the version I have in Info.

Thanks!

>   > One key reason I worry about going down that road is that I suspect it
>   > would complicate org's syntax. Two key benefits of org mode is that the
>   > basic syntax is simple and it maps reasonably consistently acorss
>   > different output formats. However, this flexibility does come at a cost.
>   > To provide consistency across export formats, the basic formatting
>   > 'concepts' need to be somewhat 'generalised', which means at times you
>   > will loose some of the more advanced or sophisticated formatting power
>   > of some export back-ends. 
>
> I suspect we are slightly miscommunicating, because Texinfo already
> generates several different formats of output, and each markup construct
> is carefully defined about how it should appear in each output format.
>
> So I'm sure it is possible to define additional markup constucts and
> make each one do, in each output format, what Texinfo would (or does)
> do with it.
>
> The only hard part is finding syntax for them.

As I stated in another message, we may not need to define additional
markup constructs. texinfo-specific markup appears to be a little bit
too specific for general-purpose formatting aimed by Org. Instead of
modifying Org syntax (and forcing all third-party packages adapt), we
can leverage the fact that parts of Org syntax are programmable in
Elisp.

Org provides links as a flexible markup element - its export to
different formats can be arbitrarily customized (via function).
See A.3 Adding Hyperlink Types of Org manual.

For paragraph-level constructs, there is
org-export-filter-parse-tree-functions and external
https://github.com/alhassy/org-special-block-extras
We may implement something more reminiscent to link customization in
future. org-export-filter-parse-tree-functions is a bit too generic (one
function for everything).

In summary, Org does not really need to change syntax in order to
support full texinfo markup. AFAIU, the missing parts are
some of the texinfo-specific markup elements, abbreviations, and
generating index. They all can be implemented as new libraries.
Abbreviations and index are WIP at
https://github.com/tecosaur/org-glossary
The texinfo-specific markup can be another (optionally loaded) Org
library.

>   > As pointed out elswhere in this thread, org could support missing
>   > texinfo syntax using texinfo specific blocks. However, that isn't a
>   > great experience from an authoring perspective. It is fine if you only
>   > need to do it occasionally, but if you have to constantly add such
>   > blocks in order to get really well formatted texinfo output,
>
> This is rather vague.  Why do you thik this would be difficult to use
> Can we find a solution to make it easy to use?

This was a reference to Org export blocks:

#+begin_export latex
This text will only appear when exported to \LaTeX, not to any other format.
#+end_export

They may be not great experience what one needs to do something like

#+begin_export texinfo
@var{test} will only appear in .texi output.
#+end_export

#+begin_export latex
\emph{test} will only appear in .tex output.
#+end_export

many times in the same .org document.

However, we have special blocks for this purpose. They are extendable,
as I descibed above. Though some improvements in Org core might be
needed if we have to use this extensibility more frequently.

Best,
Ihor

Re: Org mode and Emacs

[Robert Pluim]

>>>>> On Tue, 14 Jun 2022 21:18:58 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> #+begin_export latex
    Ihor> This text will only appear when exported to \LaTeX, not to any other format.
    Ihor> #+end_export


    Ihor> They may be not great experience what one needs to do something like

    Ihor> #+begin_export texinfo
    Ihor> @var{test} will only appear in .texi output.
    Ihor> #+end_export

    Ihor> #+begin_export latex
    Ihor> \emph{test} will only appear in .tex output.
    Ihor> #+end_export

    Ihor> many times in the same .org document.

    Ihor> However, we have special blocks for this purpose. They are extendable,
    Ihor> as I descibed above. Though some improvements in Org core might be
    Ihor> needed if we have to use this extensibility more frequently.

My first thought here was Org macros, but as far as I can tell they canʼt
be multi-line, which would make them a bit cumbersome to use. Tell us
more about special blocks, the documentation on them is a bit sparse.

Robert
-- 

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Making org mode syntax equivalent to texinfo syntax seems like a
  > mistake to me.

If this succeeds, it would be an important advance for the GNU system.
We would replace Texinfo with a much cleaner system, easier to use and
more maintainable.  Not solely for Emacs, but for ALL our
documentation!

                   The original idea was to have a light weight syntax which
  > is easyh to learn, not create a clone of texinfo. Besides, I suspect it
  > would be very difficult to maintain backwards compatibility. 

These are real concerns, but they are not real certainties.  If we look
for solutions, we may find good ones.

I hope someone will give it a try.

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

These constructs make it possible to carry out our documentation style
constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

There is nothing fundamentally hard about supporting these
distinctions.  Can someone please examine which ones Org supports and
which ones not, and propose exensions for the ones not yet supported?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Making org mode syntax equivalent to texinfo syntax seems like a
>   > mistake to me.
>
> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!
>

One key reason I worry about going down that road is that I suspect it
would complicate org's syntax. Two key benefits of org mode is that the
basic syntax is simple and it maps reasonably consistently acorss
different output formats. However, this flexibility does come at a cost.
To provide consistency across export formats, the basic formatting
'concepts' need to be somewhat 'generalised', which means at times you
will loose some of the more advanced or sophisticated formatting power
of some export back-ends. 

There are times, primarily where you need specialised or specific
formatting in a particular format, where you need to drop down to that
low level formatting 'language'. This is the org 'escape hatch', which
provides a way to better leverage off the specific capabilities of a
partricular back-end. For exmaple, you can embed latex in your document
which will be added to your formated output when you generate a PDF/PS
or *.tex output file. However, that formatting often won't work when you
use the same source file to generate an HTML or Markdown or ODT version
because there isn't a clear way to translate the latex to that format.

As pointed out elswhere in this thread, org could support missing
texinfo syntax using texinfo specific blocks. However, that isn't a
great experience from an authoring perspective. It is fine if you only
need to do it occasionally, but if you have to constantly add such
blocks in order to get really well formatted texinfo output, it will
become frustrating. Org also supports a powerful custom link format
which could be used to add missing syntax elements, but I'm unsure on
the usability of such an approach once you have a few such cusotm links.

If we want to avoid this situation, the syntax would need to be added to
the basic org syntax. However, that will also require all back-ends
being able to interpret that syntax in some 'sane' manner, which would
likley be considerable work and in some situations, could be extremely
difficult to do consistently with good results. It would also add to the
overall amount of syntax, potentially making it more complex and harder
to learn. 

Org mode is a great mode, especially for general documentation and
information management, todo and time management and to some extent, for
literate programming and/or 'executable' documents, such as lab books or
even some devops type applicaitons. Being able to have a single source
document which can be used to generate 'reasonable' versions in
different formats. You can even spend a bit of effort to customise
things so that you can get some pretty consistent advanced formatting in
some specific export formats. However, I often find when you need the
advanced formatting power of a specific back-end, your often better just
writing the document in that back-end as it tends to take less effort
and results in cleaner source. 

The other side of th coin is the on-going development of texinfo. I have
not written enough texinfo to really understand what the issues are
which drive the desire to replace it with something else, such as org. I
know there are some criticisms regarding output formats, but I also know
there is on-going work to improve that situation. Is the right strategy
to work on org mode so that it can replace texinfo or work on texinfo to
address limitations (or both?)?

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 383 bytes --]

Note that org's export to texinfo is quite good; as an example, I wrote
the "Emacspeak Turning Twenty" article in org-mode and later exported it
to texinfo among other formats. Org's export to TeX,LaTeX and PDF is
also very good; likely superior to its export to Texinfo.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!

Org mode is not easier to use than Texinfo... at least, I didn't figure
out how to use it.

IIRC it's also much slower to create similarly sized .info files from
Org source than it is with makeinfo, and it also requires an entire
manual to be contained in a single file.

Texinfo also comes with the added benefit of not requiring Emacs to edit
or to translate into other formats.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> IIRC it's also much slower to create similarly sized .info files from
> Org source than it is with makeinfo

Could you please quantify much slower?
Do you have concrete example of a .org file which can demonstrate the
problem?

Also, I think that I have to clarify about texinfo export from Org.
Org does not currently generate .info files directly. .org file is first
transformed into .texi and then uses makeinfo to translate the generated
.texi into .info.

If there is an interest in Org being used instead of texinfo, it should
be based on the extensibility of Org rather than just making Org
replicate what texinfo does. However, I am not sure if all parties in
this discussion have a clear idea about concrete advantages of Org
compared with texinfo. I am not faimiliar with texinfo. Others may not
be as faimilar with Org.

I believe that if we want to be serious about using Org for
documentation more widely, we should first identify which particular
aspects of Org are beneficial for documentation and which particular
aspects are available in other documentation generators and must be then
provided by Org (if not yet available).

> ... , and it also requires an entire
> manual to be contained in a single file.

This is not true. Org mode supports #+include: directives to incorporate
multiple smaller files into one larger file.

> Texinfo also comes with the added benefit of not requiring Emacs to edit
> or to translate into other formats.

You can edit Org files outside Emacs. Say, in vim.
The point about exporting to other formats is valid.

This last point also raises a question. Can Elisp interpreter and
libraries be factored out of Emacs to create a way to execute Elisp
programs without installing all the interactive parts of Emacs?

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>> or to translate into other formats.
>
> You can edit Org files outside Emacs. Say, in vim.
> The point about exporting to other formats is valid.
>
> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Isn't Tim Cross(?) working on something like that -- ie. a parser for
the Org language.  Once we have a solid parser, we can build a standard
(set of?) backend(s) for much of Orgmode.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

Org already have a parser. Written in Elisp. Export is built on top of
the parser.

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This last point also raises a question. Can Elisp interpreter and
>>> libraries be factored out of Emacs to create a way to execute Elisp
>>> programs without installing all the interactive parts of Emacs?
>>
>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>> the Org language.  Once we have a solid parser, we can build a standard
>> (set of?) backend(s) for much of Orgmode.
>
> Org already have a parser. Written in Elisp. Export is built on top of
> the parser.

But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
long run, it would be better to define a "parseable" language as the
standard basis for Org.  From that, people can develop (parts of) Org on
other platforms (Vim, Beorg, Orgzly) and test/prove that they are
compatible with version X of the language.  I think Organice was doing
this, but I haven't looked at it deeply.

Oh, but I see your point about "export". By backend, I was assuming a
true parser would generate a standard "internal" language which could be
fed into simpler backends to actually do the work.  The front-end parser
and back-ends could be translated/rebuilt as needed on new platorms
(iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
the idea.

-- 
David Masterson

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>>> This last point also raises a question. Can Elisp interpreter and
>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>> programs without installing all the interactive parts of Emacs?
>>>
>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>> the Org language.  Once we have a solid parser, we can build a standard
>>> (set of?) backend(s) for much of Orgmode.
>>
>> Org already have a parser. Written in Elisp. Export is built on top of
>> the parser.
>
> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
> long run, it would be better to define a "parseable" language as the
> standard basis for Org.  From that, people can develop (parts of) Org on
> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
> compatible with version X of the language.  I think Organice was doing
> this, but I haven't looked at it deeply.
>

This is all part of the aims and process. However, the first step is to
develop a robust elisp based parser. This helps to ensure the org syntax
is consistent and helps identifies ambiguities which need to be fixed as
well as provides a reference implementation which developers can use to
develop parsers in other languages.

> Oh, but I see your point about "export". By backend, I was assuming a
> true parser would generate a standard "internal" language which could be
> fed into simpler backends to actually do the work.  The front-end parser
> and back-ends could be translated/rebuilt as needed on new platorms
> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
> the idea.

Personally, I think it is unlikely we will ever see an org mode for
other platforms which is equivalent to Emacs. However, the work which is
being done to create a clean parser in elisp and use that as the basis
for an API to manipulate org data, generate font-locking and formatting
and provide a clean API for exporters to use will make it much easier
for people to develop external org support at varying levels. 

Those doing the main work, like Ihor, are very aware of the desire to
facilitate external implementations. It is one reason that a fairly
conservative attitude to change is adopted by the project and often, one
consideration is what imapct a change would have on external org
compatible projects. Sometimes, this is challenging as on one level, we
want ot advance and improve org mode, but on the other, we want it to be
as stable as possible to reduce adverse impact on external projects. So
while the main target is and will always be Emacs, an eye is kept on
efforts of other external projects and an effort is made to support them
when possible and within Emacs and FSF principals/guidelines. 

So far, the key contributors have been doing a very good job. 

Re: Org mode and Emacs

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.

> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to

It is not the goal of Emacs to support tyrant devices such as the iPhone
running proprietary operating systems such as iOS.

If people are going to refactor Org mode, I hope they will not do it
with support for iPhones in mind.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>
> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

I am not sure why you only picked up iPhones from the list suggested by
David. The aim is not supporting iPhones. The aim is supporting
other GNU and Free Software projects that are making use of Org, which
implies conservative approach to breaking changes, especially in syntax.

Best,
Ihor

Re: Org mode and Emacs

[Tim Cross]

Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>>> long run, it would be better to define a "parseable" language as the
>>> standard basis for Org.  From that, people can develop (parts of) Org on
>>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>>> compatible with version X of the language.  I think Organice was doing
>>> this, but I haven't looked at it deeply.
>
>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>

> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

There has never been suggestion that what we are doing is to facilitate
development on non-free platforms. However, if you develop good clean
code and clear algorithms which others are able to use as a reference to
understand the syntax and semantics of a system, you cannot control how
they will use that information. In fact, it can be argued that making
such information readily available and accessible is a large part of the
freedoms being promoted by the FSF.

This is completely different from, for example, providing an interface
which can be used by a non-free platform. 

Besides, if org is as bad and difficult to learn as you have indicated
in other posts, nobody will bother implementing it on those platforms
you seem so threatened by anyway!

Re: Org mode and Emacs

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>>> This last point also raises a question. Can Elisp interpreter and
>>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>>> programs without installing all the interactive parts of Emacs?
>>>>
>>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>>> the Org language.  Once we have a solid parser, we can build a standard
>>>> (set of?) backend(s) for much of Orgmode.
>>>
>>> Org already have a parser. Written in Elisp. Export is built on top of
>>> the parser.
>>
>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.
>>
>
> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to
> develop parsers in other languages.

Semantic/Bovine ??

Agree

>> Oh, but I see your point about "export". By backend, I was assuming a
>> true parser would generate a standard "internal" language which could be
>> fed into simpler backends to actually do the work.  The front-end parser
>> and back-ends could be translated/rebuilt as needed on new platorms
>> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
>> the idea.
>
> Personally, I think it is unlikely we will ever see an org mode for
> other platforms which is equivalent to Emacs. However, the work which is
> being done to create a clean parser in elisp and use that as the basis
> for an API to manipulate org data, generate font-locking and formatting
> and provide a clean API for exporters to use will make it much easier
> for people to develop external org support at varying levels.

Agree

> Those doing the main work, like Ihor, are very aware of the desire to
> facilitate external implementations. It is one reason that a fairly
> conservative attitude to change is adopted by the project and often, one
> consideration is what imapct a change would have on external org
> compatible projects. Sometimes, this is challenging as on one level, we
> want ot advance and improve org mode, but on the other, we want it to be
> as stable as possible to reduce adverse impact on external projects. So
> while the main target is and will always be Emacs, an eye is kept on
> efforts of other external projects and an effort is made to support them
> when possible and within Emacs and FSF principals/guidelines. 
>
> So far, the key contributors have been doing a very good job. 

That's good.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>> develop parsers in other languages.
>
> Semantic/Bovine ??

Org is not context-free.
Also, Org maintaners previously rejected the idea of implementing Org
parser not in Elisp. Mainly because it would limit the ability to
maintain and contribute to Org - one would need to learn another
programming language to alter anything in Org syntax.

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This is all part of the aims and process. However, the first step is to
>>> develop a robust elisp based parser. This helps to ensure the org syntax
>>> is consistent and helps identifies ambiguities which need to be fixed as
>>> well as provides a reference implementation which developers can use to
>>> develop parsers in other languages.
>>
>> Semantic/Bovine ??
>
> Org is not context-free.

But could it be moved in that direction? (ie. Organice)

> Also, Org maintaners previously rejected the idea of implementing Org
> parser not in Elisp. Mainly because it would limit the ability to
> maintain and contribute to Org - one would need to learn another
> programming language to alter anything in Org syntax.

Hmmm. That would make it difficult to keep the language "parseable" by a
different parser.  Elisp would not provide the checks for (say) keeping
the language context-free.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> Semantic/Bovine ??
>>
>> Org is not context-free.
>
> But could it be moved in that direction? (ie. Organice)

I don't think so. It is motivated by the fundamental Org syntax design,
AFAIU. (mostly by first match wins design). We are not going to change
fundamentals of the Org syntax. It will break backward compatibility.

>> Also, Org maintaners previously rejected the idea of implementing Org
>> parser not in Elisp. Mainly because it would limit the ability to
>> maintain and contribute to Org - one would need to learn another
>> programming language to alter anything in Org syntax.
>
> Hmmm. That would make it difficult to keep the language "parseable" by a
> different parser.  Elisp would not provide the checks for (say) keeping
> the language context-free.

At this point, we are trying to "freeze" Org syntax as much as possible.
So, major changes are not expected. Different parsers should not suffer
from future changes (if they do, we should not make those changes to
start with).

As for keeping checks, we do have a set of parser tests using ERT. So,
major breakage will be prevented. On top of this, we plan to make the
parser tests more friendly to third-party tools:
https://orgmode.org/list/87fsqzi4tw.fsf@localhost

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> Semantic/Bovine ??
>>>
>>> Org is not context-free.
>>
>> But could it be moved in that direction? (ie. Organice)
>
> I don't think so. It is motivated by the fundamental Org syntax design,
> AFAIU. (mostly by first match wins design). We are not going to change
> fundamentals of the Org syntax. It will break backward compatibility.

Could Org be moved toward a "well-defined" grammar that could be
separated from the Emacs implementation to allow other systems (iOS,
Android, Windows) to implement (at least part of) a "standard" Org?
Could the backward compatibility be covered by an Emacs library where
necessary? 

>>> Also, Org maintaners previously rejected the idea of implementing Org
>>> parser not in Elisp. Mainly because it would limit the ability to
>>> maintain and contribute to Org - one would need to learn another
>>> programming language to alter anything in Org syntax.
>>
>> Hmmm. That would make it difficult to keep the language "parseable" by a
>> different parser.  Elisp would not provide the checks for (say) keeping
>> the language context-free.
>
> At this point, we are trying to "freeze" Org syntax as much as possible.
> So, major changes are not expected. Different parsers should not suffer
> from future changes (if they do, we should not make those changes to
> start with).
>
> As for keeping checks, we do have a set of parser tests using ERT. So,
> major breakage will be prevented. On top of this, we plan to make the
> parser tests more friendly to third-party tools:
> https://orgmode.org/list/87fsqzi4tw.fsf@localhost

This sounds good.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> But could it be moved in that direction? (ie. Organice)
>>
>> I don't think so. It is motivated by the fundamental Org syntax design,
>> AFAIU. (mostly by first match wins design). We are not going to change
>> fundamentals of the Org syntax. It will break backward compatibility.
>
> Could Org be moved toward a "well-defined" grammar that could be
> separated from the Emacs implementation to allow other systems (iOS,
> Android, Windows) to implement (at least part of) a "standard" Org?
> Could the backward compatibility be covered by an Emacs library where
> necessary? 

Yes. See https://orgmode.org/worg/dev/org-syntax.html
I hope it is well-defined enough for you.
I don't think that we need major changes to allow implementation in
other systems. There is already a number of existing third-party parsers
for Org:

https://github.com/200ok-ch/org-parser
https://github.com/tgbugs/laundry
https://github.com/milisims/tree-sitter-org
https://github.com/tecosaur/Org.jl

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> But could it be moved in that direction? (ie. Organice)
>>>
>>> I don't think so. It is motivated by the fundamental Org syntax design,
>>> AFAIU. (mostly by first match wins design). We are not going to change
>>> fundamentals of the Org syntax. It will break backward compatibility.
>>
>> Could Org be moved toward a "well-defined" grammar that could be
>> separated from the Emacs implementation to allow other systems (iOS,
>> Android, Windows) to implement (at least part of) a "standard" Org?
>> Could the backward compatibility be covered by an Emacs library where
>> necessary? 
>
> Yes. See https://orgmode.org/worg/dev/org-syntax.html
> I hope it is well-defined enough for you.
> I don't think that we need major changes to allow implementation in
> other systems. There is already a number of existing third-party parsers
> for Org:
>
> https://github.com/200ok-ch/org-parser
> https://github.com/tgbugs/laundry
> https://github.com/milisims/tree-sitter-org
> https://github.com/tecosaur/Org.jl

Excellent!

-- 
David Masterson

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>>> or to translate into other formats.
>>
>> You can edit Org files outside Emacs. Say, in vim.
>> The point about exporting to other formats is valid.
>>
>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

No. Ihor is the one working on the parser. 

I think Ihor's question about isolating the Elisp interpreter is about
options to make org mode work outside of Emacs. There are frequent
questions to the org list about making org available to other
editors/environments. However, the big problem is that much of the
really powerful featurs of org mode are intrinsically tied to elisp
functionality - for exmaple all the babel and backend export support.
While it is possible to write a parser in any language which would
enable basic org markup/formatting, that does not solve the problem of
executable blocks, babel/noweb and interaction with back end exporters
etc. 

This would be one example of where something like an elisp LSP module
would be useful. However, the idea of an elisp LSP module was
discouraged by a couple of people on this list over concerns that such a
module would enable non-free platforms to take advantage of elisp - a
fear which I think is overstated and which I think one person referred
to as "Jumping at shadows" 

The overall conclusion was that such a module would be considerable
amount of work. Some suggested you could use emacsclient to create
something which could be used in such a manner. However, I suspect that
if you go to the effort of installing and configuring emacs, you would
just use emacs. 

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@gmail.com> writes:

> Could you please quantify much slower?

Making the Org manual takes about 27 seconds on my system:

  GEN      org.texi
  GEN      ../../info/org.info

real	0m27.210s
user	0m25.567s
sys	0m0.783s

Which is longer than it takes for the entire (much more massive) Emacs
Lisp reference manual:

  GEN      ../../info/elisp.info

real	0m24.561s
user	0m23.660s
sys	0m0.515s

And generating the .info file from the Org manual texinfo source takes
only:

  GEN      ../../info/org.info

real	0m5.381s
user	0m5.109s
sys	0m0.141s

So it's clear that the org to texinfo export takes most of the time.

> I believe that if we want to be serious about using Org for
> documentation more widely, we should first identify which particular
> aspects of Org are beneficial for documentation and which particular
> aspects are available in other documentation generators and must be then
> provided by Org (if not yet available).

I don't see why we should be serious about using Org for our
documentation, when most people already know texinfo and are quite happy
with it.

> This is not true. Org mode supports #+include: directives to incorporate
> multiple smaller files into one larger file.

That's nice, but why isn't the Org manual itself structured that way?

> You can edit Org files outside Emacs. Say, in vim.

But does it work very well?

> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Probably not without a lot of work.

Re: Org mode and Emacs

[chad]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Sat, Jun 11, 2022 at 10:52 PM Po Lu <luangruo@yahoo.com> wrote:

> I don't see why we should be serious about using Org for our
> documentation, when most people already know texinfo and are quite happy
> with it.
>

I think a reasonable examination of the emacs-devel archives as well as the
common practices of most of the people publishing emacs lisp packages today
would lead to a very different conclusion. There are several threads about
maintenance concerns around makeinfo/texinfo, and many discussions about
replacing texinfo with, for example, HTML.There are periodic threads where
people claim that they won't try to add their project to GNU because the
burden of learning and using texinfo is too high, although those have died
down in volume since it became more practical to translate other formats to
texinfo.

None of this is to say that Org is the "right" format for Emacs
documentation. I just don't think it's correct to say "most people already
know texinfo and are quite happy with it".

~Chad

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

Re: Org mode and Emacs

[Po Lu]

chad <yandros@gmail.com> writes:

> I think a reasonable examination of the emacs-devel archives as well
> as the common practices of most of the people publishing emacs lisp
> packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo,
> and many discussions about replacing texinfo with, for example,
> HTML.

Published Emacs Lisp packages in the wild typically come with no
documentation at all, aside from doc strings and a single README file.

Org mode happens to be popular for the latter, but I don't know why.
The learning curve is high and the results seem to be unjustified.  An
easier format such as HTML or Markdown can be used instead.

> There are periodic threads where people claim that they won't try to
> add their project to GNU because the burden of learning and using
> texinfo is too high, although those have died down in volume since it
> became more practical to translate other formats to texinfo.

Why would Org mode be any different?  Unlike HTML, it is completely
specific to Emacs, which actually makes it worse than Texinfo, because
Texinfo is at least widely used throughout the entire GNU project.

And anyway, I make the following offer: if someone doesn't want to
contribute documentation (or code) because he doesn't know Texinfo, he
can write the documentation in plain text, and I will translate it to
Texinfo for him.  IIRC the GDB developers made a similar offer, and for
the many years that it existed nobody ever took advantage of it, which
shows that Texinfo is not a serious barrier to contributors.

Further more, using Org mode for documentation will make Emacs lose many
people who are actually writing documentation, right now.  At least I
have no interest in ever learning Org mode, and there seem to be no
volunteers who will translate Texinfo (or plain text) to Org mode for me
if we begin to use it for documentation in the future.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Published Emacs Lisp packages in the wild typically come with no
> documentation at all, aside from doc strings and a single README file.
>
> Org mode happens to be popular for the latter, but I don't know why.
> The learning curve is high and the results seem to be unjustified.  An
> easier format such as HTML or Markdown can be used instead.

Which indicates that others may not agree with your assertion about the
learning curve.
For simple README
https://gitlab.com/publicvoit/orgdown/-/blob/master/doc/Orgdown1-Syntax-Examples.org
is more than enough.

Best,
Ihor

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And anyway, I make the following offer: if someone doesn't want to
  > contribute documentation (or code) because he doesn't know Texinfo, he
  > can write the documentation in plain text, and I will translate it to
  > Texinfo for him.

Thank you for making this offer.  I hope that many contributors will
take you up on it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Published Emacs Lisp packages in the wild typically come with no
  > documentation at all, aside from doc strings and a single README file.

Depending on the quality of the README file, this could be imperfect
but adequate, or it could be lousy.  Please don't accept it as a standard
to judge other things by.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sat, 11 Jun 2022 23:54:39 -0400
> Cc: Ihor Radchenko <yantar92@gmail.com>, Richard Stallman <rms@gnu.org>,
>  Tim Cross <theophilusx@gmail.com>, 
>  Stefan Monnier <monnier@iro.umontreal.ca>, Alan Mackenzie <acm@muc.de>, 
>  EMACS development team <emacs-devel@gnu.org>
> 
>  I don't see why we should be serious about using Org for our
>  documentation, when most people already know texinfo and are quite happy
>  with it.
> 
> I think a reasonable examination of the emacs-devel archives as well as the common practices of most of
> the people publishing emacs lisp packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo, and many discussions about
> replacing texinfo with, for example, HTML.There are periodic threads where people claim that they won't try
> to add their project to GNU because the burden of learning and using texinfo is too high, although those have
> died down in volume since it became more practical to translate other formats to texinfo.

The real issue behind these claims is that developers seldom like
writing good documentation.  So learning to use a sophisticated markup
system such as Texinfo is overkill for them.  On top of that, if the
package is not an Emacs package, there's a problem that Texinfo
editing is not really well supported outside of Emacs, which is
another reason for complaining by people who don't use Emacs for
developing their package.

Good HTML is no easier to write than good Texinfo, but the difference
is that there are tools out there other than Emacs which can be used
to create HTML, especially if the manual is relatively simple (as many
Free Software manuals are).

Bottom line: the complaints are real, but I'm not sure they help in
this discussion, because there's no known alternative to Texinfo for
creating good documentation of the kind that we are used to in Emacs.
GCC folks are switching to Sphinx, but that has its own problems, and
without good support for it in Emacs (and no, rst.el is not enough, as
it doesn't provide enough markup commands, AFAICT) it isn't (yet) a
viable alternative.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Richard Stallman <rms@gnu.org>,  Tim Cross <theophilusx@gmail.com>,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 10:15:59 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > IIRC it's also much slower to create similarly sized .info files from
> > Org source than it is with makeinfo
> 
> Could you please quantify much slower?
> Do you have concrete example of a .org file which can demonstrate the
> problem?

With an unoptimized build of Emacs 29, generation of org.texi from
org.org takes 2.5 minutes here.  An optimized build of Emacs 28 does
that in 30 sec on one system and 1 minute on another.  Generation of
org.info then takes between 3 and 4 seconds.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).

I am not sure if I follow the argument. Major modes are allowed to
change defaults. For example, special modes often change truncate-lines
value. Org mode is also tweaking defaults (yes, many of them). I do not
see any problem here in general.

If you have some specific cases when Org mode alters Emacs defaults in a
way that bites the user, please give concrete examples. Otherwise, your
criticism is not very constructive.

> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.

This is reasonable. RMS also asked for this years back
https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
Since we cannot start Org from scratch, factoring out individual modules
is taking a lot of time and having the hostile attitudes expressed in
some of the emails in this thread is not exactly encouraging.
If you want Org to be more modular, please help by reporting
inconsistencies or misbehavior to Org ML.

Best,
Ihor

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-08 21:22:07] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT the problem seen from Emacs, is that Org is large (even for
>> a basic uses, which occasionally leads to high load times) and that it
>> doesn't follow all the usual Emacs conventions, such as
>> overriding/remapping standard key-bindings (the resulting behaviors may
>> sometimes be similar to the standard one, but even when that's the case
>> the redefinition can easily bite the user).
> I am not sure if I follow the argument. Major modes are allowed to
> change defaults. For example, special modes often change truncate-lines
> value. Org mode is also tweaking defaults (yes, many of them). I do not
> see any problem here in general.
> If you have some specific cases when Org mode alters Emacs defaults in a
> way that bites the user, please give concrete examples. Otherwise, your
> criticism is not very constructive.

[ Alan gave one or two concrete examples of things that bit him.  ]

This is not a criticism, just a description of how Org is perceived from
the side of "old-time Emacs users who aren't Org users".
The key in what you wrote above is the "yes, many of them", which means
that even tho those tweaks are minor they sum up to something that
old-time users will almost inevitably bump into.

It's not a problem for Org itself, and there are good reasons for each
one of those tweaks, I'm sure, but it does create opposition to using
Org "in core", such as in etc/NEWS.

> This is reasonable. RMS also asked for this years back
> https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
> Since we cannot start Org from scratch, factoring out individual modules
> is taking a lot of time and having the hostile attitudes expressed in
> some of the emails in this thread is not exactly encouraging.

I know it's a lot of work and I'm glad you (plural) are taking it on,
and I'm not very happy about some of the things that have been said in
that thread, which is why I tried to start this thread.


        Stefan

Re: Org mode and Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If you have some specific cases when Org mode alters Emacs defaults in a
>> way that bites the user, please give concrete examples. Otherwise, your
>> criticism is not very constructive.
>
> [ Alan gave one or two concrete examples of things that bit him.  ]

Alan gave two concrete examples:

>>> I put org.org into a buffer.  I saw lots of lines ending in ..., which is
>>> fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
>>> in a new window.  Brilliant!  Did I mention I hate context dependent
>>> commands?  So I have to type M-x outline-show-subtree, which works.  I
>>> read the screenful of text, then type C-S-<down> to scroll it up.
>>> Instead of my standard scrolling command, I get an error message about
>>> "Not at a clock log".  Org mode has stolen the key sequences
>>> C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

where he
1. Complains about major mode shadowing key binding reserved for major
   modes, according to D.2 Key Binding Conventions section of the Elisp
   manual:
>      • Sequences consisting of ‘C-c’ followed by a control character or a
>     digit are reserved for major modes.

2. C-S-<down>, which is a valid point. However, arrow commands is one of
   the few core concepts used in Org major mode.
   I am not sure if re-binding C-S-<down> is so much of a sin on Org
   mode side. Emacs manual 49.3 Customizing Key Bindings says:

>         Since most modes define their own key bindings, activating a mode
>      might override your custom key bindings.  A small number of keys are
>      reserved for user-defined bindings, and should not be used by modes, so
>      key bindings using those keys are safer in this regard.  The reserved
>      key sequences are those consisting of ‘C-c’ followed by a letter (either
>      upper or lower case), and function keys <F5> through <F9> without
>      modifiers (*note Modifier Keys::).

   As a compromise, Org may provide some kind of read-only mode just to
   navigate the document. The arrow bindings can be disabled then. They
   are for editing.

>>> I see many of these bindings as context dependent, with the name of the
>>> function named after the key sequence, not the functionality - e.g.
>>> org-shiftcontroldown.  That is not the way the rest of Emacs is
>>> constructed.  I hate context dependence (except when it is properly
>>> implemented, e.g. by major modes).

    Only a handful of bindings are context dependent. In particular,
    control/shift/meta-arrow keys, <TAB>, and C-c C-c. They are most
    frequently used when editing and navigating Org buffers.

    It would help if "properly implemented" were defined more precisely
    in the above statement.

> This is not a criticism, just a description of how Org is perceived from
> the side of "old-time Emacs users who aren't Org users".
> The key in what you wrote above is the "yes, many of them", which means
> that even tho those tweaks are minor they sum up to something that
> old-time users will almost inevitably bump into.
>
> It's not a problem for Org itself, and there are good reasons for each
> one of those tweaks, I'm sure, but it does create opposition to using
> Org "in core", such as in etc/NEWS.

I understand. However, such criticism is not helpful. Only concrete
examples can lead to actual Org improvements in this area.

Best,
Ihor

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since we cannot start Org from scratch, factoring out individual modules
  > > is taking a lot of time and having the hostile attitudes expressed in
  > > some of the emails in this thread is not exactly encouraging.

  > I know it's a lot of work and I'm glad you (plural) are taking it on,

I feel the same.  To move some of the features out of Org, so that
they are available in all Emacs modes, will make enhance their
availabiliy to all Emacs users.

Also, it could eventually make Org itself simpler -- and less daunting
to try out and learn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)