Convert README.org to plain text README while installing package

Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]


Packages on ELPA or NonGNU ELPA with README.org show pretty description
both with describe-package and web browser.  But after installing a
package, describe-package shows the Org source code, which contains
many useless things (I mean useless for showing in the *Help* buffer).

Isn't it possible to convert README.org to plain text README while
compiling the package, possibly optionally?  Changing package--compile
function should do the job.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Akib Azmain Turja <akib@disroot.org> writes:

Hi Akib,

> Packages on ELPA or NonGNU ELPA with README.org show pretty
> description both with describe-package and web browser.  But after
> installing a package, describe-package shows the Org source code,
> which contains many useless things (I mean useless for showing in the
> *Help* buffer).
>
> Isn't it possible to convert README.org to plain text README while
> compiling the package, possibly optionally?  Changing package--compile
> function should do the job.

An alternative would be to enable (parts of) org-mode in the *Help*
buffer.  I've just tried `M-x describe-package RET corfu RET` which
comes with a README.org and then enabling org-mode in the *Help* buffer.
That looked really nice and didn't seem to cause bad effects except that
the [back] / [forward] buttons stopped working.

Bye,
Tassilo

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tassilo.

On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
> Akib Azmain Turja <akib@disroot.org> writes:

> Hi Akib,

> > Packages on ELPA or NonGNU ELPA with README.org show pretty
> > description both with describe-package and web browser.  But after
> > installing a package, describe-package shows the Org source code,
> > which contains many useless things (I mean useless for showing in the
> > *Help* buffer).

> > Isn't it possible to convert README.org to plain text README while
> > compiling the package, possibly optionally?  Changing package--compile
> > function should do the job.

> An alternative would be to enable (parts of) org-mode in the *Help*
> buffer.  I've just tried `M-x describe-package RET corfu RET` which
> comes with a README.org and then enabling org-mode in the *Help* buffer.
> That looked really nice and didn't seem to cause bad effects except that
> the [back] / [forward] buttons stopped working.

No, no, no, no!  Org mode is a highly complicated, obscure mode which is
NOT part of core Emacs, and mustn't become so.  What you're proposing is
a slippery slope, where ever greater portions of org mode would get
pushed into the core, causing ever greater problems for those who do not
use org mode.

You pointed out one such problem yourself, org mode key bindings will
take up key binding space currently used by other modes and by users.

Surely the solution has got to be to encourage package authors to write
plain text (or .texi) documentation, by pointing out the difficulties
the non-standard .org format creates.

> Bye,
> Tassilo

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Don't panic, Alan! 😁

The idea would be to only activate the font-lock rules of org (or markdown-mode, if available) while somehow protecting the package headers and help-mode footer section (i.e., the buttons).

Bye,
Tassilo

04.06.2022 16:18:21 Alan Mackenzie <acm@muc.de>:

> Hello, Tassilo.
> 
> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>> Akib Azmain Turja <akib@disroot.org> writes:
> 
>> Hi Akib,
> 
>>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>>> description both with describe-package and web browser.  But after
>>> installing a package, describe-package shows the Org source code,
>>> which contains many useless things (I mean useless for showing in the
>>> *Help* buffer).
> 
>>> Isn't it possible to convert README.org to plain text README while
>>> compiling the package, possibly optionally?  Changing package--compile
>>> function should do the job.
> 
>> An alternative would be to enable (parts of) org-mode in the *Help*
>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>> That looked really nice and didn't seem to cause bad effects except that
>> the [back] / [forward] buttons stopped working.
> 
> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.  What you're proposing is
> a slippery slope, where ever greater portions of org mode would get
> pushed into the core, causing ever greater problems for those who do not
> use org mode.
> 
> You pointed out one such problem yourself, org mode key bindings will
> take up key binding space currently used by other modes and by users.
> 
> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.
> 
>> Bye,
>> Tassilo
> 

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Don't panic, Alan! 😁
>
> The idea would be to only activate the font-lock rules of org (or markdown-mode, if available) while somehow protecting the package headers and help-mode footer section (i.e., the buttons).

But I don't like that idea.  I don't think that's a really good idea,
since the *Help* buffer is not for only describe-package, its for all
purposes.  I think converting Org (or Markdown or whatever) to plain
text should solve most (if not all) of the problems.  Then showing that
file shouldn't need any special treatment at all.

>
> Bye,
> Tassilo
>
> 04.06.2022 16:18:21 Alan Mackenzie <acm@muc.de>:
>
>> Hello, Tassilo.
>> 
>> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>>> Akib Azmain Turja <akib@disroot.org> writes:
>> 
>>> Hi Akib,
>> 
>>>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>>>> description both with describe-package and web browser.  But after
>>>> installing a package, describe-package shows the Org source code,
>>>> which contains many useless things (I mean useless for showing in the
>>>> *Help* buffer).
>> 
>>>> Isn't it possible to convert README.org to plain text README while
>>>> compiling the package, possibly optionally?  Changing package--compile
>>>> function should do the job.
>> 
>>> An alternative would be to enable (parts of) org-mode in the *Help*
>>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>>> That looked really nice and didn't seem to cause bad effects except that
>>> the [back] / [forward] buttons stopped working.
>> 
>> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
>> NOT part of core Emacs, and mustn't become so.  What you're proposing is
>> a slippery slope, where ever greater portions of org mode would get
>> pushed into the core, causing ever greater problems for those who do not
>> use org mode.
>> 
>> You pointed out one such problem yourself, org mode key bindings will
>> take up key binding space currently used by other modes and by users.
>> 
>> Surely the solution has got to be to encourage package authors to write
>> plain text (or .texi) documentation, by pointing out the difficulties
>> the non-standard .org format creates.
>> 
>>> Bye,
>>> Tassilo
>> 

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Stefan Kangas]

Alan Mackenzie <acm@muc.de> writes:

> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.

AFAICT, Org-mode has been distributed with Emacs for more than five
major releases by now (approx. 15 years).

> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.

In the Emacs community, Org-mode is ubiquitous.  I think we should
actively encourage using it for most types of documentation, certainly
for README type files.

Texinfo has its strengths of course, but it is unfortunately perceived
as arcane and hard to use by many.  And Org-mode can be converted to
be read in 'M-x info'.

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jun 04, 2022 at 18:09:45 +0200, Stefan Kangas wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > No, no, no, no!  Org mode is a highly complicated, obscure mode which
> > is NOT part of core Emacs, and mustn't become so.

> AFAICT, Org-mode has been distributed with Emacs for more than five
> major releases by now (approx. 15 years).

That's a strawman.  Org mode is not part of the Emacs core.  It's not
something in the part of Emacs inevitably used by any normal proficient
user.  Neither is Gnus.  Neither is CC Mode, for that matter.  By
contrast *Help*, Info, font-lock, the package manager, a lot of other
things, and a vast selection of commands bound to keys in the global key
map, are the core.

> > Surely the solution has got to be to encourage package authors to
> > write plain text (or .texi) documentation, by pointing out the
> > difficulties the non-standard .org format creates.

> In the Emacs community, Org-mode is ubiquitous.

Only in the sense that it's distributed with each Emacs release.

> I think we should actively encourage using it for most types of
> documentation, certainly for README type files.

I disagree entirely with you.  Org mode is highly complicated, obscure
(I've never managed to get a feel for what it does), and difficult to
learn (I've never managed it).  A text file is far easier to read for
those not familiar with org.  We're talking about READMEs here.  They're
typically 20 to 100 lines long.  A text file is ideal for these.

> Texinfo has its strengths of course, but it is unfortunately perceived
> as arcane and hard to use by many.

It's moderately easy to use, somewhat difficult to learn.

> And Org-mode can be converted to be read in 'M-x info'.

You mean it _has_ to be converted to be read?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Stefan Kangas]

Alan Mackenzie <acm@muc.de> writes:

> We're talking about READMEs here.  They're
> typically 20 to 100 lines long.  A text file is ideal for these.

A text file is fine of course, but they lack useful features such as
including images (e.g. screenshots).  You would need some markup for
that, and Org-mode happens to have solid built-in support.

> > Texinfo has its strengths of course, but it is unfortunately perceived
> > as arcane and hard to use by many.
>
> It's moderately easy to use, somewhat difficult to learn.

Agreed.

> > And Org-mode can be converted to be read in 'M-x info'.
>
> You mean it _has_ to be converted to be read?

I don't think so, no.  I'm just saying that you can do what we do with
the org manual, so you get to write Org-mode, but still get to read in
'M-x info'.

Re: Convert README.org to plain text README while installing package

[Michael Albinus]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Stefan.

Hi,

>> I think we should actively encourage using it for most types of
>> documentation, certainly for README type files.
>
> I disagree entirely with you.  Org mode is highly complicated, obscure
> (I've never managed to get a feel for what it does), and difficult to
> learn (I've never managed it).  A text file is far easier to read for
> those not familiar with org.  We're talking about READMEs here.  They're
> typically 20 to 100 lines long.  A text file is ideal for these.

1+. README is a text file which is consulted in order to get information
about the related software. There's no rule to visit it by Emacs only.

Best regards, Michael.

Re: Convert README.org to plain text README while installing package

[Po Lu]

Michael Albinus <michael.albinus@gmx.de> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Hello, Stefan.
>
> Hi,
>
>>> I think we should actively encourage using it for most types of
>>> documentation, certainly for README type files.
>>
>> I disagree entirely with you.  Org mode is highly complicated, obscure
>> (I've never managed to get a feel for what it does), and difficult to
>> learn (I've never managed it).  A text file is far easier to read for
>> those not familiar with org.  We're talking about READMEs here.  They're
>> typically 20 to 100 lines long.  A text file is ideal for these.
>
> 1+. README is a text file which is consulted in order to get information
> about the related software. There's no rule to visit it by Emacs only.
>
> Best regards, Michael.

Hear, hear!

It also takes a relatively long time to load Org mode, so opening a
README.org-type file for the first time will always cause an annoying
pause.

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Po Lu <luangruo@yahoo.com> writes:

>>>> I think we should actively encourage using it for most types of
>>>> documentation, certainly for README type files.
>>>
>>> I disagree entirely with you.  Org mode is highly complicated,
>>> obscure (I've never managed to get a feel for what it does), and
>>> difficult to learn (I've never managed it).  A text file is far
>>> easier to read for those not familiar with org.  We're talking about
>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>> is ideal for these.
>>
>> 1+. README is a text file which is consulted in order to get
>> information about the related software. There's no rule to visit it
>> by Emacs only.

People nowadays write README.org and README.md files instead of
plain-text READMEs because the packages' repositories are hosted on some
forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
those formats in a nice way.  So the choice of format is natural for
package authors.  And they frequently contain markup like headings,
bullet lists, bold, italics, and code snippets which simply look much
better than their text-only alternative (even in their
editing/non-rendered versions).

> Hear, hear!
>
> It also takes a relatively long time to load Org mode, so opening a
> README.org-type file for the first time will always cause an annoying
> pause.

That's true but wouldn't it be possible to extract the font-lock/faces
related parts in a separate file with no dependencies which can be
loaded quickly enough?  I mean, the purpose would only be to display org
and markdown READMEs with the highlighting intended by the package
author but there's no point in enabling the gazillon of other features
(from editing to time-tracking to spreadsheet computations) those
packages might provide in addition.

Bye,
Tassilo

Re: Convert README.org to plain text README while installing package

[Michael Albinus]

Tassilo Horn <tsdh@gnu.org> writes:

Hi Tassilo,

>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).

Not everybody uses Github/Gitlab/<you name it> infrastructure
exclusively. There are dinosaurs like me, who simply inspect tarballs
and alike.

I have no problem if there are structured README.org or README.md files
in parallel. But a README file should be plain text.

> Bye,
> Tassilo

Best regards, Michael.

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> I have no problem if there are structured README.org or README.md files
> in parallel. But a README file should be plain text.

This.  Not just either/or.

Plain-text README should be a minimal requirement.

Other formats are nice-to-have, for some contexts
& some users.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Michael Albinus <michael.albinus@gmx.de> writes:

>
> I have no problem if there are structured README.org or README.md files
> in parallel. But a README file should be plain text.
>

I've seen this mentioned multiple times in this thread and it doesn't
make sense to me. 

Org files *are* plain text. This is one of (perhaps the biggest) selling
points for org mode. They don't use any form of 'binary' data and can be
read just fine in any text editor or just using cat/less/more whatever.
They may look a little *ugly*, especially with respect to URLs, but are
still quite readable - a lot more readable than other plain text formats
such as xml or html or json etc.

I also find arguments based around org being complex and difficult to
learn to be somewhat overstated. Org is powerful and very configurable,
which means there can be a lot to learn if you want to leverage off all
it has to offer. However, like emacs, the basics are very simple and
easy to learn. 

While I'm not arguing that org should be forced upon everyone and I
would agree we need to keep potential load time issues in mind, there
seems to be lots in this thread over stating the issues and jumping to
extremes. All that seems to really be under consideration is to enable
rendering of *org files in help buffers using org font locking and
perhaps enabling folding, which could be very beneficial for large
readme files and would not matter for small ones. I also suspect this is
something which could be disabled with a simple variable setting for
those who really don't like it. 

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Tim Cross <theophilusx@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>>
>> I have no problem if there are structured README.org or README.md files
>> in parallel. But a README file should be plain text.
>>
>
> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me. 
>
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.
> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

I have consult installed, and if I want to read the package summary
generated by C-h P, I half of the buffer is just Org configuration:

--8<---------------cut here---------------start------------->8---
#+title: consult.el - Consulting completing-read
#+author: Daniel Mendler
#+language: en
#+export_file_name: consult.texi
#+texinfo_dir_category: Emacs misc features
#+texinfo_dir_title: Consult: (consult).
#+texinfo_dir_desc: Useful commands built on completing-read.

#+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
#+html: <a href="http://elpa.gnu.org/packages/consult.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
#+html: <a href="http://elpa.gnu.org/devel/consult.html"><img alt="GNU-devel ELPA" src="https://elpa.gnu.org/devel/consult.svg"/></a>
#+html: <a href="https://melpa.org/#/consult"><img alt="MELPA" src="https://melpa.org/packages/consult-badge.svg"/></a>
#+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA Stable" src="https://stable.melpa.org/packages/consult-badge.svg"/></a>

* Introduction
:properties:
:description: Why Consult?
:end:
#+cindex: introduction
--8<---------------cut here---------------end--------------->8---

While plain text, it is not what I am interested in at this point.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Philip Kaludercic <philipk@posteo.net> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>>
>>> I have no problem if there are structured README.org or README.md files
>>> in parallel. But a README file should be plain text.
>>>
>>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> I have consult installed, and if I want to read the package summary
> generated by C-h P, I half of the buffer is just Org configuration:
>
> #+title: consult.el - Consulting completing-read
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: consult.texi
> #+texinfo_dir_category: Emacs misc features
> #+texinfo_dir_title: Consult: (consult).
> #+texinfo_dir_desc: Useful commands built on completing-read.
>
> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
> #+html: <a href="http://elpa.gnu.org/packages/consult.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
> #+html: <a href="http://elpa.gnu.org/devel/consult.html"><img alt="GNU-devel ELPA" src="https://elpa.gnu.org/devel/consult.svg"/></a>
> #+html: <a href="https://melpa.org/#/consult"><img alt="MELPA" src="https://melpa.org/packages/consult-badge.svg"/></a>
> #+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA Stable" src="https://stable.melpa.org/packages/consult-badge.svg"/></a>
>
> * Introduction
> :properties:
> :description: Why Consult?
> :end:
> #+cindex: introduction
>
> While plain text, it is not what I am interested in at this point.

and in what seems to be typical in this thread, you have selected an
extreme example to demonstrate your point. This is definitely not a
typical README.org file - at least not from my experience. 

This is also a completely pointless debate as Stefan has already
indicated auto translation to ascii will likely be what is done anyway. 

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Tim Cross <theophilusx@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> Michael Albinus <michael.albinus@gmx.de> writes:
>>>
>>>>
>>>> I have no problem if there are structured README.org or README.md files
>>>> in parallel. But a README file should be plain text.
>>>>
>>>
>>> I've seen this mentioned multiple times in this thread and it doesn't
>>> make sense to me. 
>>>
>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>> points for org mode. They don't use any form of 'binary' data and can be
>>> read just fine in any text editor or just using cat/less/more whatever.
>>> They may look a little *ugly*, especially with respect to URLs, but are
>>> still quite readable - a lot more readable than other plain text formats
>>> such as xml or html or json etc.
>>
>> I have consult installed, and if I want to read the package summary
>> generated by C-h P, I half of the buffer is just Org configuration:
>>
>> #+title: consult.el - Consulting completing-read
>> #+author: Daniel Mendler
>> #+language: en
>> #+export_file_name: consult.texi
>> #+texinfo_dir_category: Emacs misc features
>> #+texinfo_dir_title: Consult: (consult).
>> #+texinfo_dir_desc: Useful commands built on completing-read.
>>
>> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU
>> Emacs"
>> src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
>> #+html: <a href="http://elpa.gnu.org/packages/consult.html"><img
>> alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
>> #+html: <a href="http://elpa.gnu.org/devel/consult.html"><img
>> alt="GNU-devel ELPA"
>> src="https://elpa.gnu.org/devel/consult.svg"/></a>
>> #+html: <a href="https://melpa.org/#/consult"><img alt="MELPA"
>> src="https://melpa.org/packages/consult-badge.svg"/></a>
>> #+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA
>> Stable"
>> src="https://stable.melpa.org/packages/consult-badge.svg"/></a>
>>
>> * Introduction
>> :properties:
>> :description: Why Consult?
>> :end:
>> #+cindex: introduction
>>
>> While plain text, it is not what I am interested in at this point.
>
> and in what seems to be typical in this thread, you have selected an
> extreme example to demonstrate your point. This is definitely not a
> typical README.org file - at least not from my experience. 

While true, I don't get why this is a problem.  I don't think making the
package developers simplify their READMEs would a better solution, since
they are also frequently used to generate manuals.  

As to whether or not the package description should be the same as the
manual, is a different question (I am inclined to think the package
description should be a brief summary of a few paragraphs at most).

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:

> Michael Albinus <michael.albinus@gmx.de> writes:


> > I have no problem if there are structured README.org or README.md
> > files in parallel. But a README file should be plain text.

I agree with this.

> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me. 

> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.

We're reduced to arguing about the meaning of "plain text".  The way I
see it, plain text means to be read as is by the user.  In that sense,
the formats you mention below, xml, html, etc. are _not_ plain text -
they're designed for machine processing.  A typical spam email in HTML
has the human readable pieces swamped by the machine readable pieces.  I
think you're arguing that this isn't the case with org mode files, though
Philip Kaludercic pointed out an org file where this wasn't the case.

> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

And their performance in standard programs like grep, wc, etc. is that
much worse than plain text.

> I also find arguments based around org being complex and difficult to
> learn to be somewhat overstated.

There are 784 key bindings in org mode.  How can you say that this isn't
complex and difficult to learn?

> Org is powerful and very configurable, ....

That contradicts your previous statement to some extent.  Any program
which is very configurable _needs_ to be configured.

> .... which means there can be a lot to learn if you want to leverage
> off all it has to offer.

I don't want to do this.  I want to avoid org mode being loaded into my
Emacs; it fouls up my key bindings to a significant extent.  Particularly
if I just want to read a 50 line README.

> However, like emacs, the basics are very simple and easy to learn. 

I don't agree that the basics of Emacs are simple and easy to learn at
all.  That's a strong reason why other editors have become popular.  Why
should I have to spend time learning a super-complicated mode just to
read a 50 line README?

> While I'm not arguing that org should be forced upon everyone ....

If a README file is README.org, then org mode is forced upon anybody
wishing to read it in Emacs.  This is why README files should be plain
text.

> .... and I would agree we need to keep potential load time issues in
> mind, there seems to be lots in this thread over stating the issues and
> jumping to extremes.

I think org mode is an extreme, with its 784 key bindings which seem to
violate written and unwritten conventions.

> All that seems to really be under consideration is to enable rendering
> of *org files in help buffers using org font locking and perhaps
> enabling folding, which could be very beneficial for large readme files
> and would not matter for small ones. I also suspect this is something
> which could be disabled with a simple variable setting for those who
> really don't like it. 

"It" could best be avoided by having plain text README files.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 3871 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tim.
>
> On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>
>> > I have no problem if there are structured README.org or README.md
>> > files in parallel. But a README file should be plain text.
>
> I agree with this.
>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>
> We're reduced to arguing about the meaning of "plain text".  The way I
> see it, plain text means to be read as is by the user.  In that sense,
> the formats you mention below, xml, html, etc. are _not_ plain text -
> they're designed for machine processing.  A typical spam email in HTML
> has the human readable pieces swamped by the machine readable pieces.  I
> think you're arguing that this isn't the case with org mode files, though
> Philip Kaludercic pointed out an org file where this wasn't the case.
>
>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> And their performance in standard programs like grep, wc, etc. is that
> much worse than plain text.
>
>> I also find arguments based around org being complex and difficult to
>> learn to be somewhat overstated.
>
> There are 784 key bindings in org mode.  How can you say that this isn't
> complex and difficult to learn?
>
>> Org is powerful and very configurable, ....
>
> That contradicts your previous statement to some extent.  Any program
> which is very configurable _needs_ to be configured.
>
>> .... which means there can be a lot to learn if you want to leverage
>> off all it has to offer.
>
> I don't want to do this.  I want to avoid org mode being loaded into my
> Emacs; it fouls up my key bindings to a significant extent.  Particularly
> if I just want to read a 50 line README.
>
>> However, like emacs, the basics are very simple and easy to learn. 
>
> I don't agree that the basics of Emacs are simple and easy to learn at
> all.  That's a strong reason why other editors have become popular.  Why
> should I have to spend time learning a super-complicated mode just to
> read a 50 line README?
>
>> While I'm not arguing that org should be forced upon everyone ....
>
> If a README file is README.org, then org mode is forced upon anybody
> wishing to read it in Emacs.  This is why README files should be plain
> text.
>
>> .... and I would agree we need to keep potential load time issues in
>> mind, there seems to be lots in this thread over stating the issues and
>> jumping to extremes.
>
> I think org mode is an extreme, with its 784 key bindings which seem to
> violate written and unwritten conventions.
>
>> All that seems to really be under consideration is to enable rendering
>> of *org files in help buffers using org font locking and perhaps
>> enabling folding, which could be very beneficial for large readme files
>> and would not matter for small ones. I also suspect this is something
>> which could be disabled with a simple variable setting for those who
>> really don't like it. 
>
> "It" could best be avoided by having plain text README files.
>
> -- 
> Alan Mackenzie (Nuremberg, Germany).
>

Just to add, many of those 784 keybindings of Org are DWIM or
context-dependent, which makes it even harder to learn.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tim.
>
> On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>
>> > I have no problem if there are structured README.org or README.md
>> > files in parallel. But a README file should be plain text.
>
> I agree with this.
>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>
> We're reduced to arguing about the meaning of "plain text".  The way I
> see it, plain text means to be read as is by the user.  In that sense,
> the formats you mention below, xml, html, etc. are _not_ plain text -
> they're designed for machine processing.  A typical spam email in HTML
> has the human readable pieces swamped by the machine readable pieces.  I
> think you're arguing that this isn't the case with org mode files, though
> Philip Kaludercic pointed out an org file where this wasn't the case.
>

Philip's example is an extreme case and not representative of normal
README.org files. 

>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> And their performance in standard programs like grep, wc, etc. is that
> much worse than plain text.


No, I disagree with this completely. Org mode markup is very simple and
rarely has any impact with regards to grep, ag or any other 'shell' tool
It is precisely why I consider it to be a plain text format  - all of
these standard plain trext processing tools work in the main just as
well as they do on any ASCII file. Again, org mode is not like XML or
HTML or other formats which do make such processing difficult. 
>
>> I also find arguments based around org being complex and difficult to
>> learn to be somewhat overstated.
>
> There are 784 key bindings in org mode.  How can you say that this isn't
> complex and difficult to learn?
>
Very simply because the vast majority of  those key bindings only come
into play when yhou use advanced features of org mode, such as the
agenda or table editing or noweb mode. If your just using basic org mode
features, very very few of those key bindings even come into play. Just
go throgh that list and remove any bindings relating to agenda mode,
table editing mode, source block modes, clock table mode, list editing
mode, etc and you end up with very few key bindings (all of which are
udner the C-c prefix, unlike your previous claim). 

>> Org is powerful and very configurable, ....
>
> That contradicts your previous statement to some extent.  Any program
> which is very configurable _needs_ to be configured.
>

ABsolute nonsense. Just because you can configure somethig doesn't
automatically mean you have to configure it. Emacs is extremely
configurable, but you can use it jsut fine without doing any
configuration at all. 

>> .... which means there can be a lot to learn if you want to leverage
>> off all it has to offer.
>
> I don't want to do this.  I want to avoid org mode being loaded into my
> Emacs; it fouls up my key bindings to a significant extent.  Particularly
> if I just want to read a 50 line README.
>

Other posts have already explained this is NOT what is being proposed
and that it would not affect your key bindings in the slightest. All
that is being propsoed here is that a read only buffer will use org mode
to format/display the content. Very simple and not the monster you
insist. 


>> However, like emacs, the basics are very simple and easy to learn. 
>
> I don't agree that the basics of Emacs are simple and easy to learn at
> all.  That's a strong reason why other editors have become popular.  Why
> should I have to spend time learning a super-complicated mode just to
> read a 50 line README?
>

Well, basically, you don't. That has already bene explained and does not
need to be reitterated here. However, to be clear, all that is being
proposed is that org formatting is applied to the contgents of this
read-only buffer. There will not be huge numbers of
unwanted/unexptrected key bindings and there won't be a huge number of
other changes. You will likely have some additional key bindings which
may make navigating larger README files easier and that is about it. All
that is really being proposed here is org font locking and outline mode
navigation. All very simple really. 

>> While I'm not arguing that org should be forced upon everyone ....
>
> If a README file is README.org, then org mode is forced upon anybody
> wishing to read it in Emacs.  This is why README files should be plain
> text.
>
>> .... and I would agree we need to keep potential load time issues in
>> mind, there seems to be lots in this thread over stating the issues and
>> jumping to extremes.
>
> I think org mode is an extreme, with its 784 key bindings which seem to
> violate written and unwritten conventions.


That 750 key bindings is an extreme over statement and not what is being
proposed. Yet again, people jumping to extremes based on ignorance and
paranoia. Spend the time to go through the key bindings listed in the
org manual and you will soon realise that the majority of those bindings
only come into effect in specialised mdoes - none of which are relevant
to a READM.org ile (for example, all the key bindings relating to agenda
mode). 

I find it odd that someone can on one hand argue so hard against using a
mode based on the complexity and vast nubmer of key bindings and at the
same time admit they have never spent the time to learn the mode. This
seems like an arguument based on fear and uncertainty about something
you don't know rather thaa one based on fact. Perhaps look more closely
at what was actually being proposed (which was not a full org mode
situation) and spend enough time to at least udnerstand the basics
before condemnation. 
>
>> All that seems to really be under consideration is to enable rendering
>> of *org files in help buffers using org font locking and perhaps
>> enabling folding, which could be very beneficial for large readme files
>> and would not matter for small ones. I also suspect this is something
>> which could be disabled with a simple variable setting for those who
>> really don't like it. 
>
> "It" could best be avoided by having plain text README files.

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Jun 2022 23:57:55 +1000
> 
> > There are 784 key bindings in org mode.  How can you say that this isn't
> > complex and difficult to learn?
> >
> Very simply because the vast majority of  those key bindings only come
> into play when yhou use advanced features of org mode, such as the
> agenda or table editing or noweb mode. If your just using basic org mode
> features, very very few of those key bindings even come into play. Just
> go throgh that list and remove any bindings relating to agenda mode,
> table editing mode, source block modes, clock table mode, list editing
> mode, etc and you end up with very few key bindings (all of which are
> udner the C-c prefix, unlike your previous claim). 

How about if you do this exercise and walk us through the results?
Visit an Org file, type "C-h b", then show us the list, and tell which
bindings you think are "basic" and which are "advanced", and why you
think so.

I can tell you what I see with my very simple 1570-line Org file,
which just has some todo items organized in 4-level hierarchy:

  . ~100 keys that look "general Org" to me
  . ~30 keys that Org remaps to its s own commands, like 'yank' and
    'backward-sentence'
  . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
    bound by default, but they are there
  . ~60 more keys that I'm not sure whether they are "basic" or not,
    but they are all bound by default, just because I visited an Org
    file 

So "just" 230 key bindings.  And this is without any minor/add-on Org
mode/feature enabled, at least according to "C-h m".

Do you see something different?  Are you still saying that it's not a
lot, or that it's "based on ignorance and paranoia"?  If so, please
point out where I'm ignorant and/or paranoiac, because I'd really like
to know.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 06 Jun 2022 23:57:55 +1000
>> 
>> > There are 784 key bindings in org mode.  How can you say that this isn't
>> > complex and difficult to learn?
>> >
>> Very simply because the vast majority of  those key bindings only come
>> into play when yhou use advanced features of org mode, such as the
>> agenda or table editing or noweb mode. If your just using basic org mode
>> features, very very few of those key bindings even come into play. Just
>> go throgh that list and remove any bindings relating to agenda mode,
>> table editing mode, source block modes, clock table mode, list editing
>> mode, etc and you end up with very few key bindings (all of which are
>> udner the C-c prefix, unlike your previous claim). 
>
> How about if you do this exercise and walk us through the results?
> Visit an Org file, type "C-h b", then show us the list, and tell which
> bindings you think are "basic" and which are "advanced", and why you
> think so.
>
> I can tell you what I see with my very simple 1570-line Org file,
> which just has some todo items organized in 4-level hierarchy:
>
>   . ~100 keys that look "general Org" to me
>   . ~30 keys that Org remaps to its s own commands, like 'yank' and
>     'backward-sentence'
>   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
>     bound by default, but they are there
>   . ~60 more keys that I'm not sure whether they are "basic" or not,
>     but they are all bound by default, just because I visited an Org
>     file 
>
> So "just" 230 key bindings.  And this is without any minor/add-on Org
> mode/feature enabled, at least according to "C-h m".
>
> Do you see something different?  Are you still saying that it's not a
> lot, or that it's "based on ignorance and paranoia"?  If so, please
> point out where I'm ignorant and/or paranoiac, because I'd really like
> to know.

I think Alan's response has pretty much confirmed what I was saying. Alan has
made it fairly clear that his real underlying concern is about a stealthy
strategy to get org mode more intrinsically tied into Emacs and in the end,
making it so critical to normal operation that everyone will be forced to learn
org mode regardless. I don't necessarily agree with such an argument, but think
having a debate around that would be more up-front and in some ways honest
compared to what appear to be somewhat contrived alternative arguments about
excessive key bindings and added complexity. My characterisation of this all
being fear and paranoia may have been overstating things, but then again I'm not
sure. There certainly does seem to be considerable fear involved. I don't know
about paranoia, but I have not seen any evil plan to have org mode assimilate
Emacs as if it was the Borg and I've seen no discussions on the org list about
ways to get org more ingrained in Emacs or make Emacs more dependent on org.

With respect to ignorance, Alan made it quite clear he has not spent
the time or effort to learn org mode, so he is ignorant about its
operation and lacks any real understanding of its operation. He has made
assertions about the number of key bindings which are wildly over
inflated and about required levels of complexity which simply would not
be necessary with the org mode rendering of a read only buffer. In
short, he has made a lot of assumptions about both the key bindings and
complexity based on only a fairly cursory scan of the manual and very
limited experience. 

All that appears to have occurred here is that someone noted that packages often
had a readme.org file and wondered if this should be translated into just a
plain readme file. Someone else suggested we could have package descriptions
render the readme.org file using org and not require export into plain readme
file and mentioned some possible advantages this could have wrt navigation,
rendering of tables, links and images or inclusion of source code snippets,
examples etc.

From this point, some somewhat extreme claims and arguments have been thrown in.
One was Alan's claim that org would introduce 785 new key bindings, many of
which not bounmd to C-c and which would presumably either pollute the key space
or conflict with existing bindings. All I have done is point out this claim was
inaccurate and overstated due to an overly simplistic analysis of what is in the
key index of the org manual. 

it should be noted that if we were to add 'native' rendering of readme.org
files, this would likely be done with an org minor mode rather than a full blown
org mode as very little of the overall org functionality would be required in
this situation. All that would be required is org rendering support and org
support for navigation (folding/unfolding, following links and possibly
list/table navigation). All the advanced org functionality relating to babel,
noweb, todo and time management, agendas, etc have no relevance in this
scenario. LIkewise, if this functionality was provided via a minor mode, it
would also be possible for people to disable that mode and be left with just the
readme.org file, which is just plain text and quite readable.

Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Running Emacs wiht -q and opeing up an org file, I find the following 

- 236 org related key bindings - far less than 785 Of these, a number are
- actually remapping of existing bindings, so not new ones. This leaves 208.

I said that for a read only buffer, many of the org key bindings are not
relevant as they relate to features which are not pertinent to a read only org
buffer. Any bindings relating to babel, todo management, time management,
agendas etc have no relevance when reading a readme.org file. Really, the only
key bindings of any relevance are navigation related bindings (including
opening/closing folds, following links, etc). Removing all unnecessary org
bindings leave us with just 77 new bindings. Of these remaining bindings, quite
a few could also be removed, but I've left them in as I only wanted to remove
those which would clearly be of no benefit in a read only buffer rendering of a
readme.org file. In an org minor mode setup for rendering of readme.org (and
possibly other read-only rendering of org files), I estimate at least another 30
bindings could be removed because they have no real benefit or are of only
marginal benefit. Creating an org minor mode for this purpose which only
introduced 20 or so new bindings would easily be possible. The key point is that
claims of 785 new bindings arfe incorrect and misleading. Even being very
conservative, we are really talking about 10% of the number being incorrectly claimed.  
if we introduced a minor mode for this, we are talking likely less than 20 or so.

Arguments regarding this being a 'slippery slope' or 'edge of the wedge' in an
org takeover are a completely different argument and there may well be some
concerns which may need to be considered. However, if that is the real concern,
then argue based on that rather than exaggerated claims about excessive key
bindings, key binding polution/conflicts and excessive complexity.

In a similar manner, lets not jump to conclusions or throw in wild speculation
about things which have not been proposed. For example, nobody has suggested
everyone would be required to provide readme.org files, nobody has suggested you
could not just have a readme or a readme.txt file, nobody has suggested making
anyone provide any specific format. All that has been rasied is whether, when a
package has a readme.org file, should it be translated into a 'plain text'
(ASCII/UTF-8) format or whether it might be useful to have the file formatted
and nicely displayed using existing org-mode formatting capabilities. It could
even be that the ability to have 'native' rendering of readme.org files could be
n optional feature which those who want it can enable.

All of this now seems to be irrelevant as Stefan has indicated he will
just automatically convert the readme.org to a plain readme. I hope that
the original readme.org file will still be included for those who do
prefer to use org to render and navigate a well formatted readme file
which includes working and concise links, source code syntax
highlighting, well formatted examples and fast navigation. 

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 16:14:52 +1000
> 
> > I can tell you what I see with my very simple 1570-line Org file,
> > which just has some todo items organized in 4-level hierarchy:
> >
> >   . ~100 keys that look "general Org" to me
> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
> >     'backward-sentence'
> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
> >     bound by default, but they are there
> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
> >     but they are all bound by default, just because I visited an Org
> >     file 
> >
> > So "just" 230 key bindings.  And this is without any minor/add-on Org
> > mode/feature enabled, at least according to "C-h m".
> >
> > Do you see something different?  Are you still saying that it's not a
> > lot, or that it's "based on ignorance and paranoia"?  If so, please
> > point out where I'm ignorant and/or paranoiac, because I'd really like
> > to know.
> 
> I think Alan's response has pretty much confirmed what I was saying. Alan has
> made it fairly clear that his real underlying concern is about a stealthy
> strategy to get org mode more intrinsically tied into Emacs and in the end,
> making it so critical to normal operation that everyone will be forced to learn
> org mode regardless. I don't necessarily agree with such an argument, but think
> having a debate around that would be more up-front and in some ways honest
> compared to what appear to be somewhat contrived alternative arguments about
> excessive key bindings and added complexity. My characterisation of this all
> being fear and paranoia may have been overstating things, but then again I'm not
> sure. There certainly does seem to be considerable fear involved. I don't know
> about paranoia, but I have not seen any evil plan to have org mode assimilate
> Emacs as if it was the Borg and I've seen no discussions on the org list about
> ways to get org more ingrained in Emacs or make Emacs more dependent on org.
> 
> With respect to ignorance, Alan made it quite clear he has not spent
> the time or effort to learn org mode, so he is ignorant about its
> operation and lacks any real understanding of its operation. He has made
> assertions about the number of key bindings which are wildly over
> inflated and about required levels of complexity which simply would not
> be necessary with the org mode rendering of a read only buffer. In
> short, he has made a lot of assumptions about both the key bindings and
> complexity based on only a fairly cursory scan of the manual and very
> limited experience. 
> 
> All that appears to have occurred here is that someone noted that packages often
> had a readme.org file and wondered if this should be translated into just a
> plain readme file. Someone else suggested we could have package descriptions
> render the readme.org file using org and not require export into plain readme
> file and mentioned some possible advantages this could have wrt navigation,
> rendering of tables, links and images or inclusion of source code snippets,
> examples etc.
> 
> >From this point, some somewhat extreme claims and arguments have been thrown in.
> One was Alan's claim that org would introduce 785 new key bindings, many of
> which not bounmd to C-c and which would presumably either pollute the key space
> or conflict with existing bindings. All I have done is point out this claim was
> inaccurate and overstated due to an overly simplistic analysis of what is in the
> key index of the org manual. 
> 
> it should be noted that if we were to add 'native' rendering of readme.org
> files, this would likely be done with an org minor mode rather than a full blown
> org mode as very little of the overall org functionality would be required in
> this situation. All that would be required is org rendering support and org
> support for navigation (folding/unfolding, following links and possibly
> list/table navigation). All the advanced org functionality relating to babel,
> noweb, todo and time management, agendas, etc have no relevance in this
> scenario. LIkewise, if this functionality was provided via a minor mode, it
> would also be possible for people to disable that mode and be left with just the
> readme.org file, which is just plain text and quite readable.
> 
> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

The above is completely irrelevant to my questions.

> Running Emacs wiht -q and opeing up an org file, I find the following 
> 
> - 236 org related key bindings - far less than 785 Of these, a number are
> - actually remapping of existing bindings, so not new ones. This leaves 208.

??? How is remapping "not new"?  It takes some very common Emacs
commands and redirects them to different commands.  E.g., 'open-line'
now does something quite different.  This means the user should either
go learn what the Org commands do, or be prepared to be surprised.

> I said that for a read only buffer, many of the org key bindings are not
> relevant as they relate to features which are not pertinent to a read only org
> buffer. Any bindings relating to babel, todo management, time management,
> agendas etc have no relevance when reading a readme.org file.

Then why does Org define them in that case?

> Really, the only
> key bindings of any relevance are navigation related bindings (including
> opening/closing folds, following links, etc). Removing all unnecessary org
> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
> a few could also be removed, but I've left them in as I only wanted to remove
> those which would clearly be of no benefit in a read only buffer rendering of a
> readme.org file. In an org minor mode setup for rendering of readme.org (and
> possibly other read-only rendering of org files), I estimate at least another 30
> bindings could be removed because they have no real benefit or are of only
> marginal benefit. Creating an org minor mode for this purpose which only
> introduced 20 or so new bindings would easily be possible. The key point is that
> claims of 785 new bindings arfe incorrect and misleading. Even being very
> conservative, we are really talking about 10% of the number being incorrectly claimed.  
> if we introduced a minor mode for this, we are talking likely less than 20 or so.

I'm sorry, but IMO this hand-waves away a real problem by ignoring the
problematic UX of suddenly having more than 200 keybindings defined
just because I visited a small file.  I hoped you will present some
serious analysis of the issue, and perhaps even show me where I'm
wrong in my conclusions.  Sadly, it sounds like all you wanted to do
is to prove that you are right, the facts notwithstanding.

You accuse Alan in extremism, but your own argumentation is another
example of a similar extremism -- just in the opposite direction.

This doesn't facilitate useful discussions of what I think is a real
problem.  Any unbiased observer should agree that adding 200+ key
bindings when a small file is visited is a problem that needs to be
solved.  Org is too large for such simple jobs, and should be
refactored to avoid loading everything under the sun when a README is
visited.  And we should work towards solving it, not sweep it under
the carpet.

P.S. Btw, here's one more demonstration that Org loads too much by
default:

  C-x C-f ~/org/Plan.org RET
   => "Package vc-mtn is deprecated"

Please believe me that NOTHING in my file refers or requires mtn.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> - 236 org related key bindings - far less than 785 Of these, a number are
>> - actually remapping of existing bindings, so not new ones. This leaves 208.
>
> ??? How is remapping "not new"?  It takes some very common Emacs
> commands and redirects them to different commands.  E.g., 'open-line'
> now does something quite different.  This means the user should either
> go learn what the Org commands do, or be prepared to be surprised.

There are 3 main purposes of remappings in Org:
1. To implement some optional functionality. That functionality is not
   enabled by default and no surprises will be given to user.
2. To implement the original functionality where the existing Emacs
   customization is not sufficient. For example, org-backward-paragraph
   has to re-implement backward-paragraph as the original function only
   supports regexp-bases paragraph boundaries. Regexps are not
   sufficient in Org syntax.
3. Like (1), but deliberately changing the default behavior to what Org
   users prefer historically. This is usually not much different from
   customisations that are usually made in define-derived-mode, except
   that Emacs does not have enough flexibility to do it there.

>> I said that for a read only buffer, many of the org key bindings are not
>> relevant as they relate to features which are not pertinent to a read only org
>> buffer. Any bindings relating to babel, todo management, time management,
>> agendas etc have no relevance when reading a readme.org file.
>
> Then why does Org define them in that case?

I am not very sure what is the problem with the number of bindings. It
is not different from Emacs itself.

For example, I am sometimes puzzled by <F2> prefix key in Emacs defining
bindings for two-column.el. Over years of using Emacs I only used them
by accident and always felt like the "break" my Emacs.

Yet, I am fine with those hundreds of default Emacs bindings I never
use. I do not have to read the full Emacs manual to understand those
bindings (they are even not all described in the Emacs manual!).

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 21:12:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> - 236 org related key bindings - far less than 785 Of these, a number are
> >> - actually remapping of existing bindings, so not new ones. This leaves 208.
> >
> > ??? How is remapping "not new"?  It takes some very common Emacs
> > commands and redirects them to different commands.  E.g., 'open-line'
> > now does something quite different.  This means the user should either
> > go learn what the Org commands do, or be prepared to be surprised.
> 
> There are 3 main purposes of remappings in Org:

I'm sure there are good reasons for that.  My point is that such
remappings effectively force the user to re-learn the commands he/she
is very familiar with.  So it's a non-trivial burden.

> >> I said that for a read only buffer, many of the org key bindings are not
> >> relevant as they relate to features which are not pertinent to a read only org
> >> buffer. Any bindings relating to babel, todo management, time management,
> >> agendas etc have no relevance when reading a readme.org file.
> >
> > Then why does Org define them in that case?
> 
> I am not very sure what is the problem with the number of bindings. It
> is not different from Emacs itself.

The difference is that we had years or decades to get used to the
Emacs defaults, and once Org is turned on in a buffer, one has a lot
of new stuff to get used to.  Unless Org is used constantly, you will
forget most of those changes till the next time, so this re-learning
experience will be repeated every time.

It isn't a catastrophe, of course, but we should recognize this as an
issue, especially if many of the bindings aren't needed.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Wed, 08 Jun 2022 21:12:11 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> - 236 org related key bindings - far less than 785 Of these, a number are
>> >> - actually remapping of existing bindings, so not new ones. This leaves 208.
>> >
>> > ??? How is remapping "not new"?  It takes some very common Emacs
>> > commands and redirects them to different commands.  E.g., 'open-line'
>> > now does something quite different.  This means the user should either
>> > go learn what the Org commands do, or be prepared to be surprised.
>> 
>> There are 3 main purposes of remappings in Org:
>
> I'm sure there are good reasons for that.  My point is that such
> remappings effectively force the user to re-learn the commands he/she
> is very familiar with.  So it's a non-trivial burden.

Not really. The remappings usually intend to re-implement the usual
expected Emacs behavior inside Org. It's just that it is not always
possible using the built-in functions. Hence, we implement a layer on
top.

There should not be anything to learn with regard to remapped commands
given that Org defaults are not changed.

The difference is that things like, say, paragraph-separate, are not
even noticed by users of various major modes. Org had to remap the
commands, which is more immediately visible.

>> >> I said that for a read only buffer, many of the org key bindings are not
>> >> relevant as they relate to features which are not pertinent to a read only org
>> >> buffer. Any bindings relating to babel, todo management, time management,
>> >> agendas etc have no relevance when reading a readme.org file.
>> >
>> > Then why does Org define them in that case?
>> 
>> I am not very sure what is the problem with the number of bindings. It
>> is not different from Emacs itself.
>
> The difference is that we had years or decades to get used to the
> Emacs defaults, and once Org is turned on in a buffer, one has a lot
> of new stuff to get used to.  Unless Org is used constantly, you will
> forget most of those changes till the next time, so this re-learning
> experience will be repeated every time.

Isn't it the same for any other major mode?

> It isn't a catastrophe, of course, but we should recognize this as an
> issue, especially if many of the bindings aren't needed.

I am not sure what you mean by aren't needed. They are needed if you use
their functionality. They are not needed if you don't.

There is no doubt that you do not need most of the bindings just to
navigate Org files or do basic editing. You do not need to learn those
other bindings either.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 23:15:16 +0800
> 
> >> There are 3 main purposes of remappings in Org:
> >
> > I'm sure there are good reasons for that.  My point is that such
> > remappings effectively force the user to re-learn the commands he/she
> > is very familiar with.  So it's a non-trivial burden.
> 
> Not really. The remappings usually intend to re-implement the usual
> expected Emacs behavior inside Org. It's just that it is not always
> possible using the built-in functions. Hence, we implement a layer on
> top.
> 
> There should not be anything to learn with regard to remapped commands
> given that Org defaults are not changed.

Are you sure this is always true?  There are several dozens of
remapped commands; did you audit all of them?

And anyway, even if what you say is 110% true, how am I as a user to
know that up front?  I'm used to read the documentation of every
command I don't already know by heart, so when faced with such massive
remapping, I have quite some reading to do before I can feel myself at
ease.

And please note that, unlike Alan, I _do_ use Org, just not very
often, at least these days.  So what I'm sating doesn't come from the
POV of an anti-Org user.  I _want_ Org to be easier and less demanding
to use.

> > The difference is that we had years or decades to get used to the
> > Emacs defaults, and once Org is turned on in a buffer, one has a lot
> > of new stuff to get used to.  Unless Org is used constantly, you will
> > forget most of those changes till the next time, so this re-learning
> > experience will be repeated every time.
> 
> Isn't it the same for any other major mode?

No, not IME.  Show me another general-purpose editing mode that
defines so many key bindings.  The only modes that get close are those
where text is read-only, so normal editing is impossible anyway.  And
even those leave alone basic movement commands, like C-S-<up> which
Alan mentioned.

> > It isn't a catastrophe, of course, but we should recognize this as an
> > issue, especially if many of the bindings aren't needed.
> 
> I am not sure what you mean by aren't needed.

Ask Tim Cross: the claim that most of the 230 bindings I counted
aren't needed comes from him.

> There is no doubt that you do not need most of the bindings just to
> navigate Org files or do basic editing. You do not need to learn those
> other bindings either.

Then why bind them by default? why not wait until I actually need that
functionality?  If that's hard or impossible to detect automatically,
let the user say so.  For example, I could envision a minor mode that
enables Org-Babel and binds the corresponding commands to keys.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not really. The remappings usually intend to re-implement the usual
>> expected Emacs behavior inside Org. It's just that it is not always
>> possible using the built-in functions. Hence, we implement a layer on
>> top.
>> 
>> There should not be anything to learn with regard to remapped commands
>> given that Org defaults are not changed.
>
> Are you sure this is always true?  There are several dozens of
> remapped commands; did you audit all of them?

I have audited a couple of them and relied on a good faith to other Org
devs + absence of bug reports relevant to the raised concern.
Rather then re-checking the remapped functions, I'd rather just fix any
concrete bug reports about misbehavior, if such reports arise.

> And anyway, even if what you say is 110% true, how am I as a user to
> know that up front?  I'm used to read the documentation of every
> command I don't already know by heart, so when faced with such massive
> remapping, I have quite some reading to do before I can feel myself at
> ease.
>
> And please note that, unlike Alan, I _do_ use Org, just not very
> often, at least these days.  So what I'm sating doesn't come from the
> POV of an anti-Org user.  I _want_ Org to be easier and less demanding
> to use.

While I understand the possible fear of unexpected or unnatural
behavior of the remapped command, I'd like to point out that command
behavior can span over quite a significant range even without remapping.
Just the fact that some major mode does not use remapping, does not mean
that you are guaranteed to get the normal command behavior.

Some major modes rely on post-command-hook instead of remapping, which
not only modifies the original command behavior in unspecified ways, but
is also not directly visible from the mode bindings. Or consider changes
in filter-buffer-substring-function and how it can modify killing text.

So, I'd say that your concern is not really justified. There is
generally no reliable way to tell if an arbitrary major mode modifies
standard command behavior just by looking at its key bindings. You have
to rely on faith and report bugs if you stumble upon them.

>> > The difference is that we had years or decades to get used to the
>> > Emacs defaults, and once Org is turned on in a buffer, one has a lot
>> > of new stuff to get used to.  Unless Org is used constantly, you will
>> > forget most of those changes till the next time, so this re-learning
>> > experience will be repeated every time.
>> 
>> Isn't it the same for any other major mode?
>
> No, not IME.  Show me another general-purpose editing mode that
> defines so many key bindings.  The only modes that get close are those
> where text is read-only, so normal editing is impossible anyway.

I am not sure what you mean by general-purpose. Auctex binds a decent
number of key-bindings. VHDL mode binds even more (I just looked into
one of the largest mode built-in prog modes). markdown-mode binds over
100 keys not counting prefix maps.

>   And even those leave alone basic movement commands, like C-S-<up>
> which Alan mentioned.

C-S-<up> is not a basic movement command. By default, it is activating
region. Without shift-select-mode, it is not bound to anything. Org does
provide support for shift-select-mode, but we never got overwhelming
requests to enable this support by default.

C-<up>/<down>, which is the actual basic movement command is not
overridden by Org.

AFAIK, the only shadowed command is M-<right>/<left> with its other
C-<right>/<left> binding being untouched. However, I am not even sure
what Org can do about this. Arrow bindings are one of the most core and
frequently used commands. Removing them will dissatisfy existing users.

Note that Emacs dedicates C-c + letter for user bindings. Anything else
is not guaranteed to be not shadowed by major/minor modes. If users
absolutely want they personal bindings to be preferred, there is always
override-global-map.

>> > It isn't a catastrophe, of course, but we should recognize this as an
>> > issue, especially if many of the bindings aren't needed.
>> 
>> I am not sure what you mean by aren't needed.
>
> Ask Tim Cross: the claim that most of the 230 bindings I counted
> aren't needed comes from him.

He was referring to reading Org files without editing.
You can argue the same for reading pretty much any other major-mode
files. Most bindings will not be needed.

>> There is no doubt that you do not need most of the bindings just to
>> navigate Org files or do basic editing. You do not need to learn those
>> other bindings either.
>
> Then why bind them by default? why not wait until I actually need that
> functionality?  If that's hard or impossible to detect automatically,
> let the user say so.  For example, I could envision a minor mode that
> enables Org-Babel and binds the corresponding commands to keys.

Org-babel bindings are needed as soon as you create a source block
inside the Org file. Are you proposing to track changes in buffer and
autoload babel as soon as a source block appears inside the buffer? I'd
say that conventional approach is binding the relevant commands and
letting Emacs to do autoloading once those commands are called.

Would it help if we hide the babel-related stuff under dedicated prefix
map?

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 09 Jun 2022 17:15:05 +0800
> 
> > Are you sure this is always true?  There are several dozens of
> > remapped commands; did you audit all of them?
> 
> I have audited a couple of them and relied on a good faith to other Org
> devs + absence of bug reports relevant to the raised concern.
> Rather then re-checking the remapped functions, I'd rather just fix any
> concrete bug reports about misbehavior, if such reports arise.

Didn't this discussion raise such "bug reports"?  Why not consider
what several people here said as such a bug report?  The difference is
that these reports come from people who use Org rarely, so the
concerns are different.  But if the Org developers want to have Org
adopted more widely, I think they should listen to these concerns, not
reject them.

> While I understand the possible fear of unexpected or unnatural
> behavior of the remapped command, I'd like to point out that command
> behavior can span over quite a significant range even without remapping.
> Just the fact that some major mode does not use remapping, does not mean
> that you are guaranteed to get the normal command behavior.

Most major modes don't remap global and frequently-used commands.
Those that do are problematic, and should be fixed rather than be used
as examples of what's "kosher".

> Some major modes rely on post-command-hook instead of remapping, which
> not only modifies the original command behavior in unspecified ways, but
> is also not directly visible from the mode bindings. Or consider changes
> in filter-buffer-substring-function and how it can modify killing text.

And that is in your opinion a Good Thing?  I don't think it is, and
therefore don't consider such problematic behavior a good example for
development.

> So, I'd say that your concern is not really justified.

??? Just because "others do that"?

> There is generally no reliable way to tell if an arbitrary major
> mode modifies standard command behavior just by looking at its key
> bindings. You have to rely on faith and report bugs if you stumble
> upon them.

I don't see how this solves the concerns.  We should still try to
minimize such problematic behaviors as much as possible.

> > No, not IME.  Show me another general-purpose editing mode that
> > defines so many key bindings.  The only modes that get close are those
> > where text is read-only, so normal editing is impossible anyway.
> 
> I am not sure what you mean by general-purpose.

Modes for editing mostly human-readable text.

> Auctex binds a decent number of key-bindings. VHDL mode binds even
> more (I just looked into one of the largest mode built-in prog
> modes). markdown-mode binds over 100 keys not counting prefix maps.

Two of these aren't part of Emacs, and VHDL is an example of
programming language, where text is not really for human consumption.

> >   And even those leave alone basic movement commands, like C-S-<up>
> > which Alan mentioned.
> 
> C-S-<up> is not a basic movement command.

Of course, it is: it moves by paragraphs.

> By default, it is activating
> region. Without shift-select-mode, it is not bound to anything.

But shift-select-mode is on by default.

> > Then why bind them by default? why not wait until I actually need that
> > functionality?  If that's hard or impossible to detect automatically,
> > let the user say so.  For example, I could envision a minor mode that
> > enables Org-Babel and binds the corresponding commands to keys.
> 
> Org-babel bindings are needed as soon as you create a source block
> inside the Org file. Are you proposing to track changes in buffer and
> autoload babel as soon as a source block appears inside the buffer?

That's one way.  Another one is to have a minor mode that users could
turn on when they need it (or turn it on by default in their init
files if they need it a lot).

> Would it help if we hide the babel-related stuff under dedicated prefix
> map?

Probably.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Thu, 09 Jun 2022 17:15:05 +0800
>> 
>> > Are you sure this is always true?  There are several dozens of
>> > remapped commands; did you audit all of them?
>> 
>> I have audited a couple of them and relied on a good faith to other Org
>> devs + absence of bug reports relevant to the raised concern.
>> Rather then re-checking the remapped functions, I'd rather just fix any
>> concrete bug reports about misbehavior, if such reports arise.
>
> Didn't this discussion raise such "bug reports"?  Why not consider
> what several people here said as such a bug report?  The difference is
> that these reports come from people who use Org rarely, so the
> concerns are different.  But if the Org developers want to have Org
> adopted more widely, I think they should listen to these concerns, not
> reject them.

To clarify, it is metally difficult to process bug reports raised in
such discussions because they are often entangled with quite negative
attitude and much more broad claims.

What I have summarised so far from the discussions:

1. Org mode shadows default C-S-<up>/C-S-<down> bindings.

   This can be solved in two ways:
   a. Change org-support-shift-select to t by default + fix the fact
      that org-support-shift-select is actually ignored by
      C-S-<up>/<down>.
   b. Disable Org mode arrow bindings by default and move them to a
      dedicated minor-mode / setting.
      (This will make existing Org users angry)

   Note that I do not see C-c / and C-c C-s as a bug. The first one is
   not recommended for major-mode but allowed. C-c C-s is dedicated to
   major modes.

2. Org mode binds too many keys

   The solution can be disabling those keys and moving them to dedicated
   minor modes / settings.

   After thinking, I do not like the idea of enabling some keys
   depending on the presence absense of tables/source blocks inside the
   Org file. This will not solve the issue in the files actually
   containing all those and may surprise users.

3. Some of the context-dependent Org command names are confusing as they
   do not reveal what the command does.

   The solution is to rename those commands to something more
   meaningful. Totally doable.
      
4. Org remaps some of the default commands.
   I can see how it can be confusing, but do not see it as a critical
   issue. See my response below.

5. Org mode is too slow to load.
   I do not see an easy way to resolve this apart from general
   refactoring and modularizing. Profiling is not very helpful in this
   specific case, unfortunately. At least, my profiling skills are not
   good enough.

>> While I understand the possible fear of unexpected or unnatural
>> behavior of the remapped command, I'd like to point out that command
>> behavior can span over quite a significant range even without remapping.
>> Just the fact that some major mode does not use remapping, does not mean
>> that you are guaranteed to get the normal command behavior.
>
> Most major modes don't remap global and frequently-used commands.
> Those that do are problematic, and should be fixed rather than be used
> as examples of what's "kosher".

Let's take one example: org-forward-paragraph that is remapping
forward-paragraph.

The docstring is:

Move forward by a paragraph, or equivalent, unit.
...
The function moves point between two structural
elements (paragraphs, tables, lists, etc.).

It also provides the following special moves for convenience:

  - on a table or a property drawer, move to its beginning;
  - on comment, example, export, source and verse blocks, stop
    at blank lines;
  - skip consecutive clocks, diary S-exps, and keywords.

In Org mode, there is no single definition of a paragraph. The
paragraphs or equivalent syntax elements are context-depended and we
have to use parser to know where to move.

The default forward-paragraph can only be controlled by paragraph-start
and paragraph-separate regexps. Both are not sufficient to cater Org
mode. Regexps do not cut it for Org mode.

Hence, the built-in version of forward-paragraph is not usable in Org
mode. If we did not remap it, forward-paragraph would not really move by
paragraph, as it is understood by human. It would be more confusing than
org version of the command.

If you know a better way to resolve the described limitation, please let
me know.

Similar limitations motivated
<remap> <backward-paragraph>	org-backward-paragraph
<remap> <backward-sentence>	org-backward-sentence
<remap> <comment-dwim>		org-comment-dwim
<remap> <forward-paragraph>	org-forward-paragraph
<remap> <forward-sentence>	org-forward-sentence
<remap> <move-beginning-of-line> org-beginning-of-line
<remap> <move-end-of-line>	org-end-of-line
<remap> <delete-indentation>	org-delete-indentation

The following functions have a natural fallback to default behavior and
might be moved to minor-modes enabled by default. However, their default
behavior is context dependent and motivated by the lack of flexibility
of the built-in equivalents. Similar to the above functions.

<remap> <open-line>		org-open-line
<remap> <move-beginning-of-line> org-beginning-of-line
<remap> <move-end-of-line>	org-end-of-line

The following functions might be implemented using built-in Emacs
functionality:

<remap> <transpose-words>	org-transpose-words
could use find-word-boundary-function-table

<remap> <fill-paragraph>	org-fill-paragraph
appears to be redundant as we do set fill-paragraph-function


<remap> <yank>			org-yank
may be instead implemented using yank-handler property (it takes care
about adjusting level of the inserted headlines), but not completely. We
cannot control properties of text from outside of Org mode and thus the
functionality cannot be fully implemented using built-in Emacs features

The following commands are providing convenience self-alignment
functionality and protect from accidental editing of invisible text
(AFAIK, this issue is not addressed by Emacs):
<remap> <kill-line>		org-kill-line
<remap> <delete-backward-char>	org-delete-backward-char
<remap> <delete-char>		org-delete-char
<remap> <self-insert-command>	org-self-insert-command

I am not sure how to achieve equivalent functionality using built-in
Emacs customisations.

The following functions are convenience building for prople coming from
outline mode. I do not see any problem here.

<remap> <outline-backward-same-level> org-backward-heading-same-level
<remap> <outline-demote>	org-demote-subtree
<remap> <outline-forward-same-level> org-forward-heading-same-level
<remap> <outline-insert-heading> org-ctrl-c-ret
<remap> <outline-mark-subtree>	org-mark-subtree
<remap> <outline-next-visible-heading> org-next-visible-heading
<remap> <outline-previous-visible-heading> org-previous-visible-heading
<remap> <outline-promote>	org-promote-subtree
<remap> <outline-show-branches> org-kill-note-or-show-branches
<remap> <outline-show-children> org-fold-show-children
<remap> <outline-show-subtree>	org-fold-show-subtree

>> Some major modes rely on post-command-hook instead of remapping, which
>> not only modifies the original command behavior in unspecified ways, but
>> is also not directly visible from the mode bindings. Or consider changes
>> in filter-buffer-substring-function and how it can modify killing text.
>
> And that is in your opinion a Good Thing?  I don't think it is, and
> therefore don't consider such problematic behavior a good example for
> development.

Maybe. But then we need a more "appropriate" way to implement the
functionality Org provides. See the above.

>> So, I'd say that your concern is not really justified.
>
> ??? Just because "others do that"?

>> There is generally no reliable way to tell if an arbitrary major
>> mode modifies standard command behavior just by looking at its key
>> bindings. You have to rely on faith and report bugs if you stumble
>> upon them.
>
> I don't see how this solves the concerns.  We should still try to
> minimize such problematic behaviors as much as possible.

I'd like you to focus on the last example with
filter-buffer-substring-function or on the ability for major mode to
customise fill-paragraph-function or paragraph-separate, etc. Those
changes can alter the default commands drastically. Yet, they will not
be noticeable by users from looking at the M-x describe-bindings.

I do not say that major modes should change the default behavior in
unexpected ways. Rather the opposite - major modes should adjust the
default commands to behave more expectedly within the major mode
context. However, remapping is by no means an indication that a major
mode is going to change things unexpectedly. Its neither sufficient nor
required condition.

>> > No, not IME.  Show me another general-purpose editing mode that
>> > defines so many key bindings.  The only modes that get close are those
>> > where text is read-only, so normal editing is impossible anyway.
>> ...
>> Auctex binds a decent number of key-bindings. VHDL mode binds even
>> more (I just looked into one of the largest mode built-in prog
>> modes). markdown-mode binds over 100 keys not counting prefix maps.
>
> Two of these aren't part of Emacs, and VHDL is an example of
> programming language, where text is not really for human consumption.

I get your point on VHDL. However, I am not sure why you reject the
non-built-in examples. Are you implying that text modes outside Emacs
core are worse than built-in? At least Auctex is largely popular among
users and yet I have never seen as many complaints about Auctex
complexity as I do for Org (I may be biased).

Auctex and markdown-mode deal with similar editing tasks with Org mode.
The fact that they also use large number of key bindings is an
indication that they are really needed for editing complex documents.

>> >   And even those leave alone basic movement commands, like C-S-<up>
>> > which Alan mentioned.
>> 
>> C-S-<up> is not a basic movement command.
>
> Of course, it is: it moves by paragraphs.

Nope. It is unbound command working because of key translation:

C-<down> (translated from C-S-<down>) runs the command
forward-paragraph (found in global-map), which is an interactive
native compiled Lisp function in ‘paragraphs.el’.

>> By default, it is activating
>> region. Without shift-select-mode, it is not bound to anything.
>
> But shift-select-mode is on by default.

Fair point.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 11:52:05 +0800
> 
> > Didn't this discussion raise such "bug reports"?  Why not consider
> > what several people here said as such a bug report?  The difference is
> > that these reports come from people who use Org rarely, so the
> > concerns are different.  But if the Org developers want to have Org
> > adopted more widely, I think they should listen to these concerns, not
> > reject them.
> 
> To clarify, it is metally difficult to process bug reports raised in
> such discussions because they are often entangled with quite negative
> attitude and much more broad claims.
> 
> What I have summarised so far from the discussions:
> 
> 1. Org mode shadows default C-S-<up>/C-S-<down> bindings.
> 
>    This can be solved in two ways:
>    a. Change org-support-shift-select to t by default + fix the fact
>       that org-support-shift-select is actually ignored by
>       C-S-<up>/<down>.
>    b. Disable Org mode arrow bindings by default and move them to a
>       dedicated minor-mode / setting.
>       (This will make existing Org users angry)
> 
>    Note that I do not see C-c / and C-c C-s as a bug. The first one is
>    not recommended for major-mode but allowed. C-c C-s is dedicated to
>    major modes.
> 
> 2. Org mode binds too many keys
> 
>    The solution can be disabling those keys and moving them to dedicated
>    minor modes / settings.
> 
>    After thinking, I do not like the idea of enabling some keys
>    depending on the presence absense of tables/source blocks inside the
>    Org file. This will not solve the issue in the files actually
>    containing all those and may surprise users.
> 
> 3. Some of the context-dependent Org command names are confusing as they
>    do not reveal what the command does.
> 
>    The solution is to rename those commands to something more
>    meaningful. Totally doable.
>       
> 4. Org remaps some of the default commands.
>    I can see how it can be confusing, but do not see it as a critical
>    issue. See my response below.
> 
> 5. Org mode is too slow to load.
>    I do not see an easy way to resolve this apart from general
>    refactoring and modularizing. Profiling is not very helpful in this
>    specific case, unfortunately. At least, my profiling skills are not
>    good enough.

Thank you, I think this is a very good summary.  I hope some of these
issues could be worked on and improved in the future.

> > Most major modes don't remap global and frequently-used commands.
> > Those that do are problematic, and should be fixed rather than be used
> > as examples of what's "kosher".
> 
> Let's take one example: org-forward-paragraph that is remapping
> forward-paragraph.
> 
> The docstring is:
> 
> Move forward by a paragraph, or equivalent, unit.
> ...
> The function moves point between two structural
> elements (paragraphs, tables, lists, etc.).
> 
> It also provides the following special moves for convenience:
> 
>   - on a table or a property drawer, move to its beginning;
>   - on comment, example, export, source and verse blocks, stop
>     at blank lines;
>   - skip consecutive clocks, diary S-exps, and keywords.
> 
> In Org mode, there is no single definition of a paragraph. The
> paragraphs or equivalent syntax elements are context-depended and we
> have to use parser to know where to move.
> 
> The default forward-paragraph can only be controlled by paragraph-start
> and paragraph-separate regexps. Both are not sufficient to cater Org
> mode. Regexps do not cut it for Org mode.
> 
> Hence, the built-in version of forward-paragraph is not usable in Org
> mode. If we did not remap it, forward-paragraph would not really move by
> paragraph, as it is understood by human. It would be more confusing than
> org version of the command.

For a serious discussion of this example, I'd need more detail about
the aspects you mentioned above (it is not trivial to deduce them from
looking at the top levels of org-forward-paragraph).  Specifically:

  . how does paragraph definition depend on context in Org, and what
    are the possible definitions in the existing contexts?
  . why don't paragraph-start and paragraph-separate fit Org, and can
    we come up with a small number of additional variables that would
    augment these two so that the built-in forward-paragraph could be
    extended to cover Org?

> If you know a better way to resolve the described limitation, please let
> me know.

The better way, IMO, is this:

  . try to find a way of extending the built-in forward-paragraph to
    cover the Org use cases, by adding variables in addition to the 2
    existing ones (I don't yet understand why "regexps do not cut it
    for Org mode -- it sounds like a very strong assertion)
  . if the above doesn't work, make forward-paragraph work through
    forward-paragraph-function, so that Org could define its own
    function
  . in any case, the add-on convenience features of
    org-forward-paragraph should be provided as options that the user
    can control

My main concern is that forward-paragraph is used when editing the
purely textual parts of an Org document, and when used there, casual
users of Org will expect it to behave as the command behaves in plain
text.  Any deviation from the behavior of the built-in command will
confuse such users when they edit the plain-text parts, and should
therefore be avoided.

> The following functions have a natural fallback to default behavior and
> might be moved to minor-modes enabled by default. However, their default
> behavior is context dependent and motivated by the lack of flexibility
> of the built-in equivalents. Similar to the above functions.
> 
> <remap> <open-line>		org-open-line
> <remap> <move-beginning-of-line> org-beginning-of-line
> <remap> <move-end-of-line>	org-end-of-line

Each of these (and other examples you provided) should require a
separate serious discussion.  Let's take open-line as an example.  It
is basically the built-in open-line, unless org-special-ctrl-o is
non-nil.  A trivial extension of the built-in command could have
prevented the need to remap.  (And why does the support for tables in
org-open-line need to be turned on outside of orgtbl-mode?)

> <remap> <yank>			org-yank
> may be instead implemented using yank-handler property (it takes care
> about adjusting level of the inserted headlines), but not completely. We
> cannot control properties of text from outside of Org mode and thus the
> functionality cannot be fully implemented using built-in Emacs features

IMO, org-yank is sufficiently different to justify not taking over the
built-in command.  Again, consider a user who edits the plain-text
portions of an Org document and expects the "usual" behavior of C-y.

> >> Some major modes rely on post-command-hook instead of remapping, which
> >> not only modifies the original command behavior in unspecified ways, but
> >> is also not directly visible from the mode bindings. Or consider changes
> >> in filter-buffer-substring-function and how it can modify killing text.
> >
> > And that is in your opinion a Good Thing?  I don't think it is, and
> > therefore don't consider such problematic behavior a good example for
> > development.
> 
> Maybe. But then we need a more "appropriate" way to implement the
> functionality Org provides. See the above.

Each of these commands should be discussed separately, and the best
solution found and implemented.  There's no single answer to that
question for all of these remapped commands.  But remapping should be
avoided as much as possible; see below.

> > I don't see how this solves the concerns.  We should still try to
> > minimize such problematic behaviors as much as possible.
> 
> I'd like you to focus on the last example with
> filter-buffer-substring-function or on the ability for major mode to
> customise fill-paragraph-function or paragraph-separate, etc. Those
> changes can alter the default commands drastically. Yet, they will not
> be noticeable by users from looking at the M-x describe-bindings.

When fill-paragraph-function and other means of customizing a command
to the particular mode make sense, they will not need to be noticed,
because they will do what the users expect.  Moreover, customizing the
behavior of commands through those variables has a couple of important
advantages:

  . it doesn't use remapping, so it doesn't stick out and bother
    conscientious Emacs users, who are used to study unfamiliar
    commands before using them
  . they don't require separate documentation, apart of saying that
    paragraph-start and paragraph-separate can be modified by modes as
    appropriate -- all the rest of the command's documentation is
    still intact and easily grasped in the context of any mode (this
    is also one more reason why "niceties" like special behavior
    inside tables should be separately controlled)

> I do not say that major modes should change the default behavior in
> unexpected ways. Rather the opposite - major modes should adjust the
> default commands to behave more expectedly within the major mode
> context. However, remapping is by no means an indication that a major
> mode is going to change things unexpectedly. Its neither sufficient nor
> required condition.

My point is that there are better ways of customizing the behavior of
a built-in command.  Org is now an integral part of Emacs, so it
should implement these customizations in ways that are available to a
core package; remapping is basically a tool used by unbundled packages
that cannot affect the core in any way.

> >> Auctex binds a decent number of key-bindings. VHDL mode binds even
> >> more (I just looked into one of the largest mode built-in prog
> >> modes). markdown-mode binds over 100 keys not counting prefix maps.
> >
> > Two of these aren't part of Emacs, and VHDL is an example of
> > programming language, where text is not really for human consumption.
> 
> I get your point on VHDL. However, I am not sure why you reject the
> non-built-in examples. Are you implying that text modes outside Emacs
> core are worse than built-in?

No, I'm saying that, sadly, we have no real control on what the
developers of unbundled packages decide to do.  Thus, what you see
there is the evidence of our lack of control more than anything else.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Thank you, I think this is a very good summary.  I hope some of these
> issues could be worked on and improved in the future.

It is in my todo-list. I may find some time in future to work on it.

> For a serious discussion of this example, I'd need more detail about
> the aspects you mentioned above (it is not trivial to deduce them from
> looking at the top levels of org-forward-paragraph).  Specifically:
>
>   . how does paragraph definition depend on context in Org, and what
>     are the possible definitions in the existing contexts?
>   . why don't paragraph-start and paragraph-separate fit Org, and can
>     we come up with a small number of additional variables that would
>     augment these two so that the built-in forward-paragraph could be
>     extended to cover Org?

I do not want to start a serious discussion on this just yet as I do not
plant to work on this specific area in near future, however I would like
to answer some of your questions in order to provide some insight for
Emacs developers.

To answer your question about paragraph definition, we need to clarify
what "paragraph" means.

(*)
Plain text documents, in their simplest forms, usually consist of a
series of sections containing a series of paragraphs each. With
paragraphs being a series of sentences. The paragraphs are usually
separated by blank lines or by indentation.

Org mode extends the above simple structure by defining additional
syntactic elements, which are equivalent to paragraphs. Apart from
paragraphs, there can be tables, lists, other specially treated blocks
of text (source code blocks, comments, verse blocks, drawers, etc).
These elements may or may not be separated with blank lines. For
example,
| This table    | is considered as a new "paragraph"-like element |
| Not separated | from the above paragraph by blank lines         |
#+begin_src emacs-lisp
(message "And here we have yet another paragraph-like element")

;; This element may contain empty lines inside, yet these empty lines
;; are not indicating "paragraph"-like element boundary.

(message "They may, in emacs-lisp specifically, from some point of view;

but consider other possible programming languages with arbitrary syntax;
or even elisp, with the above blank line inside string.")
#+end_src

When a user calls forward-paragraph on a simple paragraph as defined in
(*), built-in forward-paragraph will behave narutally. However, things
become tricky when we have various paragraph-like elements. How can one
configure forward-paragraph to forward classic paragraph, tables, source
blocks, and all other possible elements?

To make things more complex, Org mode syntax is not context-free.
#+end_src
in here does not belong to a source block because it lacks #+begin_src
opener.
Also,
:DRAWER-LIKE-THIS:
Cannot intersect with other source-block-looking constructs like
#+begin_src emacs-lisp
(message "I am not a source block")
:END:
The
#+end_src
here is also not a part of source block because it cannot intersect
drawer boundary.

>> If you know a better way to resolve the described limitation, please let
>> me know.
>
> The better way, IMO, is this:
>
>   . try to find a way of extending the built-in forward-paragraph to
>     cover the Org use cases, by adding variables in addition to the 2
>     existing ones (I don't yet understand why "regexps do not cut it
>     for Org mode -- it sounds like a very strong assertion)

For regexps, I hope that the above example illustrated the problem
clearly. Also, note that a number of people attempted to develop
BNF/EBNF parsers for Org mode, but failed because of issues tike the
above. Org is not context-free and cannot be parsed using LR parsers,
let alone simple regexps.

For more insight, you may also consult org-element-paragraph-parser
code, which has to deal with the problems illustrated above. (note that
even this function alone is not sufficient to find paragraphs; it also
relies on other parser code).

>   . if the above doesn't work, make forward-paragraph work through
>     forward-paragraph-function, so that Org could define its own
>     function

Introducing the required flexibility into Emacs core is certainly an
option. Though it is somewhat difficult one because Org also has to
support older versions of Emacs. We are somewhat limited in using newly
introduced built-in features.

>   . in any case, the add-on convenience features of
>     org-forward-paragraph should be provided as options that the user
>     can control

They are not necessarily add-ons. First of all, org-forward-paragraph is
dealing with the above complications. Using forward-paragraph directly
would behave unexpectedly on various Org elements.

> My main concern is that forward-paragraph is used when editing the
> purely textual parts of an Org document, and when used there, casual
> users of Org will expect it to behave as the command behaves in plain
> text.  Any deviation from the behavior of the built-in command will
> confuse such users when they edit the plain-text parts, and should
> therefore be avoided.

There is no difference between forward-paragraph and
org-forward-paragraph on purely texture parts. The difference is only on
the Org-specific constructs, where forward-paragraph would behave
unexpectedly.

>> The following functions have a natural fallback to default behavior and
>> might be moved to minor-modes enabled by default. However, their default
>> behavior is context dependent and motivated by the lack of flexibility
>> of the built-in equivalents. Similar to the above functions.
>> 
>> <remap> <open-line>		org-open-line
>> <remap> <move-beginning-of-line> org-beginning-of-line
>> <remap> <move-end-of-line>	org-end-of-line
>
> Each of these (and other examples you provided) should require a
> separate serious discussion.  Let's take open-line as an example.  It
> is basically the built-in open-line, unless org-special-ctrl-o is
> non-nil.  A trivial extension of the built-in command could have
> prevented the need to remap.  (And why does the support for tables in
> org-open-line need to be turned on outside of orgtbl-mode?)

You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
provided for users who want to edit tables "like in Org mode", but in
_other_ major modes. Inside Org mode, tables are always supported.

Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
on Org tables can produce incorrect formatting. For example, calling
open-line on the following table

| this | is        | table |
| with | <point> 2 | lines |

will produce

| this | is | table |
| with | 
 2 | lines |

while org-open-line will produce

| this | is | table |
|      |    |       |
| with |  2 | lines |

The default behavior is not satisfactory and somewhat unexpected.

>> <remap> <yank>			org-yank
>> may be instead implemented using yank-handler property (it takes care
>> about adjusting level of the inserted headlines), but not completely. We
>> cannot control properties of text from outside of Org mode and thus the
>> functionality cannot be fully implemented using built-in Emacs features
>
> IMO, org-yank is sufficiently different to justify not taking over the
> built-in command.  Again, consider a user who edits the plain-text
> portions of an Org document and expects the "usual" behavior of C-y.

I personally tend to agree here.

> When fill-paragraph-function and other means of customizing a command
> to the particular mode make sense, they will not need to be noticed,
> because they will do what the users expect.  Moreover, customizing the
> behavior of commands through those variables has a couple of important
> advantages:
>
>   . it doesn't use remapping, so it doesn't stick out and bother
>     conscientious Emacs users, who are used to study unfamiliar
>     commands before using them
>   . they don't require separate documentation, apart of saying that
>     paragraph-start and paragraph-separate can be modified by modes as
>     appropriate -- all the rest of the command's documentation is
>     still intact and easily grasped in the context of any mode (this
>     is also one more reason why "niceties" like special behavior
>     inside tables should be separately controlled)

I get your point. As long as default customisations are sufficient to
achive the expected and reasonable behavior, they should indeed be
preferred.

>> I get your point on VHDL. However, I am not sure why you reject the
>> non-built-in examples. Are you implying that text modes outside Emacs
>> core are worse than built-in?
>
> No, I'm saying that, sadly, we have no real control on what the
> developers of unbundled packages decide to do.  Thus, what you see
> there is the evidence of our lack of control more than anything else.

I do not get your point here. I was referring to the markdown-mode and
Auctex to illustrate the _need_ to have numerous key bindings in order
to edit complex Org documents. It is not just something introduced into
Org out of the blue. Other people solved similar problems and did not
come up with significantly less key bindings. If you say that usual
40-50 major mode bindings are sufficient, I am not convinced.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 16:40:31 +0800
> 
> > For a serious discussion of this example, I'd need more detail about
> > the aspects you mentioned above (it is not trivial to deduce them from
> > looking at the top levels of org-forward-paragraph).  Specifically:
> >
> >   . how does paragraph definition depend on context in Org, and what
> >     are the possible definitions in the existing contexts?
> >   . why don't paragraph-start and paragraph-separate fit Org, and can
> >     we come up with a small number of additional variables that would
> >     augment these two so that the built-in forward-paragraph could be
> >     extended to cover Org?
> 
> I do not want to start a serious discussion on this just yet as I do not
> plant to work on this specific area in near future, however I would like
> to answer some of your questions in order to provide some insight for
> Emacs developers.

Thanks, but if we are not going to continue this discussion until we
come to some conclusions, I see no reason to keep talking about it.  I
do understand better the difficulties now (thanks!), but I'm not yet
convinced that the existing solution is the best one.

> > My main concern is that forward-paragraph is used when editing the
> > purely textual parts of an Org document, and when used there, casual
> > users of Org will expect it to behave as the command behaves in plain
> > text.  Any deviation from the behavior of the built-in command will
> > confuse such users when they edit the plain-text parts, and should
> > therefore be avoided.
> 
> There is no difference between forward-paragraph and
> org-forward-paragraph on purely texture parts. The difference is only on
> the Org-specific constructs, where forward-paragraph would behave
> unexpectedly.

How does Org make sure that the different behavior doesn't happen
without the user's intent?  Isn't it possible that Org perceives
something as a sign of an "Org-specific construct" when the user
didn't mean that?  When that happens, what can the user do to avoid
the unintended (from that user's POV) Org-specific behavior?

These situations are where user control is necessary, especially for
casual, non-seasoned users of Org.

> >> <remap> <open-line>		org-open-line
> >> <remap> <move-beginning-of-line> org-beginning-of-line
> >> <remap> <move-end-of-line>	org-end-of-line
> >
> > Each of these (and other examples you provided) should require a
> > separate serious discussion.  Let's take open-line as an example.  It
> > is basically the built-in open-line, unless org-special-ctrl-o is
> > non-nil.  A trivial extension of the built-in command could have
> > prevented the need to remap.  (And why does the support for tables in
> > org-open-line need to be turned on outside of orgtbl-mode?)
> 
> You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
> provided for users who want to edit tables "like in Org mode", but in
> _other_ major modes. Inside Org mode, tables are always supported.

What if the user doesn't intend to use Org tables?  How can such a
user turn off this automatic support, which interprets some patterns
of buffer text as an indication of a table, and turns on the special
behavior reserved for Org tables?

> Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
> on Org tables can produce incorrect formatting. For example, calling
> open-line on the following table
> 
> | this | is        | table |
> | with | <point> 2 | lines |
> 
> will produce
> 
> | this | is | table |
> | with | 
>  2 | lines |
> 
> while org-open-line will produce
> 
> | this | is | table |
> |      |    |       |
> | with |  2 | lines |
> 
> The default behavior is not satisfactory and somewhat unexpected.

You assume that users always expect what Org does in that case.  This
is not a given, if we want to make Org more attractive for editing
text that is "less Org-specific", so to speak, like README files, NEWS
files, etc.

> > No, I'm saying that, sadly, we have no real control on what the
> > developers of unbundled packages decide to do.  Thus, what you see
> > there is the evidence of our lack of control more than anything else.
> 
> I do not get your point here. I was referring to the markdown-mode and
> Auctex to illustrate the _need_ to have numerous key bindings in order
> to edit complex Org documents. It is not just something introduced into
> Org out of the blue. Other people solved similar problems and did not
> come up with significantly less key bindings. If you say that usual
> 40-50 major mode bindings are sufficient, I am not convinced.

I'm not arguing with the need, I'm arguing with the particular
implementation that caters to that need.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I do not want to start a serious discussion on this just yet as I do not
>> plant to work on this specific area in near future, however I would like
>> to answer some of your questions in order to provide some insight for
>> Emacs developers.
>
> Thanks, but if we are not going to continue this discussion until we
> come to some conclusions, I see no reason to keep talking about it.  I
> do understand better the difficulties now (thanks!), but I'm not yet
> convinced that the existing solution is the best one.

There are some interim conclusions for now.
I agree that the approach Org takes on remapping may be improved.
I will keep in mind to reach to emacs-devel about possibility to extend
built-in Emacs functionality if it is required to get rid of the Org
remappings.

>> There is no difference between forward-paragraph and
>> org-forward-paragraph on purely texture parts. The difference is only on
>> the Org-specific constructs, where forward-paragraph would behave
>> unexpectedly.
>
> How does Org make sure that the different behavior doesn't happen
> without the user's intent?  Isn't it possible that Org perceives
> something as a sign of an "Org-specific construct" when the user
> didn't mean that?  When that happens, what can the user do to avoid
> the unintended (from that user's POV) Org-specific behavior?
>
> These situations are where user control is necessary, especially for
> casual, non-seasoned users of Org.
...
>> You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
>> provided for users who want to edit tables "like in Org mode", but in
>> _other_ major modes. Inside Org mode, tables are always supported.
>
> What if the user doesn't intend to use Org tables?  How can such a
> user turn off this automatic support, which interprets some patterns
> of buffer text as an indication of a table, and turns on the special
> behavior reserved for Org tables?

Let me be clear here. Org syntax is fixed as much as possible. One
cannot just tell Org to ignore tables in Org buffers. However, users can
alter behavior of specific Org commands if they wish to. Using available
customization and other usual Elisp tools.

One would not expect, say, tex-mode behave correctly if the user
arbitrarily changes buffer syntax table. Similarly, tables are the core
part of Org syntax.

>> Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
>> on Org tables can produce incorrect formatting. For example, calling
>> open-line on the following table
>> 
>> | this | is        | table |
>> | with | <point> 2 | lines |
>>  ...
>> The default behavior is not satisfactory and somewhat unexpected.
>
> You assume that users always expect what Org does in that case.  This
> is not a given, if we want to make Org more attractive for editing
> text that is "less Org-specific", so to speak, like README files, NEWS
> files, etc.

We do not assume that users always expect specific behavior. That's wy
org-special-ctrl-o is customization. It's default value has been agreed
upon and has not been questioned by many. The current behavior is what
most of Org users expect. The default open-line behavior is not.
Otherwise, nobody would bother implementing org-open-line.

>> > No, I'm saying that, sadly, we have no real control on what the
>> > developers of unbundled packages decide to do.  Thus, what you see
>> > there is the evidence of our lack of control more than anything else.
>> 
>> I do not get your point here. I was referring to the markdown-mode and
>> Auctex to illustrate the _need_ to have numerous key bindings in order
>> to edit complex Org documents. It is not just something introduced into
>> Org out of the blue. Other people solved similar problems and did not
>> come up with significantly less key bindings. If you say that usual
>> 40-50 major mode bindings are sufficient, I am not convinced.
>
> I'm not arguing with the need, I'm arguing with the particular
> implementation that caters to that need.

Sorry, but I do not understand what you mean here, except that you are
dissatisfied with the existing implementation. AFAIU, you objected the
number of Org bindings. That many of them are not needed. I think my
reply was targeting your objection. I did not recognize any kind of
reference to implementation specifics.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 17:46:04 +0800
> 
> > Thanks, but if we are not going to continue this discussion until we
> > come to some conclusions, I see no reason to keep talking about it.  I
> > do understand better the difficulties now (thanks!), but I'm not yet
> > convinced that the existing solution is the best one.
> 
> There are some interim conclusions for now.
> I agree that the approach Org takes on remapping may be improved.
> I will keep in mind to reach to emacs-devel about possibility to extend
> built-in Emacs functionality if it is required to get rid of the Org
> remappings.

Thanks, I think this will indeed help in the future.

> > What if the user doesn't intend to use Org tables?  How can such a
> > user turn off this automatic support, which interprets some patterns
> > of buffer text as an indication of a table, and turns on the special
> > behavior reserved for Org tables?
> 
> Let me be clear here. Org syntax is fixed as much as possible. One
> cannot just tell Org to ignore tables in Org buffers. However, users can
> alter behavior of specific Org commands if they wish to. Using available
> customization and other usual Elisp tools.
> 
> One would not expect, say, tex-mode behave correctly if the user
> arbitrarily changes buffer syntax table. Similarly, tables are the core
> part of Org syntax.

The syntax of TeX is dictated by an external tool.  By contrast, the
syntax of Org is determined by Org itself.

Is an Org table always preceded by some directive that makes it
impossible to interpret innocent text as a table?  If so, perhaps the
problems that were bothering me don't actually exist.

But if Org interprets some text patterns as meaning that there's a
table here, that could be contrary to user expectations.

> >> | this | is        | table |
> >> | with | <point> 2 | lines |
> >>  ...
> >> The default behavior is not satisfactory and somewhat unexpected.
> >
> > You assume that users always expect what Org does in that case.  This
> > is not a given, if we want to make Org more attractive for editing
> > text that is "less Org-specific", so to speak, like README files, NEWS
> > files, etc.
> 
> We do not assume that users always expect specific behavior. That's wy
> org-special-ctrl-o is customization.

Relying on a user option for something that can need frequent
adjustment is not the best UX.  defcustom is only a good solution when
it expresses a more-or-less constant user preference that change only
very rarely.  If I may need to change the value before invoking a
command, that's inconvenient.

> It's default value has been agreed upon and has not been questioned
> by many.

Maybe because most Org users are frequent Org users and always know
what they want well enough?  As mentioned, I have other kind of users
in mind.

> >> I do not get your point here. I was referring to the markdown-mode and
> >> Auctex to illustrate the _need_ to have numerous key bindings in order
> >> to edit complex Org documents. It is not just something introduced into
> >> Org out of the blue. Other people solved similar problems and did not
> >> come up with significantly less key bindings. If you say that usual
> >> 40-50 major mode bindings are sufficient, I am not convinced.
> >
> > I'm not arguing with the need, I'm arguing with the particular
> > implementation that caters to that need.
> 
> Sorry, but I do not understand what you mean here, except that you are
> dissatisfied with the existing implementation. AFAIU, you objected the
> number of Org bindings. That many of them are not needed. I think my
> reply was targeting your objection. I did not recognize any kind of
> reference to implementation specifics.

Adding keybindings is a solution to a problem.  I'm saying that
alternatives to that solution were not seriously explored fro those
unbundled packages.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> The syntax of TeX is dictated by an external tool.  By contrast, the
> syntax of Org is determined by Org itself.

Not exactly. We have to keep backwards compatibility and keep in mind 
external packages. So, syntax changes in Org must be avoided unless
strictly necessary. And they should be backwards compatible even if the
changes are necessary.

> Is an Org table always preceded by some directive that makes it
> impossible to interpret innocent text as a table?  If so, perhaps the
> problems that were bothering me don't actually exist.
>
> But if Org interprets some text patterns as meaning that there's a
> table here, that could be contrary to user expectations.

Org tables are always preceded by ^[ \t]*|
I am not sure what is the problem here. One can just not start lines
with | or put zero-width space if starting lines from | is absolutely
necessary.

>> We do not assume that users always expect specific behavior. That's wy
>> org-special-ctrl-o is customization.
>
> Relying on a user option for something that can need frequent
> adjustment is not the best UX.  defcustom is only a good solution when
> it expresses a more-or-less constant user preference that change only
> very rarely.  If I may need to change the value before invoking a
> command, that's inconvenient.

I am not sure why you need a frequent adjustment of org-special-ctrl-o.
This is the first time I hear about such a need.

>> It's default value has been agreed upon and has not been questioned
>> by many.
>
> Maybe because most Org users are frequent Org users and always know
> what they want well enough?  As mentioned, I have other kind of users
> in mind.

I understand, but we have to consider the needs to majority of users,
right? Customizations are there for other kind of users. Almost any kind
of default behavior, be it Org or Emacs, will suffer from some users not
liking it.

>> > I'm not arguing with the need, I'm arguing with the particular
>> > implementation that caters to that need.
>> 
>> Sorry, but I do not understand what you mean here, except that you are
>> dissatisfied with the existing implementation. AFAIU, you objected the
>> number of Org bindings. That many of them are not needed. I think my
>> reply was targeting your objection. I did not recognize any kind of
>> reference to implementation specifics.
>
> Adding keybindings is a solution to a problem.  I'm saying that
> alternatives to that solution were not seriously explored fro those
> unbundled packages.

How can you tell it with confidence? I, personally, cannot come up with
any better solution and haven't seen any alternative to the existing
numerous key bindings, except context-depended dwim bindings (which are
also frown upon). If you have an ideas about any better way to deal with
editing complex markups, please share it.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:13:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The syntax of TeX is dictated by an external tool.  By contrast, the
> > syntax of Org is determined by Org itself.
> 
> Not exactly. We have to keep backwards compatibility and keep in mind 
> external packages. So, syntax changes in Org must be avoided unless
> strictly necessary. And they should be backwards compatible even if the
> changes are necessary.

This is a tangent, entirely unrelated to what I said.  My point was
that, unlike TeX source, which can only be submitted to TeX, and
therefore must follow TeX syntax and is written by people who are
familiar with TeX, Org is being advertised as a mode for editing
mostly-plain text.  And in plain text, what Org decided to take as an
indication of a table could very well be something else, because the
user just typed plain text, and was oblivious to its meaning in Org.

So Org is _unlike_ TeX mode, in that the assumption that every
environment or markup are only used where their Org meaning is
intended -- that assumption is not necessarily true, while it is in
TeX.

> > Is an Org table always preceded by some directive that makes it
> > impossible to interpret innocent text as a table?  If so, perhaps the
> > problems that were bothering me don't actually exist.
> >
> > But if Org interprets some text patterns as meaning that there's a
> > table here, that could be contrary to user expectations.
> 
> Org tables are always preceded by ^[ \t]*|

That's what I remembered, and that is exactly what bothers me in the
context of this discussion: seemingly innocent text sequences are
interpreted by Org in some very special way, and the user doesn't
always have good means to disable that interpretation when it is not
intended.

> I am not sure what is the problem here. One can just not start lines
> with | or put zero-width space if starting lines from | is absolutely
> necessary.

Imagine a user who has no idea that a space and a | at the beginning
of a line means it's an Org table, and thus won't realize that this
magically changes how commands behave in that chunk of text.  That is
the problem that was in my mind when I said that the special
table-related behavior should be better controllable and perhaps off
by default.

> >> We do not assume that users always expect specific behavior. That's wy
> >> org-special-ctrl-o is customization.
> >
> > Relying on a user option for something that can need frequent
> > adjustment is not the best UX.  defcustom is only a good solution when
> > it expresses a more-or-less constant user preference that change only
> > very rarely.  If I may need to change the value before invoking a
> > command, that's inconvenient.
> 
> I am not sure why you need a frequent adjustment of org-special-ctrl-o.

If some instances of the same sequence of characters are indeed meant
to be a table, and other instances in the same file aren't, or if I
need to edit a file where they have this meaning and soon after a file
where they don't, I'd need to flip the option on and off very often.

> >> It's default value has been agreed upon and has not been questioned
> >> by many.
> >
> > Maybe because most Org users are frequent Org users and always know
> > what they want well enough?  As mentioned, I have other kind of users
> > in mind.
> 
> I understand, but we have to consider the needs to majority of users,
> right?

Majority of Org users or majority of Emacs users?

If we want to promote Org to be used more widely and frequently, that
would inevitably mean it will be used by Emacs users who use Org only
infrequently, and those are the ones I'm thinking about.  We should
make it easier to use Org infrequently, by people who don't do
everything in Org.

> >> > I'm not arguing with the need, I'm arguing with the particular
> >> > implementation that caters to that need.
> >> 
> >> Sorry, but I do not understand what you mean here, except that you are
> >> dissatisfied with the existing implementation. AFAIU, you objected the
> >> number of Org bindings. That many of them are not needed. I think my
> >> reply was targeting your objection. I did not recognize any kind of
> >> reference to implementation specifics.
> >
> > Adding keybindings is a solution to a problem.  I'm saying that
> > alternatives to that solution were not seriously explored fro those
> > unbundled packages.
> 
> How can you tell it with confidence?

I can, because I'm talking about what the Emacs core developers did
(or, rather, did not do).  If you somehow interpreted that to allude
to the developers of the respective packages, that was not the intent.

IOW, I'm saying that when you compare the ELPA packages to Org in
these aspects, the comparison is not really valid, because Org gets
much more attention from the core developers than the ELPA packages,
especially since there's a tendency to use Org in more situations.

> If you have an ideas about any better way to deal with editing
> complex markups, please share it.

As I said before, these issues should be discussed one by one.
There's no reason to believe there will be a single solution for all
of them.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not exactly. We have to keep backwards compatibility and keep in mind 
>> external packages. So, syntax changes in Org must be avoided unless
>> strictly necessary. And they should be backwards compatible even if the
>> changes are necessary.
>
> This is a tangent, entirely unrelated to what I said.  My point was
> that, unlike TeX source, which can only be submitted to TeX, and
> therefore must follow TeX syntax and is written by people who are
> familiar with TeX, Org is being advertised as a mode for editing
> mostly-plain text.  And in plain text, what Org decided to take as an
> indication of a table could very well be something else, because the
> user just typed plain text, and was oblivious to its meaning in Org.
>
> So Org is _unlike_ TeX mode, in that the assumption that every
> environment or markup are only used where their Org meaning is
> intended -- that assumption is not necessarily true, while it is in
> TeX.

I think we are mis-communicating. Org mode is not just a major mode, it
is a markup language. org-mode in Emacs is a mode for editing documents
written using Org markup. The markup is fixed with tables being a part
of it. Altering the markup rules is not currently supported for
technical reasons (specific Org markup is not parameterized in various
places in the code-base; though efforts to modulate everything, including
some markup semantics, are ongoing).

Note that Org markup has facilities to escape text constructs using
verbatim/code or #+begin_quote/#+begin_example constructs.

>> Org tables are always preceded by ^[ \t]*|
>
> That's what I remembered, and that is exactly what bothers me in the
> context of this discussion: seemingly innocent text sequences are
> interpreted by Org in some very special way, and the user doesn't
> always have good means to disable that interpretation when it is not
> intended.

I do not think that Org will support major user changes in Org syntax
any time soon or in future. At least, there is no intention to guarantee
such support.

I am not sure why you consider using pre-defined markup constructs as
"innocent". It is problematic in any kind of markup language, be it Org,
markdown, texinfo, or anything else.

>> I am not sure what is the problem here. One can just not start lines
>> with | or put zero-width space if starting lines from | is absolutely
>> necessary.
>
> Imagine a user who has no idea that a space and a | at the beginning
> of a line means it's an Org table, and thus won't realize that this
> magically changes how commands behave in that chunk of text.  That is
> the problem that was in my mind when I said that the special
> table-related behavior should be better controllable and perhaps off
> by default.

Fontification will hint about different semantic meaning.
FYI, I can blame comment-dwim for having the same problem - if user does
not know about comment syntax, comment-dwim will magically change its
meaning in different chunks of text. The same goes for open-line when
there is fill-prefix or even not-easily-visible 'left-margin text
property.

>> > Relying on a user option for something that can need frequent
>> > adjustment is not the best UX.  defcustom is only a good solution when
>> > it expresses a more-or-less constant user preference that change only
>> > very rarely.  If I may need to change the value before invoking a
>> > command, that's inconvenient.
>> 
>> I am not sure why you need a frequent adjustment of org-special-ctrl-o.
>
> If some instances of the same sequence of characters are indeed meant
> to be a table, and other instances in the same file aren't, or if I
> need to edit a file where they have this meaning and soon after a file
> where they don't, I'd need to flip the option on and off very often.

I'd say that you should not use Org on files that do not use Org syntax.
If you do what to do it, you should not be surprised that the same mode
behaves the same way despite you having different idea in mind.

More practically, you can always use file-local variables.

Or, if you think that flipping between some Emacs-global default
behavior and major-mode-specific behavior is frequent or important
use-case, why would you not introduce global Emacs functionality or at
least convention to do exactly this? I do not understand why this very
general problem is blamed on Org specifically and not on other major
modes.

>> > Maybe because most Org users are frequent Org users and always know
>> > what they want well enough?  As mentioned, I have other kind of users
>> > in mind.
>> 
>> I understand, but we have to consider the needs to majority of users,
>> right?
>
> Majority of Org users or majority of Emacs users?
>
> If we want to promote Org to be used more widely and frequently, that
> would inevitably mean it will be used by Emacs users who use Org only
> infrequently, and those are the ones I'm thinking about.  We should
> make it easier to use Org infrequently, by people who don't do
> everything in Org.

I agree here. However, I am not sure how to achieve this. We can limit
Org functionality to lower the entry barrier for infrequent Org users,
yes. However, it will necessarily mean that most of frequent Org users
have to adjust their Org mode configurations - not good.

Should we mark some of the Org commands disabled by default? Though it
would make sense to mark the whole groups of commands. Is it possible?
Is it also possible to disable commands only for users who never
customized Org?

>> > Adding keybindings is a solution to a problem.  I'm saying that
>> > alternatives to that solution were not seriously explored fro those
>> > unbundled packages.
>> 
>> How can you tell it with confidence?
>
> I can, because I'm talking about what the Emacs core developers did
> (or, rather, did not do).  If you somehow interpreted that to allude
> to the developers of the respective packages, that was not the intent.
>
> IOW, I'm saying that when you compare the ELPA packages to Org in
> these aspects, the comparison is not really valid, because Org gets
> much more attention from the core developers than the ELPA packages,
> especially since there's a tendency to use Org in more situations.

I get your point.
However, I do not see a good solution here. Suggestions are welcome.
(I'd prefer if the discussion about the principles of key binding /
editing design were in a separate thread, CCing both emacs-devel and Org
mode ML). This thread is raising so many issues of various levels of
criticality, generality, and validity that it is very hard to stay on
topic.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 17 Jun 2022 13:55:41 +0800
> 
> > So Org is _unlike_ TeX mode, in that the assumption that every
> > environment or markup are only used where their Org meaning is
> > intended -- that assumption is not necessarily true, while it is in
> > TeX.
> 
> I think we are mis-communicating.

Most probably, see below.

> >> Org tables are always preceded by ^[ \t]*|
> >
> > That's what I remembered, and that is exactly what bothers me in the
> > context of this discussion: seemingly innocent text sequences are
> > interpreted by Org in some very special way, and the user doesn't
> > always have good means to disable that interpretation when it is not
> > intended.
> 
> I do not think that Org will support major user changes in Org syntax
> any time soon or in future. At least, there is no intention to guarantee
> such support.
> 
> I am not sure why you consider using pre-defined markup constructs as
> "innocent". It is problematic in any kind of markup language, be it Org,
> markdown, texinfo, or anything else.

Neither of the other markup modes is being proposed for viewing and
editing documents that were not originally edited under those modes.
By contrast, there's a fraction of Emacs contributors and developers
who repeatedly suggest to use Org for documents that were not
originally written in Org.  A notable example (not the only one) is
recent discussions of turning on Org when visiting NEWS files.

If you think these ideas are problematic from the POV of Org
developers, please voice this opinion whenever such proposals are
brought up.  Those proposals, and in general the proposals to use Org
widely in unrelated contexts, is what I had in mind all the time in
this discussion.  Perhaps now you can better understand some of my
comments and responses.

For example, what is your opinion of using Org markup in email
messages?  There are a lot of examples of that, both here and on the
bug tracker.  People use Org markup and Org-style code blocks quite a
lot, and reading that is always jarring to me.  For some reason,
people assume that I read my email in Org mode or some derivative of
Org.

> I'd say that you should not use Org on files that do not use Org syntax.

See above: there are suggestions to do just that.  People are
proposing to use Org in many situations where Org was not originally
used, and therefore the "usual" Org assumptions don't hold.

> > If we want to promote Org to be used more widely and frequently, that
> > would inevitably mean it will be used by Emacs users who use Org only
> > infrequently, and those are the ones I'm thinking about.  We should
> > make it easier to use Org infrequently, by people who don't do
> > everything in Org.
> 
> I agree here. However, I am not sure how to achieve this. We can limit
> Org functionality to lower the entry barrier for infrequent Org users,
> yes. However, it will necessarily mean that most of frequent Org users
> have to adjust their Org mode configurations - not good.
> 
> Should we mark some of the Org commands disabled by default? Though it
> would make sense to mark the whole groups of commands. Is it possible?
> Is it also possible to disable commands only for users who never
> customized Org?

I don't know the answers.  But I'd appreciate if these consequences
and prerequisites of using Org in places where it wasn't used
originally could be voiced when such suggestions are brought up, and
by Org developers on top of that.  I quite frequently opposition to
such proposals is regarded as pure conservatism, so if there are valid
technical reasons and problems to solve before such proposals can be
considered practical, it would be good to have them part of those
discussions.

> This thread is raising so many issues of various levels of
> criticality, generality, and validity that it is very hard to stay on
> topic.

Agreed.  There should be a separate thread for each specific topic.

Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I do not think that Org will support major user changes in Org syntax
>> any time soon or in future. At least, there is no intention to guarantee
>> such support.
>
> Neither of the other markup modes is being proposed for viewing and
> editing documents that were not originally edited under those modes.
> By contrast, there's a fraction of Emacs contributors and developers
> who repeatedly suggest to use Org for documents that were not
> originally written in Org.  A notable example (not the only one) is
> recent discussions of turning on Org when visiting NEWS files.

Do I understand correctly that users would like to have some Org
features (like fontification) in text that partially follows Org markup,
but not fully?

If so, Org mode cannot be used there as a major mode. non-Org parts of
syntax will cause undefined behavior. Which is to be expected as
major mode (every major mode) uses certain assumptions about the text in
buffer. Even text-mode will behave weirdly if binary data is mixed with
plain text. Org major mode expects text in buffers to be using Org
markup.

On the other hand, some parts of Org functionality are available in a
form of minor modes or individual commands. As usual, minor modes are
less demanding on the text in buffer.

AFAIK, we have the following functionality exposed to non-Org buffers:
- org-open-at-point-global/org-insert-link-global commands
- orgtbl-mode to edit tables, which use Org table syntax outside Org mode
- (ELPA) orgalist mode to edit lists written using Org list syntax outside Org mode
- (MELPA) org-msg mode to edit and send messages using Org markup (it
  exports to html mime parts)
- outline minor mode share a lot of functionality with Org without
  relying on Org markup. AFAIK, some Org features have been ported back
  to outline mode as well.

This dedicated functionality is designed to be used outside Org markup
buffers.

People who request Org mode to be used in non-Org buffers probably have
a specific subset of useful Org features in mind. I'd ask them which
features they want to and then consider exposing them out from Org mode
into minor mode (if not yet covered by the above).

> If you think these ideas are problematic from the POV of Org
> developers, please voice this opinion whenever such proposals are
> brought up.  Those proposals, and in general the proposals to use Org
> widely in unrelated contexts, is what I had in mind all the time in
> this discussion.  Perhaps now you can better understand some of my
> comments and responses.

I do understand you comments now. However, I do not follow emacs-devel
closely. So, it would be more efficient to CC Org ML in such
discussions.

> For example, what is your opinion of using Org markup in email
> messages?  There are a lot of examples of that, both here and on the
> bug tracker.  People use Org markup and Org-style code blocks quite a
> lot, and reading that is always jarring to me.  For some reason,
> people assume that I read my email in Org mode or some derivative of
> Org.

I also noticed that. AFAIU it is sometimes the text part sent by org-msg
and sometimes people just using Org syntax, because why not.

I personally do not see a problem with using Org markup to indicate code
blocks. At least, it is not much different from markdown some other
people are using in emails. Even without Org mode, such markup is
perfectly readable. AFAIK, there is no convention on indicating
"special" parts of plain text email (e.g. code snippets) in mailing
lists.

The convention for indicating "markup" in emails is associating the
correct mime-type to the message part. If it is html, the mail agent
should render html. If it is Org, Org can be rendered. If it is plain
text, the behavior is undefined in principle - people sending plain text
should make the text readable as plain text in text-mode (Org can be
made readable usually, but same can be said for e.g. Markdown).

Best,
Ihor

Re: Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sat, 18 Jun 2022 12:40:10 +0800
> 
> > Neither of the other markup modes is being proposed for viewing and
> > editing documents that were not originally edited under those modes.
> > By contrast, there's a fraction of Emacs contributors and developers
> > who repeatedly suggest to use Org for documents that were not
> > originally written in Org.  A notable example (not the only one) is
> > recent discussions of turning on Org when visiting NEWS files.
> 
> Do I understand correctly that users would like to have some Org
> features (like fontification) in text that partially follows Org markup,
> but not fully?

It's more than that.  People suggest to start using Org for files that
were not written under Org mode, and the reason is not just
fontifications, but also Org outline-related and other commands.  IIUC
what you mean by "Org markup", it basically isn't used at all in those
files (but would probably "seep in" given enough time, if we indeed
begin using Org there).

> If so, Org mode cannot be used there as a major mode. non-Org parts of
> syntax will cause undefined behavior. Which is to be expected as
> major mode (every major mode) uses certain assumptions about the text in
> buffer. Even text-mode will behave weirdly if binary data is mixed with
> plain text. Org major mode expects text in buffers to be using Org
> markup.

Yup.

> On the other hand, some parts of Org functionality are available in a
> form of minor modes or individual commands. As usual, minor modes are
> less demanding on the text in buffer.
> 
> AFAIK, we have the following functionality exposed to non-Org buffers:
> - org-open-at-point-global/org-insert-link-global commands
> - orgtbl-mode to edit tables, which use Org table syntax outside Org mode
> - (ELPA) orgalist mode to edit lists written using Org list syntax outside Org mode
> - (MELPA) org-msg mode to edit and send messages using Org markup (it
>   exports to html mime parts)
> - outline minor mode share a lot of functionality with Org without
>   relying on Org markup. AFAIK, some Org features have been ported back
>   to outline mode as well.

Indeed, files that currently use Outline mode are the ones where the
above proposals are voiced most frequently.

The other minor modes are not necessarily relevant; at least I don't
recall those parts of Org's functionality being used as the reasons
for using Org where it wasn't before.

> People who request Org mode to be used in non-Org buffers probably have
> a specific subset of useful Org features in mind. I'd ask them which
> features they want to and then consider exposing them out from Org mode
> into minor mode (if not yet covered by the above).

Thanks, will keep this in mind next time such an issue comes up.

> > If you think these ideas are problematic from the POV of Org
> > developers, please voice this opinion whenever such proposals are
> > brought up.  Those proposals, and in general the proposals to use Org
> > widely in unrelated contexts, is what I had in mind all the time in
> > this discussion.  Perhaps now you can better understand some of my
> > comments and responses.
> 
> I do understand you comments now. However, I do not follow emacs-devel
> closely. So, it would be more efficient to CC Org ML in such
> discussions.

Fair enough.

> The convention for indicating "markup" in emails is associating the
> correct mime-type to the message part. If it is html, the mail agent
> should render html. If it is Org, Org can be rendered. If it is plain
> text, the behavior is undefined in principle - people sending plain text
> should make the text readable as plain text in text-mode (Org can be
> made readable usually, but same can be said for e.g. Markdown).

The difference between HTML and Org is that all MUAs support the
former, whereas Org can only be supported by Emacs-based MUA (and is
not supported by all Emacs-based MUA: e.g., Rmail doesn't currently
support it), and then only if the mime-type has been set correctly.

Re: Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

I will reply to the last paragraph only. Other questions appear to be
settled.

Eli Zaretskii <eliz@gnu.org> writes:

>> The convention for indicating "markup" in emails is associating the
>> correct mime-type to the message part. If it is html, the mail agent
>> should render html. If it is Org, Org can be rendered. If it is plain
>> text, the behavior is undefined in principle - people sending plain text
>> should make the text readable as plain text in text-mode (Org can be
>> made readable usually, but same can be said for e.g. Markdown).
>
> The difference between HTML and Org is that all MUAs support the
> former, whereas Org can only be supported by Emacs-based MUA (and is
> not supported by all Emacs-based MUA: e.g., Rmail doesn't currently
> support it), and then only if the mime-type has been set correctly.

I think I was not very clear in the above.

I do not advocate that MUA should support Org out of the box (it would
be nice if it could, especially if people use Org instead of html, but
that is a bit tangent to what I was talking about). I'd like to focus on
plain text.

If we forget about Org _major mode_, there is Org markup.
Some people may disagree, but Org markup is perfectly readable as a
non-rendered plain text. Similarly, Markdown is perfectly readable. Or
any other ad-hoc self-consistent plain text convention. Admittedly, Org
may be a bit too verbose in some cases (for some people, not for me;
and it does not mean that alternative opinions should be ignored).

(I do not say that abusing Org markup cannot make plain text unreadable,
it can be said even for the purest plain text email; it was discussed in
another branch).

On top of being readable Org/Markdown can also be exported to html
adding a benefit to users who prefer html rendering. Unlike Markdown
(AFAIK), Org can also be exported to plain text, removing all the
"unnecessary markup clutter" some people are so triggered about.

Ideally, if some user prefers to use Org markup when writing emails, MUA
should take care about supplying appropriately converted plain text
version and html version of the mime-parts. That will let people read
whatever format they prefer. Not all MUA support this, unfortunately.
So, we have what we have and some people prefer to *write* emails using
Org markup and others do not like to *read* such emails.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> For example, what is your opinion of using Org markup in email
> messages?  There are a lot of examples of that, both here and on the
> bug tracker.  People use Org markup and Org-style code blocks quite a
> lot, and reading that is always jarring to me.  For some reason,
> people assume that I read my email in Org mode or some derivative of
> Org.

+1.  I hope people will stop doing that.  It is the reason I can't use
my work email to talk to those people, since the filter thinks the Org
syntax is a virus.

It's also visually jarring.

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> +1.  I hope people will stop doing that.  It is the reason I can't use
> my work email to talk to those people, since the filter thinks the Org
> syntax is a virus.
>
> It's also visually jarring.

Most Emacs mail readers understand the syntax and format it in a
readable and pretty way, so I hope people won't stop doing that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Most Emacs mail readers understand the syntax and format it in a
> readable and pretty way, so I hope people won't stop doing that.

Really?  I don't mind the Org syntax in email, FWIW, but I can't
remember ever seeing a MUA that recognizes it and displays it in
a pretty way.  Does Gnus does that?  How/when?


        Stefan

Re: Convert README.org to plain text README while installing package

[tomas]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

On Sat, Jun 18, 2022 at 09:33:16AM -0400, Stefan Monnier wrote:
> > Most Emacs mail readers understand the syntax and format it in a
> > readable and pretty way, so I hope people won't stop doing that.
> 
> Really?  I don't mind the Org syntax in email, FWIW, but I can't
> remember ever seeing a MUA that recognizes it and displays it in
> a pretty way.  Does Gnus does that?  How/when?

Same with me: I actually use mutt and my reader/writer is vim (yes,
really: I do most of my stuff with Emacs, but mail is one of the
last surviving realms vor vim). I even catch myself formatting
things in an Orgish way with vim (no special plugin: I know there
are).

But I can totally understand that some people don't like it. What
to do?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Convert README.org to plain text README while installing package

[Visuwesh]

[சனி ஜூன் 18, 2022 17:50] tomas@tuxteam.de wrote:

> On Sat, Jun 18, 2022 at 09:33:16AM -0400, Stefan Monnier wrote:
>> > Most Emacs mail readers understand the syntax and format it in a
>> > readable and pretty way, so I hope people won't stop doing that.
>> 
>> Really?  I don't mind the Org syntax in email, FWIW, but I can't
>> remember ever seeing a MUA that recognizes it and displays it in
>> a pretty way.  Does Gnus does that?  How/when?
>
> Same with me: I actually use mutt and my reader/writer is vim (yes,
> really: I do most of my stuff with Emacs, but mail is one of the
> last surviving realms vor vim). I even catch myself formatting
> things in an Orgish way with vim (no special plugin: I know there
> are).
>
> But I can totally understand that some people don't like it. What
> to do?
>

I find it jarring when people throw in the very verbose org format for
source blocks.  Other org syntax is just fine IMO.

> Cheers

Re: Convert README.org to plain text README while installing package

[tomas]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On Sat, Jun 18, 2022 at 09:30:47PM +0530, Visuwesh wrote:
> [சனி ஜூன் 18, 2022 17:50] tomas@tuxteam.de wrote:

[...]

> > But I can totally understand that some people don't like it. What
> > to do?
> >
> 
> I find it jarring when people throw in the very verbose org format for
> source blocks.  Other org syntax is just fine IMO.

How would you delimit a source block?

Perhaps there are some ideas for the Org folks (I'm a happy user, no
more :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Convert README.org to plain text README while installing package

[Yuri Khan]

On Sun, 19 Jun 2022 at 00:14, <tomas@tuxteam.de> wrote:

> How would you delimit a source block?

Classic Markdown does it by indenting with +4 spaces, but this has no
way to declare the language of the block.

    (defun greet ()
        (message "Hello World!"))

GitHub-flavored Markdown does it with three backticks, optionally
followed by the language name on the same line, followed by lines of
the source block, followed by three backticks on a separate line.

```elisp
(defun greet()
    (message "Hello World!"))
```

Either is much less verbose/visually noisy than the equivalent Org syntax.

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> GitHub-flavored Markdown does it with three backticks, optionally
> followed by the language name on the same line, followed by lines of
> the source block, followed by three backticks on a separate line.
> 
> ```elisp
> (defun greet()
>     (message "Hello World!"))
> ```

StackExchange/Overflow uses the same.

> Either is much less verbose/visually noisy than the equivalent Org
> syntax.

Yes.

(There might be good reasons / a need
for Org's noisier delimiting.  Dunno.)

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> Date: Sat, 18 Jun 2022 19:13:39 +0200
> From: tomas@tuxteam.de
> 
> > I find it jarring when people throw in the very verbose org format for
> > source blocks.  Other org syntax is just fine IMO.
> 
> How would you delimit a source block?

Why do you need to delimit it at all?  A code block is usually
preceded by some text that makes it quite clear that what follows is a
code fragment.  And if you indent the code block by 2 or more spaces,
that makes it stand out enough to understand where it ends.

Re: Convert README.org to plain text README while installing package

[Visuwesh]

[சனி ஜூன் 18, 2022 19:13] tomas@tuxteam.de wrote:

> On Sat, Jun 18, 2022 at 09:30:47PM +0530, Visuwesh wrote:
>> [சனி ஜூன் 18, 2022 17:50] tomas@tuxteam.de wrote:
>
> [...]
>
>> > But I can totally understand that some people don't like it. What
>> > to do?
>> >
>> 
>> I find it jarring when people throw in the very verbose org format for
>> source blocks.  Other org syntax is just fine IMO.
>
> How would you delimit a source block?
>

I see no reason for fancy markup in emails when the code snippets are
going to be small.  If it is long, then it is better to attach it as a
file anyway.  Personally, I indent source blocks by four spaces.

> Perhaps there are some ideas for the Org folks (I'm a happy user, no
> more :)
>
> Cheers

Re: Convert README.org to plain text README while installing package

[tomas]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On Sat, Jun 18, 2022 at 11:25:13PM +0530, Visuwesh wrote:
> [சனி ஜூன் 18, 2022 19:13] tomas@tuxteam.de wrote:
> 
> > On Sat, Jun 18, 2022 at 09:30:47PM +0530, Visuwesh wrote:
> >> [சனி ஜூன் 18, 2022 17:50] tomas@tuxteam.de wrote:
> >
> > [...]
> >
> >> > But I can totally understand that some people don't like it. What
> >> > to do?
> >> >
> >> 
> >> I find it jarring when people throw in the very verbose org format for
> >> source blocks.  Other org syntax is just fine IMO.
> >
> > How would you delimit a source block?
> >
> 
> I see no reason for fancy markup in emails when the code snippets are
> going to be small.  If it is long, then it is better to attach it as a
> file anyway.  Personally, I indent source blocks by four spaces.

[...]

Thanks (also Eli and Yuri :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> > I find it jarring when people throw in the very verbose org format
> > for source blocks.
> 
> How would you delimit a source block?

Dunno what OP had in mind, but if, as for
the context of my comment, there's no need
to delimit it, then, well ... no need to
delimit it.

Maybe it's a habit.  Dunno.

Re: [External] : Re: Convert README.org to plain text README while installing package

[Tim Cross]

Drew Adams <drew.adams@oracle.com> writes:

>> > I find it jarring when people throw in the very verbose org format
>> > for source blocks.
>> 
>> How would you delimit a source block?
>
> Dunno what OP had in mind, but if, as for
> the context of my comment, there's no need
> to delimit it, then, well ... no need to
> delimit it.
>
> Maybe it's a habit.  Dunno.

I suspect there are likely a number of contributing factors, including 

- Increase in number of people using org's literate programming
  facilities to manage their config files. When showing configs as part
  of a question or providing an answer, it is very easy to select and
  cut the src block for pasting 

- A number of Emacs MUAs can do some nice things with source blocks
  delimited with #+begin_src/#+end_src. For example, mu4e will apply
  syntax highlighting/font-locking for the specified language. As it
  uses Gnus under the hood, I expect Gnus does this as well. You don't
  get such highlighting if you only 'indent'. I get this behaviour, but
  did not enable any special feature - came 'out of the box'. 

- Some org users will use the org-mime package. This package makes it
  trivial to convert a buffer which uses org mode syntax to a multi-part
  mail message with plain text and html where the html is rendered with
  formatted source blocks and the org markup, embedded images, different
  headings etc. Of course, if you setup your MUA to prefer plain text
  over html, you will see the org markup text (though I think org-mime
  can be configured to export to plain text for the text/plain part). I
  rarely use this as I prefer plain text email. However, I have had some
  clients who depend on HTML email messages and using org mode and
  org-mime makes it very easy to work with them and still remain in my
  Emacs environment. Not sure this would be possible without org mode.
  Really nice thing is that I can decide to use org mode for composing a
  message on a message by message basis. I can even start composing the
  message and when I realise it will probably look better formatted as
  HTML, switch to composing using org markup and then send it as a
  multi-part message. 
  
I still think the worst thing that ever happened to email was moving
away from plain text and allowing HTML. Unfortunately, HTML has now
become so ubiquitous in email, people now see plain text as 'old' or
'primitive'. There is also the issue that most people tend to use
proportional fonts these days, so the old days of being able to create OK
formatted tabular data in plain text are gone - you need HTML now. 

As to the verbosity or jarring nature of #+begin_src/#+end_src as
delimiters - I really just don't notice them. I guess that after years
of reading config files and using programming languages with too much
boiler plate code, I've developed the skill to easily ignore such
trivial constructs. 

Of course, if your using emacs, it is reasonably trivial to make them
less intrusive with various font locking techniques. If your not using
Emacs for your email, well I guess that is what happens when you use an
inferior solutions ;-) If you are using emacs and haven't bothered doing
anything to reduce the jarring nature, I guess it isn't that bad an itch
after all.
 

Re: [External] : Re: Convert README.org to plain text README while installing package

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2615 bytes --]

On Sun, Jun 19, 2022 at 10:24:10AM +1000, Tim Cross wrote:
> 
> Drew Adams <drew.adams@oracle.com> writes:
> 
> >> > I find it jarring when people throw in the very verbose org format
> >> > for source blocks.
> >> 
> >> How would you delimit a source block?
> >
> > Dunno what OP had in mind, but if, as for
> > the context of my comment, there's no need
> > to delimit it, then, well ... no need to
> > delimit it.
> >
> > Maybe it's a habit.  Dunno.
> 
> I suspect there are likely a number of contributing factors, including 

[good insights]

> I still think the worst thing that ever happened to email was moving
> away from plain text and allowing HTML. Unfortunately, HTML has now
> become so ubiquitous in email, people now see plain text as 'old' or
> 'primitive'. There is also the issue that most people tend to use
> proportional fonts these days, so the old days of being able to create OK
> formatted tabular data in plain text are gone - you need HTML now. 

Definitely.

> As to the verbosity or jarring nature of #+begin_src/#+end_src as
> delimiters - I really just don't notice them. I guess that after years
> of reading config files and using programming languages with too much
> boiler plate code, I've developed the skill to easily ignore such
> trivial constructs. 

But that's the point. To some people, obviously, #+begin_foo is at
least as jarring as HTML markup seems to us (I'm not talking about
sick Microsoft markup, where 3/4 of the file is fonts and CSS, that's
even worse. I'm talking about somewhat "readable" HTML, some renderers
try to do this).

> Of course, if your using emacs, it is reasonably trivial to make them
> less intrusive with various font locking techniques. If your not using
> Emacs for your email, well I guess that is what happens when you use an
> inferior solutions ;-) If you are using emacs and haven't bothered doing
> anything to reduce the jarring nature, I guess it isn't that bad an itch
> after all.

I think here we are at the core of the conflict. "Well, duh, use Emacs"
(and implicitly: "use Org mode") is akin to "well, duh, use a browser".

As fond as I am of Org myself (I admitted in this thread to writing
it by hand without mode support), I think expecting people to follow
suit is bound to create this conflict.

No, to me, Org markup isn't jarring. But I can perfectly well relate
to this feeling.

I think, at this point it would be much wiser to take a step back
and think about a solution instead of just repeating "but Org is
better" :-)

Cheers
-- 
t

>  
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [External] : Re: Convert README.org to plain text README while installing package

[Tim Cross]

<tomas@tuxteam.de> writes:

> [[PGP Signed Part:Undecided]]
> On Sun, Jun 19, 2022 at 10:24:10AM +1000, Tim Cross wrote:
>> 
>> Drew Adams <drew.adams@oracle.com> writes:
>> 
>> >> > I find it jarring when people throw in the very verbose org format
>> >> > for source blocks.
>> >> 
>> >> How would you delimit a source block?
>> >
>> > Dunno what OP had in mind, but if, as for
>> > the context of my comment, there's no need
>> > to delimit it, then, well ... no need to
>> > delimit it.
>> >
>> > Maybe it's a habit.  Dunno.
>> 
>> I suspect there are likely a number of contributing factors, including 
>
> [good insights]
>
>> I still think the worst thing that ever happened to email was moving
>> away from plain text and allowing HTML. Unfortunately, HTML has now
>> become so ubiquitous in email, people now see plain text as 'old' or
>> 'primitive'. There is also the issue that most people tend to use
>> proportional fonts these days, so the old days of being able to create OK
>> formatted tabular data in plain text are gone - you need HTML now. 
>
> Definitely.
>
>> As to the verbosity or jarring nature of #+begin_src/#+end_src as
>> delimiters - I really just don't notice them. I guess that after years
>> of reading config files and using programming languages with too much
>> boiler plate code, I've developed the skill to easily ignore such
>> trivial constructs. 
>
> But that's the point. To some people, obviously, #+begin_foo is at
> least as jarring as HTML markup seems to us (I'm not talking about
> sick Microsoft markup, where 3/4 of the file is fonts and CSS, that's
> even worse. I'm talking about somewhat "readable" HTML, some renderers
> try to do this).
>
>> Of course, if your using emacs, it is reasonably trivial to make them
>> less intrusive with various font locking techniques. If your not using
>> Emacs for your email, well I guess that is what happens when you use an
>> inferior solutions ;-) If you are using emacs and haven't bothered doing
>> anything to reduce the jarring nature, I guess it isn't that bad an itch
>> after all.
>
> I think here we are at the core of the conflict. "Well, duh, use Emacs"
> (and implicitly: "use Org mode") is akin to "well, duh, use a browser".
>
> As fond as I am of Org myself (I admitted in this thread to writing
> it by hand without mode support), I think expecting people to follow
> suit is bound to create this conflict.
>
> No, to me, Org markup isn't jarring. But I can perfectly well relate
> to this feeling.
>
> I think, at this point it would be much wiser to take a step back
> and think about a solution instead of just repeating "but Org is
> better" :-)
>
> Cheers

I think you missed my point. For the record, I was not implying "Just
use org". I was suggesting that if your using Emacs, there are numerous
things you can do to reduce the 'jarring' nature of org block syntax in
email messages (such as using font-lock to alter the appearance and
reduce that impact. However, if your not using something as powerful as
Emacs to read your email, your likely screwed and if you are, but have
done nothing to reduce the visual jarring/offence of org block syntax,
then it obviously isn't as common or offensive to motivate you using
that powerful editor to change the situation. 

at no point did I say or imply the only solution was to use org mode.
Instead, I was simply pointing out there are some good reasons some
people will use it and there are some advantages with Emacs mail readers
(such as font-lock source code snippets). The implication is that it
isn't going away, so either you have to put up with it or you have to
find a way to reduce its impact and Emacs provides a wealth of
functionality to do the latter. 

of course there is also the option to just complain about it, which is
easy, but doesn't change a lot.

Re: [External] : Re: Convert README.org to plain text README while installing package

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

On Sun, Jun 19, 2022 at 03:36:43PM +1000, Tim Cross wrote:

[...]

> I think you missed my point. For the record, I was not implying "Just
> use org".

I didn't intend to convey that. I was just inviting to squint a bit
and to realise that the org side arguments are as valid as those for
any other markup.

I think we are touching a fundamental problem: either you have a
sufficiently unequivocal markup as to be machine readable, then
it automatically becomes somewhat heavy-handed, or you accept that
the machine has to apply some sort of mushy heuristics to try to
extract part of the markup.

And don't get me started on the "semantic" part (watch the Texinfo
branch of this thread :). The one's semantic is the other's syntax.
Semantics is like an onion: once you peel off all layers, you are
left with... nothing.

My point is rater that we'll possibly have to accept that to some,
even seasoned Emacs users, Org might be too heavy handed, and the
mitigations you propose might be of no help.

This shouldn't preclude us of exploring what is in-between. But
we need a very open mind, and if Po Lu, Alan, Drew et al say
"it feels jarring" we better take it seriously instead of saying
"nah, you'll get used to it" (I'm exaggerating a bit, I know).

Would I like a unified markup to have for Texinfo, docstrings,
quick notes, comments? Sure. Would I like for it to be machine
readable? You bet. Would I like for it to be easy on the eyes, even
as raw text? No less. Would I like for it to have links my Emacs
can automatically follow? Yes, please. Would I like for it to be
extensible, so I can come up with an "this is a FOO span"? Exactly.

But with that wishlist, it could well be ony Alan Perlis's
lollipop will do.

Still worth a try :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> My point is rater that we'll possibly have to accept that to some,
> even seasoned Emacs users, Org might be too heavy handed, and the
> mitigations you propose might be of no help.
> 
> This shouldn't preclude us of exploring what is in-between. But
> we need a very open mind, and if Po Lu, Alan, Drew et al say
> "it feels jarring" 

Just to be clear, I never said if feels
jarring.  I said that IF it doesn't add
anything (and at least for me it doesn't)
then it's just noise (for me it is).  Occam
says don't add noise uselessly.

I said, "IF, as for the context of my comment
[which was emacs.StackExchange], there's no
need to delimit it, THEN, well ... no need to
delimit it."  [Emphasis added.]

And I said:

 "(There might be good reasons / a need
   for Org's noisier delimiting.  Dunno.)"

My comments were not particularly about email,
which the thread seems now to be focusing on.
___

As for email: I don't use Emacs to read email.
So for me, at least, Org delimiting offers no
positive; it's just noise.

It's noise I can live with.  This isn't about
me.  It's about what's good for Emacs.  My
point was that in contexts where it doesn't
really add anything, maybe people just use it
out of habit (dunno).  And IF it doesn't add
anything THEN....

FWIW, I use Outlook for email.  I have no
problem with HTML mail.  There's HTML markup
under the surface, of course, but I don't see
its noise.  Same with using a web browser.
And HTML markup and its rendering does have a
purpose/use.

Re: Convert README.org to plain text README while installing package

[Po Lu]

tomas@tuxteam.de writes:

> How would you delimit a source block?
>
> Perhaps there are some ideas for the Org folks (I'm a happy user, no
> more :)

4 spaces at the start.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Po Lu <luangruo@yahoo.com> writes:

> tomas@tuxteam.de writes:
>
>> How would you delimit a source block?
>>
>> Perhaps there are some ideas for the Org folks (I'm a happy user, no
>> more :)
>
> 4 spaces at the start.

which is fine for basic markdown where all you need is to delineate the
block. However, org has much more functionality than that for source
blocks and many options and/or arguments that can be passed into such
blocks. All of which makes using just indentation inadequate.  

Re: Convert README.org to plain text README while installing package

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

> which is fine for basic markdown where all you need is to delineate the
> block. However, org has much more functionality than that for source
> blocks and many options and/or arguments that can be passed into such
> blocks. All of which makes using just indentation inadequate.  

And please explain those options are necessary in mail, where the only
meaning intended to be communicated is that the block of text is code,
and not something else?

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Date: Sun, 19 Jun 2022 12:08:56 +1000
> 
> > 4 spaces at the start.
> 
> which is fine for basic markdown where all you need is to delineate the
> block. However, org has much more functionality than that for source
> blocks and many options and/or arguments that can be passed into such
> blocks. All of which makes using just indentation inadequate.  

Doesn't HTML express all of those adequately?  Then sending HTML email
is a much better alternative, IMO, as it will allow much more MUAs to
produce a readable and clear display of the email.

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Date: Sun, 19 Jun 2022 12:08:56 +1000
>> 
>> > 4 spaces at the start.
>> 
>> which is fine for basic markdown where all you need is to delineate the
>> block. However, org has much more functionality than that for source
>> blocks and many options and/or arguments that can be passed into such
>> blocks. All of which makes using just indentation inadequate.  
>
> Doesn't HTML express all of those adequately?  Then sending HTML email
> is a much better alternative, IMO, as it will allow much more MUAs to
> produce a readable and clear display of the email.

Agreed. That is why I mentioned org-mime as that is what it can help do. 

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> I find it jarring when people throw in the very verbose org format for
> source blocks.  Other org syntax is just fine IMO.

Yeah, it can be noisy.

I see people sometimes adding it around a code
snippet posted on emacs.StackExchange.  Not only
useless but almost obnoxious.

The markup highlights Lisp text, but it of course
does nothing useful with the Org noise around the
Lisp.

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Really?  I don't mind the Org syntax in email, FWIW, but I can't
> remember ever seeing a MUA that recognizes it and displays it in
> a pretty way.  Does Gnus does that?  How/when?

Isn't this fontified like Emacs Lisp for you?

#+BEGIN_SRC emacs-lisp
  (defun org-xor (a b)
    "Exclusive or."
    (if a (not b) b))
 #+END_SRC

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> Isn't this fontified like Emacs Lisp for you?
> 
> #+BEGIN_SRC emacs-lisp
>   (defun org-xor (a b)
>     "Exclusive or."
>     (if a (not b) b))
>  #+END_SRC

(I realize you're asking Stefan, and I'm not he.)

But out of curiosity, about what proportion of
Emacs users do you suppose use Emacs for email?

(Not to mention the proportion of those that
use Org mode with email.)

Re: Convert README.org to plain text README while installing package

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

> For example, what is your opinion of using Org markup in email
> messages?  There are a lot of examples of that, both here and on the
> bug tracker.  People use Org markup and Org-style code blocks quite a
> lot, and reading that is always jarring to me.  For some reason,
> people assume that I read my email in Org mode or some derivative of
> Org.

(There's been a whole subthread about this particular point, but I don't
think I saw anyone spell this out clearly so here we go:)

lisp/gnus/mm-uu.el has picked up on Org-style code blocks since
2011-01-20 "mm-decode.el (mm-inline-media-tests): Add text/x-sh. […]"
(6e3165fb24).

That's around the time I discovered Emacs; I used Gnus as my daily mail
driver for years before I opened my first Org file.  I've probably sent
a couple #+begin_src's before even knowing what Org was, assuming it was
one of those conventions Emacs picked up on (like "> " citations, "--"
signature blocks and "--- 8< ---" scissor lines).

(I don't even remember when I realized that Emacs shipped more than one
MUA, and that not every maintainer used Gnus)

So apologies for partaking in that practice.  FWIW, if my story is any
indication, not everyone who sent those blocks was inherently advocating
for using Org syntax pervasively: they might just have assumed that it
made life better for anyone reading their mail with Emacs (because they
automatically got syntax highlighting out of it), and not realize that
instead they were making it worse (by sandwiching perfectly good code
between noisy sigils).

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

>
> So apologies for partaking in that practice.  FWIW, if my story is any
> indication, not everyone who sent those blocks was inherently advocating
> for using Org syntax pervasively: they might just have assumed that it
> made life better for anyone reading their mail with Emacs (because they
> automatically got syntax highlighting out of it), and not realize that
> instead they were making it worse (by sandwiching perfectly good code
> between noisy sigils).

I don't believe there is anything to apologise for. Some people find the
#+begin_src/#+end_src delimiting of blocks irritating and others don't.
To be honest, I had never even given it a thought until this thread. 
Given it isn't something seen as a common complaint, I wouldn't worry
about it too much. I actually quite like getting code snippets which are
syntax highlighted, but I do use Emacs for my MUA. 

The fact we see few complaints about it means we don't really have any
idea how many people find it a problem. The low numbers complaining
could indicate few care or few find it an issue or it could mean the
practice is not common enough to motivate complaining. Either way, just
a storm in a tea cup. 

For those who do use Emacs as there MUA and who also use org-mode, the
good news is you can avoid the ire of some users by
converting your org content to fairly good looking HTML formatted messages. 
Everyone gets to see syntax highlighted code - at least everyone who
has a mail reader capable of rendering HTML mail, which is probably the
vast majority these days. Bad news is, this further encourages HTML for
email (I would have preferred #+begin_src/#+end_src), but that argument
is pretty much dead in the water having been won by HTML. For those who
do prefer HTML mail, org-mode is probably your only real solution when
using Emacs as your MUA. 

Re: Convert README.org to plain text README while installing package

[Kévin Le Gouguec]

Ihor Radchenko <yantar92@gmail.com> writes:

>>   . if the above doesn't work, make forward-paragraph work through
>>     forward-paragraph-function, so that Org could define its own
>>     function
>
> Introducing the required flexibility into Emacs core is certainly an
> option. Though it is somewhat difficult one because Org also has to
> support older versions of Emacs. We are somewhat limited in using newly
> introduced built-in features.

A tiny note on that one very specific issue: it is possible to expose
bleeding-edge core Emacs features to older Emacsen using ELPA's ":core"
package type.

Eglot, for example, makes use of improvements in eldoc.el, flymake.el,
project.el, xref.el that live exclusively on emacs's master branch, and
have not made it into a formal release yet.

As long as those core libraries are protected from changes which make
them reliant on newer features that cannot be released on ELPA[1], they
allow their dependents to keep targeting older Emacs releases.  E.g. the
master branch's project.el still aims to support Emacs 26.1.

So to some extent, "the technology exists" to allow Org to add hooks to
core Emacs features, and use those hooks through GNU ELPA, *and* support
older Emacsen.


[1] See e.g. 2022-02-03 "Don't use `string-search` in soap-client
    (bug#53744)" (1b0342628e).

Re: Common keybindings idea for multiple markups

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2022-06-12 20:52]:
> I was referring to the markdown-mode and
> Auctex to illustrate the _need_ to have numerous key bindings in order
> to edit complex Org documents. It is not just something introduced into
> Org out of the blue. Other people solved similar problems and did not
> come up with significantly less key bindings.

There are many markups and using various of them is very beneficial
due to various outputs and qualities. Example is Asciidoc and
Asciidoctor implementation 

AsciiDoc Writer’s Guide | Asciidoctor
https://asciidoctor.org/docs/asciidoc-writers-guide/#

and if I may say, there may be reasons why is Asciidoctor so much
financially supported:

Asciidoctor - Open Collective
https://opencollective.com/asciidoctor

or txt2tags:

txt2tags
https://txt2tags.org/

and many others like reStructuredText and Wiki and what else, too
many.

The way to go to minimize at least some key bindings is to find the
union on what is common to each of those markups.

Common markup tags are headers, italic, bold, underlined, monospaced,
lists, links, tables, separators, etc. etc.

Once it is found what is common, then Emacs and markup modes may
define how those markups look like.

Then we can use functions similar to facemenu-set-bold to get that
bold markup in a way how a markup mode specifies it.

Then if key binding is for example "C-c b" for "bold" that bold would
appear different in HTML mode, rather like <strong>word</strong> but
in Markdown it would appear as **strong** and in Org as *strong* and
so on. Markup mode would define how those common markups are to be
applied, while Emacs could have its standardized key bindings. That
would make life easier for us who use multiple markups.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: acm@muc.de, emacs-devel@gnu.org
>> Date: Tue, 07 Jun 2022 16:14:52 +1000
>> 
>> > I can tell you what I see with my very simple 1570-line Org file,
>> > which just has some todo items organized in 4-level hierarchy:
>> >
>> >   . ~100 keys that look "general Org" to me
>> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
>> >     'backward-sentence'
>> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
>> >     bound by default, but they are there
>> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
>> >     but they are all bound by default, just because I visited an Org
>> >     file 
>> >
>> > So "just" 230 key bindings.  And this is without any minor/add-on Org
>> > mode/feature enabled, at least according to "C-h m".
>> >
>> > Do you see something different?  Are you still saying that it's not a
>> > lot, or that it's "based on ignorance and paranoia"?  If so, please
>> > point out where I'm ignorant and/or paranoiac, because I'd really like
>> > to know.
>> 
>> I think Alan's response has pretty much confirmed what I was saying. Alan has
>> made it fairly clear that his real underlying concern is about a stealthy
>> strategy to get org mode more intrinsically tied into Emacs and in the end,
>> making it so critical to normal operation that everyone will be forced to learn
>> org mode regardless. I don't necessarily agree with such an argument, but think
>> having a debate around that would be more up-front and in some ways honest
>> compared to what appear to be somewhat contrived alternative arguments about
>> excessive key bindings and added complexity. My characterisation of this all
>> being fear and paranoia may have been overstating things, but then again I'm not
>> sure. There certainly does seem to be considerable fear involved. I don't know
>> about paranoia, but I have not seen any evil plan to have org mode assimilate
>> Emacs as if it was the Borg and I've seen no discussions on the org list about
>> ways to get org more ingrained in Emacs or make Emacs more dependent on org.
>> 
>> With respect to ignorance, Alan made it quite clear he has not spent
>> the time or effort to learn org mode, so he is ignorant about its
>> operation and lacks any real understanding of its operation. He has made
>> assertions about the number of key bindings which are wildly over
>> inflated and about required levels of complexity which simply would not
>> be necessary with the org mode rendering of a read only buffer. In
>> short, he has made a lot of assumptions about both the key bindings and
>> complexity based on only a fairly cursory scan of the manual and very
>> limited experience. 
>> 
>> All that appears to have occurred here is that someone noted that packages often
>> had a readme.org file and wondered if this should be translated into just a
>> plain readme file. Someone else suggested we could have package descriptions
>> render the readme.org file using org and not require export into plain readme
>> file and mentioned some possible advantages this could have wrt navigation,
>> rendering of tables, links and images or inclusion of source code snippets,
>> examples etc.
>> 
>> >From this point, some somewhat extreme claims and arguments have been thrown in.
>> One was Alan's claim that org would introduce 785 new key bindings, many of
>> which not bounmd to C-c and which would presumably either pollute the key space
>> or conflict with existing bindings. All I have done is point out this claim was
>> inaccurate and overstated due to an overly simplistic analysis of what is in the
>> key index of the org manual. 
>> 
>> it should be noted that if we were to add 'native' rendering of readme.org
>> files, this would likely be done with an org minor mode rather than a full blown
>> org mode as very little of the overall org functionality would be required in
>> this situation. All that would be required is org rendering support and org
>> support for navigation (folding/unfolding, following links and possibly
>> list/table navigation). All the advanced org functionality relating to babel,
>> noweb, todo and time management, agendas, etc have no relevance in this
>> scenario. LIkewise, if this functionality was provided via a minor mode, it
>> would also be possible for people to disable that mode and be left with just the
>> readme.org file, which is just plain text and quite readable.
>> 
>> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 
>
> The above is completely irrelevant to my questions.
>
>> Running Emacs wiht -q and opeing up an org file, I find the following 
>> 
>> - 236 org related key bindings - far less than 785 Of these, a number are
>> - actually remapping of existing bindings, so not new ones. This leaves 208.
>
> ??? How is remapping "not new"?  It takes some very common Emacs
> commands and redirects them to different commands.  E.g., 'open-line'
> now does something quite different.  This means the user should either
> go learn what the Org commands do, or be prepared to be surprised.

This are not new key bindings, they are redefinitions of existing key
bindings to make them work in an org mode context in a wayu which is
consistent with user expectations. They are not new in the sense of them
being a key binding which did not exist prior to loading org mode. 


>
>> I said that for a read only buffer, many of the org key bindings are not
>> relevant as they relate to features which are not pertinent to a read only org
>> buffer. Any bindings relating to babel, todo management, time management,
>> agendas etc have no relevance when reading a readme.org file.
>
> Then why does Org define them in that case?
>

because standard org mode is not typically used in read only buffers.
How org is defined now and how it would be defined/loaded when it comes
to a specialised use case as discussed here are completely different. As
I said in previous posts, I would expect that you would define an org
minor mode which would only include the key bindings which would be
appropriate for the use case of viewing a read-only org file. 

>> Really, the only
>> key bindings of any relevance are navigation related bindings (including
>> opening/closing folds, following links, etc). Removing all unnecessary org
>> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
>> a few could also be removed, but I've left them in as I only wanted to remove
>> those which would clearly be of no benefit in a read only buffer rendering of a
>> readme.org file. In an org minor mode setup for rendering of readme.org (and
>> possibly other read-only rendering of org files), I estimate at least another 30
>> bindings could be removed because they have no real benefit or are of only
>> marginal benefit. Creating an org minor mode for this purpose which only
>> introduced 20 or so new bindings would easily be possible. The key point is that
>> claims of 785 new bindings arfe incorrect and misleading. Even being very
>> conservative, we are really talking about 10% of the number being incorrectly claimed.  
>> if we introduced a minor mode for this, we are talking likely less than 20 or so.
>
> I'm sorry, but IMO this hand-waves away a real problem by ignoring the
> problematic UX of suddenly having more than 200 keybindings defined
> just because I visited a small file.  I hoped you will present some
> serious analysis of the issue, and perhaps even show me where I'm
> wrong in my conclusions.  Sadly, it sounds like all you wanted to do
> is to prove that you are right, the facts notwithstanding.

Sorry, but it seems you are the one trying to drag this off into a
different argument about whether org mode is too large or defines too
much or loads too much. As stated multiple times in my previous posts in
this thread, should org mode be used in the context being discussed, you
would define a new org minor mode whic only included the functionality
and key bindings relevant for this use case. You would not just use a
full org mode for this situation.

>
> You accuse Alan in extremism, but your own argumentation is another
> example of a similar extremism -- just in the opposite direction.
>

I have never said that 200 key bindings wasn't a lot. All I've said is
that 200 key bindings is a LOT less than the 784 being claimed. I went
further to state that in a specialised read-only org mode, you would be
able to reduce that number to around 50 or possibly even fewer. I also
reject what was stated about the org mode key bindings polluting the
global key space. I find no evidence of such pollution and in fact
believe org mode is very good at not doing so. I find very few (if any)
examples of org doing global key binding and in fact have to define new
personal bindings when I want global access to org functionality (such
as org capture or the org agenda).  

> This doesn't facilitate useful discussions of what I think is a real
> problem.  Any unbiased observer should agree that adding 200+ key
> bindings when a small file is visited is a problem that needs to be
> solved.  Org is too large for such simple jobs, and should be
> refactored to avoid loading everything under the sun when a README is
> visited.  And we should work towards solving it, not sweep it under
> the carpet.
>

which is exactly why I was saying that should you want org rendering of
small read only files, you would create an org minor mode which only
loaded the functionality and key bindings that would be relevant in that
context. It is only you who are assuming that a full org mode would be
used in this context. 

Nobody has tried to sweep anything under the carpet. Right back in
my very 1st post to this thread, I stated that should org be used to
render content in things like a help buffer or package description
whhich would presumably be a read only presentation, you would want an
org minor mode which only loads the functionality, including appropriate
key bindings which made sense in that context. This sentiment was also
mirrored by others in the thread. . 

Questions about general org mode and whether it defines too many key
bindings or too many features when loaded is a completely separate
question. While there could be some important discussion which needs to
happen in that context, it has little to do with what was being
discussed here and even less relevance to what I was saying, whihc was
that claiming opening an org file would define 784 new key bindings and
would pollute the key space was an exaggeration and over statement of
what happens when you open an org file. Even your own example confirmed
this.   

As to whether defining 200 key bindings is too many - well I don't think
you can possibly say either way without significantly more analysis. It
also isn't at all given that adding 200 new key bindings in a new mode
is of itself problematic. There could be very good reasons to add that
many bindings. Besides, I'm not sure it even is a problem. Emacs has
lots of key bindings - the vast majority of which I never use. I don't
find this a problem and I'm not convinced just citing absolute binding
numbers in itself is evidence of a problem. 

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Thu, 09 Jun 2022 05:34:09 +1000
> 
> > ??? How is remapping "not new"?  It takes some very common Emacs
> > commands and redirects them to different commands.  E.g., 'open-line'
> > now does something quite different.  This means the user should either
> > go learn what the Org commands do, or be prepared to be surprised.
> 
> This are not new key bindings, they are redefinitions of existing key
> bindings to make them work in an org mode context in a wayu which is
> consistent with user expectations.

That's exactly my point: suddenly commands that I'm familiar with are
working in ways that are different, even if slightly different.  A
conscientious Emacs user will want/need to study the differences,
before using those remapped commands.

> They are not new in the sense of them being a key binding which did
> not exist prior to loading org mode.

This completely misses the point I explain above.  The _commands_
behave differently, and that is all that counts.

> >> Any bindings relating to babel, todo management, time management,
> >> agendas etc have no relevance when reading a readme.org file.
> >
> > Then why does Org define them in that case?
> 
> because standard org mode is not typically used in read only buffers.

Which is one more argument towards more modularity in Org, so that
only the relevant bindings and commands are defined for each use case.

> As to whether defining 200 key bindings is too many - well I don't think
> you can possibly say either way without significantly more analysis.

Which is why I suggested you do this kind of analysis.

> It also isn't at all given that adding 200 new key bindings in a new
> mode is of itself problematic. There could be very good reasons to
> add that many bindings.

You have just explained that many of them are not relevant, even for
editing simple enough Org files.  For example, everything related to
org-babel -- why do these bindings have to be available by default?
Many Org files have no code blocks at all.

> Besides, I'm not sure it even is a problem. Emacs has lots of key
> bindings - the vast majority of which I never use. I don't find this
> a problem and I'm not convinced just citing absolute binding numbers
> in itself is evidence of a problem.

Well, I _am_ convinced, looking at this from the POV of the Emacs
maintainer.  I also think almost everyone agrees that too many
unneeded keybindings are a problem in general -- witness the removal
of some veteran global bindings in recent Emacs version, which is
evidence that we consider even half a dozen less-then-useful bindings
not a good thing.

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Thu, Jun 09, 2022 at 05:34:09 +1000, Tim Cross wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

[ .... ]

> >> Alan's claim was 785 additional key bindings, many of which not
> >> bound to C-c. 

It wasn't a "claim" it was a statement of fact.  These 784 bindings are
DOCUMENTED in org.info.  You attempted to explain it away by saying, at
first vaguely, that not all of them "come into play".  Only later did you
make that meaningful by saying that not all of them get loaded at any
time.

Many of these binding are not bound to C-c C-<letter> or other major mode
sequences.

> > The above is completely irrelevant to my questions.

> >> Running Emacs wiht -q and opeing up an org file, I find the following 

> >> - 236 org related key bindings - far less than 785.

236 and 784 are both good approximations to infinity when it comes to key
binding numbers.  Even 236 is barely manageable for anybody wanting to
learn the mode in the way I would.

[ .... ]

> > Then why does Org define them [lots of key bindings] in that case?

> because standard org mode is not typically used in read only buffers.
> How org is defined now and how it would be defined/loaded when it comes
> to a specialised use case as discussed here are completely different.
> As I said in previous posts, I would expect that you would define an
> org minor mode which would only include the key bindings which would be
> appropriate for the use case of viewing a read-only org file. 

I think the use case is that of a small org file rather than a read-only
one.  For some value of "small".

[ .... ]

> > I'm sorry, but IMO this hand-waves away a real problem by ignoring the
> > problematic UX of suddenly having more than 200 keybindings defined
> > just because I visited a small file.  I hoped you will present some
> > serious analysis of the issue, and perhaps even show me where I'm
> > wrong in my conclusions.  Sadly, it sounds like all you wanted to do
> > is to prove that you are right, the facts notwithstanding.

> Sorry, but it seems you are the one trying to drag this off into a
> different argument about whether org mode is too large or defines too
> much or loads too much.

That has been the main topic of this discussion for several rounds in the
thread.  I've found your responses very frustrating, feeling like you've
been broad brushing precise topics with vagueness, and like I'm dealing
with a politician rather than a hacker.  In a typical thread in this
mailing list, pin-point accuracy is rare, but participants attempt to
answer what the other person means rather than the precise literal
meaning of what was said (sometimes getting it wrong).  I don't feel this
has been happening in this part of this thread.

> As stated multiple times in my previous posts in this thread, ....

And overbearing with it.  Why?

> .... should org mode be used in the context being discussed, you would
> define a new org minor mode whic only included the functionality and
> key bindings relevant for this use case. You would not just use a full
> org mode for this situation.

The context being discussed is, I think, "small" org files rather than
read-only ones, whatever small means here.  I think we're all agreed that
full org mode is inappropriate for these.

> > You accuse Alan in extremism, but your own argumentation is another
> > example of a similar extremism -- just in the opposite direction.

> I have never said that 200 key bindings wasn't a lot. All I've said is
> that 200 key bindings is a LOT less than the 784 being claimed.

It's not a LOT less, both being unmanageably large.

> I went further to state that in a specialised read-only org mode, you
> would be able to reduce that number to around 50 or possibly even
> fewer. I also reject what was stated about the org mode key bindings
> polluting the global key space. I find no evidence of such pollution
> ....

I find plenty of evidence in org.info.

> and in fact believe org mode is very good at not doing so.

Go into org.info and count up all the "new" (as you have defined it
above) bindings which don't begin C-c C-<letter> or other major mode
sequences.  There are more of these than the total number of bindings a
typical mode has.

Examples:

* C-c /:                                 Sparse Trees.
* M-DOWN:                                Structure Editing.


> I find very few (if any) examples of org doing global key binding and
> in fact have to define new personal bindings when I want global access
> to org functionality (such as org capture or the org agenda).  

> > This doesn't facilitate useful discussions of what I think is a real
> > problem.  Any unbiased observer should agree that adding 200+ key
> > bindings when a small file is visited is a problem that needs to be
> > solved.  Org is too large for such simple jobs, and should be
> > refactored to avoid loading everything under the sun when a README is
> > visited.  And we should work towards solving it, not sweep it under
> > the carpet.

> which is exactly why I was saying that should you want org rendering of
> small read only files, you would create an org minor mode which only
> loaded the functionality and key bindings that would be relevant in that
> context. It is only you who are assuming that a full org mode would be
> used in this context. 

I think Eli has experienced this.  He is not "assuming" it at all.

> Nobody has tried to sweep anything under the carpet. Right back in
> my very 1st post to this thread, I stated that should org be used to
> render content in things like a help buffer or package description
> whhich would presumably be a read only presentation, you would want an
> org minor mode which only loads the functionality, including appropriate
> key bindings which made sense in that context. This sentiment was also
> mirrored by others in the thread. . 

> Questions about general org mode and whether it defines too many key
> bindings or too many features when loaded is a completely separate
> question.

Not at all.  Without org mode's massive numbers of bindings (several
hundred), the question of refactoring it wouldn't even have been an
issue.

> While there could be some important discussion which needs to happen in
> that context, it has little to do with what was being discussed here
> and even less relevance to what I was saying, which was that claiming
> opening an org file would define 784 new key bindings ....

That was not claimed.  The thing remarked upon was that there are 784 key
bindings in org mode.

[ .... ]

> As to whether defining 200 key bindings is too many - well I don't think
> you can possibly say either way without significantly more analysis. It
> also isn't at all given that adding 200 new key bindings in a new mode
> is of itself problematic.

Problems with it have been identified in this thread.

> There could be very good reasons to add that many bindings. Besides,
> I'm not sure it even is a problem. Emacs has lots of key bindings - the
> vast majority of which I never use. I don't find this a problem and I'm
> not convinced just citing absolute binding numbers in itself is
> evidence of a problem. 

Maybe not, but the problems are there nonetheless.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Alan Mackenzie <acm@muc.de> writes:

>> >> Alan's claim was 785 additional key bindings, many of which not
>> >> bound to C-c. 
>
> It wasn't a "claim" it was a statement of fact.  These 784 bindings are
> DOCUMENTED in org.info.  You attempted to explain it away by saying, at
> first vaguely, that not all of them "come into play".  Only later did you
> make that meaningful by saying that not all of them get loaded at any
> time.
>
> Many of these binding are not bound to C-c C-<letter> or other major mode
> sequences.

Please note that not only C-c C-<letter> are dedicated to major modes.
Also, C-c <non-letter> (though not all C-c <non-letter> are
recommended).

I vent through all the bindings we define in the manual and the only
bindings that do not use reserved major-mode bindings space are:
C-'
C-,
<control character/s>-<arrow>
<control character/s>-<RET>

Out of the above bindings only C-S-<up>/<down> override the default
bindings. But only when shift-select-mode is enabled (which had not been
the default when these Org bindings were introduced).

Other bindings might cause unexpected behavior when user has personal
bindings. However, please note that Emacs recommends binding
C-c <letter> for user space. Anything else is a "gray" zone.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Tue, Jun 07, 2022 at 16:14:52 +1000, Tim Cross wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

> >> From: Tim Cross <theophilusx@gmail.com>
> >> Cc: emacs-devel@gnu.org
> >> Date: Mon, 06 Jun 2022 23:57:55 +1000

> >> > There are 784 key bindings in org mode.  How can you say that this isn't
> >> > complex and difficult to learn?

> >> Very simply because the vast majority of  those key bindings only come
> >> into play when you use advanced features of org mode, such as the
> >> agenda or table editing or noweb mode. If your just using basic org mode
> >> features, very very few of those key bindings even come into play. Just
> >> go throgh that list and remove any bindings relating to agenda mode,
> >> table editing mode, source block modes, clock table mode, list editing
> >> mode, etc and you end up with very few key bindings (all of which are
> >> udner the C-c prefix, unlike your previous claim). 

> > How about if you do this exercise and walk us through the results?
> > Visit an Org file, type "C-h b", then show us the list, and tell which
> > bindings you think are "basic" and which are "advanced", and why you
> > think so.

> > I can tell you what I see with my very simple 1570-line Org file,
> > which just has some todo items organized in 4-level hierarchy:

> >   . ~100 keys that look "general Org" to me
> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
> >     'backward-sentence'
> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
> >     bound by default, but they are there
> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
> >     but they are all bound by default, just because I visited an Org
> >     file 

> > So "just" 230 key bindings.  And this is without any minor/add-on Org
> > mode/feature enabled, at least according to "C-h m".

> > Do you see something different?  Are you still saying that it's not a
> > lot, or that it's "based on ignorance and paranoia"?  If so, please
> > point out where I'm ignorant and/or paranoiac, because I'd really like
> > to know.

> I think Alan's response has pretty much confirmed what I was saying. Alan has
> made it fairly clear that his real underlying concern is about a stealthy
> strategy to get org mode more intrinsically tied into Emacs and in the end,
> making it so critical to normal operation that everyone will be forced to learn
> org mode regardless. I don't necessarily agree with such an argument, but think
> having a debate around that would be more up-front and in some ways honest
> compared to what appear to be somewhat contrived alternative arguments about
> excessive key bindings and added complexity.

You've misread the situation.  My objections to org mode are almost
entirely due to its excessive complexity as measured by the number of key
bindings.

[ .... ]

> With respect to ignorance, Alan made it quite clear he has not spent
> the time or effort to learn org mode, so he is ignorant about its
> operation and lacks any real understanding of its operation.

This is indeed the case.

> He has made assertions about the number of key bindings which are
> wildly over inflated .....

You have said this several times in your post without any justification.
There are 784 entries in the key index in org.info.  You're saying it's
"wildly over inflated" to use this number, at least as a first estimate.
Why are you making such a counter intuitive claim without any
justification? 

Eli counted around 230 bindings from C-h m.  It thus seems likely that on
average, an org mode key sequence is used for 3½ distinct functions.
This is not a good thing.

> .... and about required levels of complexity which simply would not be
> necessary with the org mode rendering of a read only buffer.

You're not saying that loading a RO readme.org would somehow only load a
small part of org mode, surely?

> In short, he has made a lot of assumptions about both the key bindings
> and complexity based on only a fairly cursory scan of the manual and
> very limited experience. 

And now you're trying to be patronising, but failing.  ;-)

Why do you make no attempt to explain why my assumptions are invalid?

> All that appears to have occurred here is that someone noted that
> packages often had a readme.org file and wondered if this should be
> translated into just a plain readme file. Someone else suggested we
> could have package descriptions render the readme.org file using org
> and not require export into plain readme file and mentioned some
> possible advantages this could have wrt navigation, rendering of
> tables, links and images or inclusion of source code snippets, examples
> etc.

> From this point, some somewhat extreme claims and arguments have been
> thrown in.  One was Alan's claim that org would introduce 785 new key
> bindings, many of which not bounmd to C-c and which would presumably
> either pollute the key space or conflict with existing bindings. All I
> have done is point out this claim was inaccurate and overstated due to
> an overly simplistic analysis of what is in the key index of the org
> manual.

You have made an unfounded assertion of my inaccuracy.  Are you
suggesting, perhaps, that of these 784 entries in org.info's key index,
the vast bulk are dummies?  Some of these bindings (I don't think I used
the word "many", but may have done) are not in the standard major mode
space C-c C-<letter>.  Some of these clash with existing bindings,
examples of which I gave last night.

> it should be noted that if we were to add 'native' rendering of readme.org
> files, this would likely be done with an org minor mode rather than a full blown
> org mode as very little of the overall org functionality would be required in
> this situation.

In other words, an attempt would be made to fix the problems which I am
drawing to people's attention.

> All that would be required is org rendering support and org support for
> navigation (folding/unfolding, following links and possibly list/table
> navigation). All the advanced org functionality relating to babel,
> noweb, todo and time management, agendas, etc have no relevance in this
> scenario. LIkewise, if this functionality was provided via a minor
> mode, it would also be possible for people to disable that mode and be
> left with just the readme.org file, which is just plain text and quite
> readable.

> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Actually it was 784, 28².  Again, why is this number invalid?

> Running Emacs wiht -q and opeing up an org file, I find the following 

> - 236 org related key bindings - far less than 785 Of these, a number are
> - actually remapping of existing bindings, so not new ones. This leaves 208.

> I said that for a read only buffer, many of the org key bindings are not
> relevant as they relate to features which are not pertinent to a read only org
> buffer. Any bindings relating to babel, todo management, time management,
> agendas etc have no relevance when reading a readme.org file. Really, the only
> key bindings of any relevance are navigation related bindings (including
> opening/closing folds, following links, etc). Removing all unnecessary org
> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
> a few could also be removed, but I've left them in as I only wanted to remove
> those which would clearly be of no benefit in a read only buffer rendering of a
> readme.org file. In an org minor mode setup for rendering of readme.org (and
> possibly other read-only rendering of org files), I estimate at least another 30
> bindings could be removed because they have no real benefit or are of only
> marginal benefit. Creating an org minor mode for this purpose which only
> introduced 20 or so new bindings would easily be possible.

You're describing what might be rather than what is.

> The key point is that claims of 785 new bindings are incorrect and
> misleading.

I resent you continally repeating this unfounded assertion.  There are
784 entries in org.info's key index.  That is the current state.  That is
the truth.  How can that be "incorrect"?  Even if 750 of the 784 bindings
somehow "don't come into play", they still exist, and are still
problematic.  Are you saying that only every tenth entry actually
signifies an actual binding, or something like that?

> Even being very conservative, we are really talking about 10% of the
> number being incorrectly claimed.  if we introduced a minor mode for
> this, we are talking likely less than 20 or so.

> Arguments regarding this being a 'slippery slope' or 'edge of the
> wedge' in an org takeover are a completely different argument and there
> may well be some concerns which may need to be considered. However, if
> that is the real concern, then argue based on that rather than
> exaggerated claims about excessive key bindings, key binding
> polution/conflicts and excessive complexity.

If it weren't for org mode's excessive key bindings and complexity, the
"takeover scenario" would scarcely be a worry.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Stefan Monnier]

Rather than try and figure out who's right or wrong, I hope we can take
advantage of this discussion to get some code out that try to solve
the problem.

AFAICT the problem seen from Emacs, is that Org is large (even for
a basic uses, which occasionally leads to high load times) and that it
doesn't follow all the usual Emacs conventions, such as
overriding/remapping standard key-bindings (the resulting behaviors may
sometimes be similar to the standard one, but even when that's the case
the redefinition can easily bite the user).

The problem seen from Org is that Emacs doesn't use Org enough :-)
[ This paragraph's shorter length probably reflects the fact that I'm
  less familiar with Org than with Emacs.  ]

I think the way forward is to define a "basic org-mode".  This one could
be used at many more places where there's currently an occasional desire
to use Org that's resisted because of the above problems.
Ideally, org-mode would then be defined as an extension of this "basic
org-mode".  Also ideally some of those extensions would be reworked so
they can be used "Emacs-wide" rather than only in Org (obviously, that
can only make sense for some of them).

We could start this "basic org-mode" as a trivial copycat of
`outline-mode` and then start adding Org features to it.  The driving
constraint is to keep it lightweight and rules-abiding enough that there
won't be any resistance to using it, while at the same time making sure
that the features added to it can be removed from the "org-mode
extension", so that org-mode and "basic org-mode" don't diverge.


        Stefan

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.

I think that people that use Org mode wouldn't be very interested in
that, and people that don't use Org can just continue editing those
files with `M-x find-file-literally', really.  Doing so is fine.

But I don't understand the discussion this thread warped out of.  When
we display a README from Package, we shouldn't be showing the raw text
of README.org or README.md or README.html, lightly fontified -- we
should be showing a rendering of it.  People that read the README aren't
supposed to edit it, after all.

Which would make the 40K key bindings that Org apparently has completely
irrelevant for non-Org users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs

[Stefan Monnier]

> I think that people that use Org mode wouldn't be very interested in
> that, and people that don't use Org can just continue editing those
> files with `M-x find-file-literally', really.  Doing so is fine.

I was thinking of files like etc/NEWS where having "a bit more than
outline-mode" would be welcome.

> But I don't understand the discussion this thread warped out of.  When
> we display a README from Package, we shouldn't be showing the raw text
> of README.org or README.md

Why not?  If it can be prettified on the fly by font-lock?

Separating "viewing" from "editing" is un-Emacsish, because it puts
a barrier between the "consumers" and the "producers" whereas Emacs
likes to try and get end-users exposed to the source as mush as possible
so it takes little effort for them to change role.


        Stefan

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 14:48:31 -0400
> 
> > But I don't understand the discussion this thread warped out of.  When
> > we display a README from Package, we shouldn't be showing the raw text
> > of README.org or README.md
> 
> Why not?

Because it's butt-ugly!

Re: Org mode and Emacs

[Stefan Monnier]

>> > But I don't understand the discussion this thread warped out of.  When
>> > we display a README from Package, we shouldn't be showing the raw text
>> > of README.org or README.md
>> Why not?
> Because it's butt-ugly!

I don't think it has to be.


        Stefan

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But I don't understand the discussion this thread warped out of.  When
>> we display a README from Package, we shouldn't be showing the raw text
>> of README.org or README.md
>
> Why not?  If it can be prettified on the fly by font-lock?
>
> Separating "viewing" from "editing" is un-Emacsish, because it puts
> a barrier between the "consumers" and the "producers" whereas Emacs
> likes to try and get end-users exposed to the source as mush as possible
> so it takes little effort for them to change role.

That's true, but on the other hand, we don't show texinfo files to
people for a good reason.  That's about as "plain text" as normal Org
files, or as some Markdown files get.  (Most Markdown files are kinda
pleasant, but there's plenty of scope for unreadability -- it drops down
into HTML for tables, for instance.)

Or take doc strings as an example -- it's always been a markup language,
too (\\ and \\[] and \\<>), but it's moving more clearly in that
direction -- because we want to make the documentation we present the
user pretty and easy to read.  And we have one-key commands to take us
to the source code, which we could have in the Package README rendering,
too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Rather than try and figure out who's right or wrong, I hope we can take
> advantage of this discussion to get some code out that try to solve
> the problem.
>
> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).
>
> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.
>
>

Thanks Stefan and I agree. I also think this is in general exactly the
direction current org maintainers/developers are taking. 

At this point, considerable work is being done (by Ihor and others) to 

- Implement a concise and efficient parser. This has also involved
  refining and clarifying org syntax, which has had some holes and
  ambiguities. 

- Replace org's largely regexp based font locking mechanism with one
  based on the parsing and tagging made possible by the org parser. This
  should improve both performance and accuracy in parsing. 

- Improving efficiency of folding etc

- Improving efficiency with respect to larger org files through the use
  of caching etc. 

- Increasing org's use of built-in Emacs capabilities rather than using
  org specific implementations. For example, adopting transient instead
  of an org implemented module which replicates similar functionality. 

- Increase modularity an enable loading of the functionality that you
  want. 

In particular, the important work being done by Ihor wrt folding,
parsing and caching will provide a core of functionality which can be
used to define something like an org minor mode which could be used in
those situations where we want some basic org functionality, like
folding, navigation and formatting/presentation, but we don't need all
the additional advanced features, such as babel, todo and time
management, spreadsheets, tables and table formulas etc. 

Of course, to what extent Emacs will/should leverage off org mode is a
completely different debate. 

Re: Org mode and Emacs

[Visuwesh]

[புதன் ஜூன் 08, 2022] Tim Cross wrote:

> [...]
> - Increasing org's use of built-in Emacs capabilities rather than using
>   org specific implementations. For example, adopting transient instead
>   of an org implemented module which replicates similar functionality. 

Where can I read more about this?  I see it being mentioned a few times
in the org-mode mailing list, and in the matrix room.  Is the plan to
completely remove the org-mks interface and replace it with transient
without having an option to use the former?  I find org-mks perfectly
fine to use and would be sad if it was replaced with transient.
Transient needs time to get used to, and the default settings is quite
un-Emacsy; I'm not too excited about configuring yet another package
that has a hard-to-understand manual TBH.

Re: Org mode and Emacs

[Tim Cross]

Visuwesh <visuweshm@gmail.com> writes:

> [புதன் ஜூன் 08, 2022] Tim Cross wrote:
>
>> [...]
>> - Increasing org's use of built-in Emacs capabilities rather than using
>>   org specific implementations. For example, adopting transient instead
>>   of an org implemented module which replicates similar functionality. 
>
> Where can I read more about this?  I see it being mentioned a few times
> in the org-mode mailing list, and in the matrix room.  Is the plan to
> completely remove the org-mks interface and replace it with transient
> without having an option to use the former?  I find org-mks perfectly
> fine to use and would be sad if it was replaced with transient.
> Transient needs time to get used to, and the default settings is quite
> un-Emacsy; I'm not too excited about configuring yet another package
> that has a hard-to-understand manual TBH.


The specifics of what is planned are still being worked out. Initially,
the likely initial candidate for change will be to the export menu.
Unless you have extensive low level customisation, it should not be a
change which has significant impact on users. In fact, maintaining
backwards compatibility and consistency for end users is important to
the org developers. It is also quite possible that after an initial
investigation, it may be decided transient is not a good option for org
mode or perhaps it will be a good option once additonal functionality or
enhancements are added. Right now, all that has been agreed is that it
would be worthwhile looking at it to see if it can be of benefit in
helping to reduce org maintenance overheads and/or increase org's
consistency with other emacs packages. 

What will determine what remains and what choices are available will
depend on what involvement people have in the development and what will
be maintainable. There are a number of areas in org mode where
functionality has been implemented that is 80+% equivalent to
functionality which exists or has been added to core emacs. Having this
duplication of effort is adding to the burden of maintenance, which is
already significant. In general, org maintainers keep an eye on what is
being added/expanded in Emacs core and when things are added, like
transient mode, they are assessed to see if adopting that functionality
would reduce the org maintenance load and improve org's consistency
withi the rest of Emacs. At this stage, transient has been added as
something to look at in the backlog. When this task makes if off the
backlog, the typical process would be for it to be discussed on the org
devel list, for initial implementations to be done either in its own
branch (if considered a significantly large enough change) or on the
development branch otherwise and people will be asked to try it out. 

So, if your interested in this area, the first thing would be to get on
the org devel mail list. Anyone who is particularly keen to see such
things added might also initiate discussions and development in their
own branch, which could then be added in as a PR. However, like Emacs,
any significant development work on org mode also requires FSF copyright
assignment. 

At this specific point in time, the main focus with org has been on
stability, performance improvements, especially for large org files with
many babel blocks and sections and improvements to the org syntax and provision of an
org parser,  

In particular, I think the work being done by Ihor on folding and
the org parser will have huge benefits. In addition to making the code
base easier to maintain and improving performance, it will help reduce
org's current dependency on complicated and difficult to maintain
regular expressions in the font-locking layer. This should improve
performance and reduce errors or unexpected results that sometimes occur
because of regexp errors etc and reduce time spent by maintainers in
tracking down such errors and maintenance of those complex regexps.   

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we are going to redesign Org mode syntax,
can we please add room for extensibility
so that an extension of Org mode syntax
could have all the features of Texinfo?

If the new syntax can express all the distinctions that Texinfo
can already express, that would make it possible to switch to it
for our documentation sources.

This needs to include conversion into some variant of TeX syntax so
that we can generate high-quality printed manuals through TeX.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?
>
> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.


Nobody has said that org mode syntax was getting redesigned. What I said
was that we were working to clarify the existing syntax and remove
ambiguities and provide a standard parser. 

Making org mode syntax equivalent to texinfo syntax seems like a
mistake to me. The original idea was to have a light weight syntax which
is easyh to learn, not create a clone of texinfo. Besides, I suspect it
would be very difficult to maintain backwards compatibility. 

I"m also not convinced that org mode would be the right solution for
Emacs documentation. That is a completely different conversation and I'm
not convinced that org mode is a better answer wrt documentation than
texinfo. While there are certainly some areas where texinfo is showing
its age, it still works well for providing Emacs documentation within
the editor. There may be some role for org mode to augment what is
provided by tgexinfo, but again, that is a whole different conversation.
Note also that the previous thread was about handling of existing org
files within Emacs package ecosystem, not about replacing org
documentation with org mode. 

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 09:10:50 +1000
> 
> I"m also not convinced that org mode would be the right solution for
> Emacs documentation.

Then why Org's own manual is written in Org instead of Texinfo?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I"m also not convinced that org mode would be the right solution for
>> Emacs documentation.
>
> Then why Org's own manual is written in Org instead of Texinfo?

I do not know if Org mode would be a good solution for Emacs
documentation. However, compared to texinfo, our manual is lacking index
entries in all the exported versions but texinfo.

The indexes in pdf version of the manual are actually generated by
org -> texinfo -> tex -> pdf.

AFAIK.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 14:20:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I"m also not convinced that org mode would be the right solution for
> >> Emacs documentation.
> >
> > Then why Org's own manual is written in Org instead of Texinfo?
> 
> I do not know if Org mode would be a good solution for Emacs
> documentation. However, compared to texinfo, our manual is lacking index
> entries in all the exported versions but texinfo.

Lack of an index makes for a less useful manual, IME.

Anyway, is the lack of the index the _only_ missing feature from what
the Org markup provides as compared to Texinfo?  Tim's message sounded
like there are many more omissions.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Fri, 10 Jun 2022 14:20:37 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I"m also not convinced that org mode would be the right solution for
>> >> Emacs documentation.
>> >
>> > Then why Org's own manual is written in Org instead of Texinfo?
>> 
>> I do not know if Org mode would be a good solution for Emacs
>> documentation. However, compared to texinfo, our manual is lacking index
>> entries in all the exported versions but texinfo.
>
> Lack of an index makes for a less useful manual, IME.
>
> Anyway, is the lack of the index the _only_ missing feature from what
> the Org markup provides as compared to Texinfo?  Tim's message sounded
> like there are many more omissions.

My statement is simply waht it says. I am not convinced org mode would
be the right solution for Emacs documentation. This is for many reasons,
some relating to org, some relating to the fact I'm not convinced there
is a need to replace texinfo, especially given that in some areas, I
think texinfo is more feature rich and consistent and some relating to the feeling I have
that the result is hard to justify given the amount of work it would
require. At the end of the day, proposing org mode as a replacement for
texinfo feels too much like a solution looking for a problem and one
that doesn't appear to have clear enough advantages to justify the work
needed to make such a transition, especially given work being done in
the texinfo area to improve its support for new formats, like epub. 

Org mode is a fantastic tool and I use it every day. There are ways org
mode could be used to augment the Emacs environment, but I don't think
it is an all or nothing situation. I'm also not convinced that embedding
org into Emacs more and more is necessarily a good thing. Tighter
coupling comes with both pros and cons.  

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: Ihor Radchenko <yantar92@gmail.com>, rms@gnu.org,
>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 14:49:21 +1000
> 
> >> >> I"m also not convinced that org mode would be the right solution for
> >> >> Emacs documentation.
> >> >
> >> > Then why Org's own manual is written in Org instead of Texinfo?
> >> 
> >> I do not know if Org mode would be a good solution for Emacs
> >> documentation. However, compared to texinfo, our manual is lacking index
> >> entries in all the exported versions but texinfo.
> >
> > Lack of an index makes for a less useful manual, IME.
> >
> > Anyway, is the lack of the index the _only_ missing feature from what
> > the Org markup provides as compared to Texinfo?  Tim's message sounded
> > like there are many more omissions.
> 
> My statement is simply waht it says. I am not convinced org mode would
> be the right solution for Emacs documentation. This is for many reasons,
> some relating to org, some relating to the fact I'm not convinced there
> is a need to replace texinfo, especially given that in some areas, I
> think texinfo is more feature rich and consistent and some relating to the feeling I have
> that the result is hard to justify given the amount of work it would
> require.

Well, it is hard to say anything intelligent without knowing those
"many reasons".

And if your POV is shared by the Org developers, then I ask again: why
is the Org manual, which weighs in at 23,000 lines, is being written
in Org?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: Ihor Radchenko <yantar92@gmail.com>, rms@gnu.org,
>>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
>> Date: Sat, 11 Jun 2022 14:49:21 +1000
>> 
>> >> >> I"m also not convinced that org mode would be the right solution for
>> >> >> Emacs documentation.
>> >> >
>> >> > Then why Org's own manual is written in Org instead of Texinfo?
>> >> 
>> >> I do not know if Org mode would be a good solution for Emacs
>> >> documentation. However, compared to texinfo, our manual is lacking index
>> >> entries in all the exported versions but texinfo.
>> >
>> > Lack of an index makes for a less useful manual, IME.
>> >
>> > Anyway, is the lack of the index the _only_ missing feature from what
>> > the Org markup provides as compared to Texinfo?  Tim's message sounded
>> > like there are many more omissions.
>> 
>> My statement is simply waht it says. I am not convinced org mode would
>> be the right solution for Emacs documentation. This is for many reasons,
>> some relating to org, some relating to the fact I'm not convinced there
>> is a need to replace texinfo, especially given that in some areas, I
>> think texinfo is more feature rich and consistent and some relating to the feeling I have
>> that the result is hard to justify given the amount of work it would
>> require.
>
> Well, it is hard to say anything intelligent without knowing those
> "many reasons".
>
> And if your POV is shared by the Org developers, then I ask again: why
> is the Org manual, which weighs in at 23,000 lines, is being written
> in Org?

I have never claimed my POV was that of the org developers, I've never
claimed to be an org developer or to speak in any way for org
developers. 

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: yantar92@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca, acm@muc.de,
>  emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 17:59:46 +1000
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> My statement is simply waht it says. I am not convinced org mode would
> >> be the right solution for Emacs documentation. This is for many reasons,
> >> some relating to org, some relating to the fact I'm not convinced there
> >> is a need to replace texinfo, especially given that in some areas, I
> >> think texinfo is more feature rich and consistent and some relating to the feeling I have
> >> that the result is hard to justify given the amount of work it would
> >> require.
> >
> > Well, it is hard to say anything intelligent without knowing those
> > "many reasons".
> >
> > And if your POV is shared by the Org developers, then I ask again: why
> > is the Org manual, which weighs in at 23,000 lines, is being written
> > in Org?
> 
> I have never claimed my POV was that of the org developers, I've never
> claimed to be an org developer or to speak in any way for org
> developers. 

I didn't say you have.  The above question was aimed at the Org
developers, as should be obvious from the context; apologies if that
wasn't clear enough.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> And if your POV is shared by the Org developers, then I ask again: why
> is the Org manual, which weighs in at 23,000 lines, is being written
> in Org?

The original reason is simply

https://list.orgmode.org/AANLkTin=gUscE4znd4pYY85X9d2wLkC+JADVcDN8AeoJ@mail.gmail.com/
>>> I was just wondering... Is the manual written in texinfo markup, or is there
>>> some obscure .org file behind the manual still?
>>> 
>>> If it really is written in texinfo, is this not a shortcoming? Org mode is
>>> capable of generating html and pdf etc. Why not use it for the manual then
>>> to set the example and show its powers!?

and then

https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
>>> It's a real pleasure to know that Org can now eat its own dog food (and
>>> to see Carsten's memorable phrase on the Org mode list again). Many
>>> thanks for picking up this orphaned project. I look forward to the day
>>> it serves as the source for org.texi.

So, now we have our manual written in Org mode and we never had reasons
to come back to texi.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 17:05:02 +0800
> 
> https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
> >>> It's a real pleasure to know that Org can now eat its own dog food (and
> >>> to see Carsten's memorable phrase on the Org mode list again). Many
> >>> thanks for picking up this orphaned project. I look forward to the day
> >>> it serves as the source for org.texi.
> 
> So, now we have our manual written in Org mode and we never had reasons
> to come back to texi.

And I get to wait for 2 minutes for the build to finish each time
org.org is touched.  Right.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> So, now we have our manual written in Org mode and we never had reasons
>> to come back to texi.
>
> And I get to wait for 2 minutes for the build to finish each time
> org.org is touched.  Right.

This is not very polite. The change did not aim to make your life
harder.

You are free to open discussion in Org ML about turning back to texi
format, but it will, for example, make things harder for me and other
people who are not familiar with texinfo format. I am not sure if build
time justifies extra maintenance burden.

I just pushed several improvements to ox.el. They reduce manual
generation time 2x on my system (using main branch). Feel free to try it
on your side. AFAIU, the effect should be more noticeable on slower
systems.

It may also help if you try to profile org-make-manuals from
mk/org-fixup.el and share the results.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:04:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So, now we have our manual written in Org mode and we never had reasons
> >> to come back to texi.
> >
> > And I get to wait for 2 minutes for the build to finish each time
> > org.org is touched.  Right.
> 
> This is not very polite. The change did not aim to make your life
> harder.

It is an expression of frustration I feel each time the build needs to
rebuild the Org manual.  I wrote it to contrast that frustration with
what I perceived as satisfaction with this solution expressed in your
description of the history of this.

I really wonder how come no one on the Org list paid any attention to
the 10-fold to 40-fold slowdown in the time it takes to build the
manual, as result of that change.  But that's water under the bridge.

> You are free to open discussion in Org ML about turning back to texi
> format, but it will, for example, make things harder for me and other
> people who are not familiar with texinfo format. I am not sure if build
> time justifies extra maintenance burden.

I don't see any chance I will convince Org developers to go back to
Texinfo at this point.

> I just pushed several improvements to ox.el. They reduce manual
> generation time 2x on my system (using main branch). Feel free to try it
> on your side. AFAIU, the effect should be more noticeable on slower
> systems.

Thanks, I hope to see this soon in the Emacs repository.

> It may also help if you try to profile org-make-manuals from
> mk/org-fixup.el and share the results.

If profiling can help, wouldn't it be simpler to invoke the same
commands from an interactive Emacs session, then show the profile?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> It is an expression of frustration I feel each time the build needs to
> rebuild the Org manual.  I wrote it to contrast that frustration with
> what I perceived as satisfaction with this solution expressed in your
> description of the history of this.

That's not my satisfaction. I quoted the emails.
In any case, while I understand your frustration, you managed to spread
it to me at least. It did not make parsing this thread any easier.

> I really wonder how come no one on the Org list paid any attention to
> the 10-fold to 40-fold slowdown in the time it takes to build the
> manual, as result of that change.  But that's water under the bridge.

We rarely have bugs related to manual builds. I recall two in many
years.

Usually documentation is built automatically on ELPA and by our
publishing scripts on orgmode.org.

>> I just pushed several improvements to ox.el. They reduce manual
>> generation time 2x on my system (using main branch). Feel free to try it
>> on your side. AFAIU, the effect should be more noticeable on slower
>> systems.
>
> Thanks, I hope to see this soon in the Emacs repository.

Not soon. Unless you want major changes for Emacs 28.2. We restricted
stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

>> It may also help if you try to profile org-make-manuals from
>> mk/org-fixup.el and share the results.
>
> If profiling can help, wouldn't it be simpler to invoke the same
> commands from an interactive Emacs session, then show the profile?

This is exactly what I meant. To run org-make-manuals from interactive
Emacs session.

org-make-manuals takes about 20 seconds on my system (for combined Org
export and texinfo invocation). Your system is clearly different and
might have different bottlenecks.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I really wonder how come no one on the Org list paid any attention to
> > the 10-fold to 40-fold slowdown in the time it takes to build the
> > manual, as result of that change.  But that's water under the bridge.
> 
> We rarely have bugs related to manual builds. I recall two in many
> years.
> 
> Usually documentation is built automatically on ELPA and by our
> publishing scripts on orgmode.org.

So basically no one builds Org, including the manual, on their own
system?  Even not the Org developers?

> >> I just pushed several improvements to ox.el. They reduce manual
> >> generation time 2x on my system (using main branch). Feel free to try it
> >> on your side. AFAIU, the effect should be more noticeable on slower
> >> systems.
> >
> > Thanks, I hope to see this soon in the Emacs repository.
> 
> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

This is not needed for the emacs-28 branch, so I meant master.

> >> It may also help if you try to profile org-make-manuals from
> >> mk/org-fixup.el and share the results.
> >
> > If profiling can help, wouldn't it be simpler to invoke the same
> > commands from an interactive Emacs session, then show the profile?
> 
> This is exactly what I meant. To run org-make-manuals from interactive
> Emacs session.

Then why would I need org-fixup.el?

> org-make-manuals takes about 20 seconds on my system (for combined Org
> export and texinfo invocation). Your system is clearly different and
> might have different bottlenecks.

Your Emacs is probably built with optimizations, whereas mine isn't.
The optimized version build org.info in about 30 sec, as I said, which
is not very different from your timing.

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Ihor Radchenko <yantar92@gmail.com> Cc:
    >> theophilusx@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca,
    >> acm@muc.de, emacs-devel@gnu.org Date: Sun, 12 Jun 2022 18:38:45
    >> +0800
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > I really wonder how come no one on the Org list paid any
    >> attention to > the 10-fold to 40-fold slowdown in the time it
    >> takes to build the > manual, as result of that change.  But
    >> that's water under the bridge.
    >> 
    >> We rarely have bugs related to manual builds. I recall two in
    >> many years.
    >> 
    >> Usually documentation is built automatically on ELPA and by our
    >> publishing scripts on orgmode.org.

    > So basically no one builds Org, including the manual, on their own
    > system?  Even not the Org developers?

I build org-mode from git and have noticed the slowness in building the
documentation.

Weekly building from git is my way of avoiding having to deal
immediately with a myriad of changes in past Versions of org-mode.

Best wishes.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> We rarely have bugs related to manual builds. I recall two in many
>> years.
>> 
>> Usually documentation is built automatically on ELPA and by our
>> publishing scripts on orgmode.org.
>
> So basically no one builds Org, including the manual, on their own
> system?  Even not the Org developers?

Org releases are not frequent for most users and occasional build time
should not matter too much. Nobody complained on the list.

Org developers (at least, me) usually run make test. It does not touch
documentation. Full re-build is rarely needed because the documentation
does not change often.

>> > Thanks, I hope to see this soon in the Emacs repository.
>> 
>> Not soon. Unless you want major changes for Emacs 28.2. We restricted
>> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
>
> This is not needed for the emacs-28 branch, so I meant master.

I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
master only tracks our stable tagged releases. Not the working branches
(both bugfix and main). I do not know the motivation behind it and I do
not know if it is a good idea to change it. Do you think that Emacs
master should instead track our Org dev branch?

>> >> It may also help if you try to profile org-make-manuals from
>> >> mk/org-fixup.el and share the results.
>> >
>> > If profiling can help, wouldn't it be simpler to invoke the same
>> > commands from an interactive Emacs session, then show the profile?
>> 
>> This is exactly what I meant. To run org-make-manuals from interactive
>> Emacs session.
>
> Then why would I need org-fixup.el?

I assume that you figured that org-fixup.el contains functions to build
manual.

Best,
Ihor

Re: Org mode and Emacs

[David Engster]

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We rarely have bugs related to manual builds. I recall two in many
>>> years.
>>> 
>>> Usually documentation is built automatically on ELPA and by our
>>> publishing scripts on orgmode.org.
>>
>> So basically no one builds Org, including the manual, on their own
>> system?  Even not the Org developers?
>
> Org releases are not frequent for most users and occasional build time
> should not matter too much. Nobody complained on the list.

Well, for the record, I did complain, but that was over 11 years ago. :-)

https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

I still think Org is way too slow to be a viable documentation system
for a project like Emacs.

-David

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>>> So basically no one builds Org, including the manual, on their own
>>> system?  Even not the Org developers?
>>
>> Org releases are not frequent for most users and occasional build time
>> should not matter too much. Nobody complained on the list.
>
> Well, for the record, I did complain, but that was over 11 years ago. :-)
>
> https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

Point taken.

> I still think Org is way too slow to be a viable documentation system
> for a project like Emacs.

Things have changed since 2014.
On my system, using Emacs 28 and latest Org, exporting org-manual +
org-guide takes 18.8 sec. This time includes genering the .texi files
and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

org export takes: 15.4 sec, which is 4.5 x texinfo time.
Considering that Org export is more sofisticated compared to texinfo and
that Org export is written in Elisp, I see this performance as
acceptable for a documentation system.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 15:36:16 +0800
> 
> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
> 
> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

I don't know about "acceptable", sorry.  It is definitely "endurable",
but 18.8 seconds is annoyingly long for an optimized build of Emacs to
do anything this simple.

Can you find any other single command that's part of building Emacs
which takes a comparable amount of time in an optimized build?

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>> 
>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> I don't know about "acceptable", sorry.  It is definitely "endurable",
> but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> do anything this simple.

Ok. I pushed things a bit further. The latest Org main takes ~8sec to
generate Org manual. This _includes_ makeinfo call. Now, Org export
takes the same time with texinfo - ~4sec on my system.

Further improvements are not trivial, given my familiarity with Org
babel and Org export code.

> Can you find any other single command that's part of building Emacs
> which takes a comparable amount of time in an optimized build?

Optimized Emacs build takes >30 minutes on my system. Mainly because of
native compilation. 40, 20, or 10 seconds spent generating Org
documentation are negligible for me.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: deng@randomsample.de,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 13:36:35 +0800
> 
> > I don't know about "acceptable", sorry.  It is definitely "endurable",
> > but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> > do anything this simple.
> 
> Ok. I pushed things a bit further. The latest Org main takes ~8sec to
> generate Org manual. This _includes_ makeinfo call. Now, Org export
> takes the same time with texinfo - ~4sec on my system.

Thanks, this is awesome.  Hope to see this speedup in the Emacs
repository soon.

> > Can you find any other single command that's part of building Emacs
> > which takes a comparable amount of time in an optimized build?
> 
> Optimized Emacs build takes >30 minutes on my system. Mainly because of
> native compilation.

The slowness of native-compilation is a long-standing bug report, so
I'd rather not use that as a reference for acceptable speed.  I meant
to compare with other commands run by the build.

> 40, 20, or 10 seconds spent generating Org documentation are
> negligible for me.

They aren't negligible, even in a native-compilation build, because
they happen at the end, when all the rest was already done, and no
large N in "-j N" of the Make command can help with that last long
wait.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Optimized Emacs build takes >30 minutes on my system. Mainly because of
>> native compilation.
>
> The slowness of native-compilation is a long-standing bug report, so
> I'd rather not use that as a reference for acceptable speed.  I meant
> to compare with other commands run by the build.

The other relatively slow part of the built process without native-comp
is Scraping autoloads.

Re: Org mode and Emacs

[David Engster]

> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

On my system, generating org.texi from org.org takes 21s, makeinfo
org.texi takes 2.7s. But it's nice to see that it has become much faster
compared to 2014.

> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

Why does the implementation language matter whether a documentation
system is acceptable? And while sophisticated, Org is actually still
missing features (like a proper index) to be a suitable replacement.

Org is about 1/16th of the whole Emacs documentation, so we're looking
at over 5min if everything was written in Org, give or take. People were
already up in arms when the switch from Texinfo v4 to v5 was done (which
switched to a Perl implementation). When documentation generation takes
a long time, writing it becomes more painful, as you cannot quickly
check the resulting output. And I think you underestimate how important
a quick build process is. Apart from developer annoyance, you need less
resources for CI, for instance.

-David

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>
> On my system, generating org.texi from org.org takes 21s, makeinfo
> org.texi takes 2.7s. But it's nice to see that it has become much faster
> compared to 2014.

Can you test the latest Org main?

>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> Why does the implementation language matter whether a documentation
> system is acceptable? And while sophisticated, Org is actually still
> missing features (like a proper index) to be a suitable replacement.

Well. It was a rather hand-waving argument: Elisp is not as fast of many
other programming languages. So software written in Elisp may be slow
despite the best effort. etc etc

In any case, that argument is futile now. I managed to get Org export
work as fast as texinfo.

As for proper index, we do support index for texinfo export
specifically and for publishing websites. A more general-purpose index
support is in the works. See https://github.com/tecosaur/org-glossary

> Org is about 1/16th of the whole Emacs documentation, so we're looking
> at over 5min if everything was written in Org, give or take. People were
> already up in arms when the switch from Texinfo v4 to v5 was done (which
> switched to a Perl implementation). When documentation generation takes
> a long time, writing it becomes more painful, as you cannot quickly
> check the resulting output. And I think you underestimate how important
> a quick build process is. Apart from developer annoyance, you need less
> resources for CI, for instance.

Fair point.

Best,
Ihor

Re: Org mode and Emacs

[David Engster]

> David Engster <deng@randomsample.de> writes:
>
>>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>>> org-guide takes 18.8 sec. This time includes genering the .texi files
>>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>>
>> On my system, generating org.texi from org.org takes 21s, makeinfo
>> org.texi takes 2.7s. But it's nice to see that it has become much faster
>> compared to 2014.
>
> Can you test the latest Org main?

I tested with latest Emacs master and with increased gc-cons-threshold
it takes ~5 seconds to generate org.texi. Great job!

-David

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:19:10 +0800
> 
> >> > Thanks, I hope to see this soon in the Emacs repository.
> >> 
> >> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> >> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
> >
> > This is not needed for the emacs-28 branch, so I meant master.
> 
> I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
> master only tracks our stable tagged releases. Not the working branches
> (both bugfix and main). I do not know the motivation behind it and I do
> not know if it is a good idea to change it. Do you think that Emacs
> master should instead track our Org dev branch?

I see no need for such significant changes in our merging practices.
Rather, I hoped that the speedup you achieved will be important enough
to make an exception and install just it in the Emacs master branch.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> I see no need for such significant changes in our merging practices.
> Rather, I hoped that the speedup you achieved will be important enough
> to make an exception and install just it in the Emacs master branch.

See the attached patches.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-export-get-footnote-definition-Pre-cache-referen.patch --]
[-- Type: text/x-patch, Size: 3096 bytes --]

From 3342137359f122ed7168dc75096c6a5d3839a0c2 Mon Sep 17 00:00:00 2001
Message-Id: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:05:16 +0800
Subject: [PATCH 1/8] org-export-get-footnote-definition: Pre-cache references
 in parse tree

* lisp/ox.el (org-export-get-footnote-definition): Pre-process parse
tree once to filter out all non-footnote elements.  This speeds up
subsequent footnote definition searches.
---
 lisp/ox.el | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 2a3edaa50..7f90dc36f 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3748,28 +3748,33 @@ (defun org-export-get-footnote-definition (footnote-reference info)
     (if (not label) (org-element-contents footnote-reference)
       (let ((cache (or (plist-get info :footnote-definition-cache)
 		       (let ((hash (make-hash-table :test #'equal)))
+                         ;; Cache all the footnotes in document for
+                         ;; later search.
+                         (org-element-map (plist-get info :parse-tree)
+                             '(footnote-definition footnote-reference)
+                           (lambda (f)
+		             ;; Skip any standard footnote reference
+		             ;; since those cannot contain a
+		             ;; definition.
+                             (unless (eq (org-element-property :type f) 'standard)
+                               (puthash
+                                (cons :element (org-element-property :label f))
+                                f
+                                hash)))
+                           info)
 			 (plist-put info :footnote-definition-cache hash)
 			 hash))))
 	(or
 	 (gethash label cache)
 	 (puthash label
-		  (org-element-map (plist-get info :parse-tree)
-		      '(footnote-definition footnote-reference)
-		    (lambda (f)
-		      (cond
-		       ;; Skip any footnote with a different label.
-		       ;; Also skip any standard footnote reference
-		       ;; with the same label since those cannot
-		       ;; contain a definition.
-		       ((not (equal (org-element-property :label f) label)) nil)
-		       ((eq (org-element-property :type f) 'standard) nil)
-		       ((org-element-contents f))
-		       ;; Even if the contents are empty, we can not
-		       ;; return nil since that would eventually raise
-		       ;; the error.  Instead, return the equivalent
-		       ;; empty string.
-		       (t "")))
-		    info t)
+                  (let ((hashed (gethash (cons :element label) cache)))
+                    (when hashed
+                      (or (org-element-contents hashed)
+		          ;; Even if the contents are empty, we can not
+		          ;; return nil since that would eventually raise
+		          ;; the error.  Instead, return the equivalent
+		          ;; empty string.
+                          "")))
 		  cache)
 	 (error "Definition not found for footnote %s" label))))))
 
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-export-resolve-fuzyy-link-Pre-cache-all-possible.patch --]
[-- Type: text/x-patch, Size: 2598 bytes --]

From cff9ed2f7660abaa5bd67e5de87416cb393b2cb8 Mon Sep 17 00:00:00 2001
Message-Id: <cff9ed2f7660abaa5bd67e5de87416cb393b2cb8.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:06:47 +0800
Subject: [PATCH 2/8] org-export-resolve-fuzyy-link: Pre-cache all possible
 search cells

* lisp/ox.el (org-export-resolve-fuzzy-link): Before matching LINK,
pre-process and cache all the non-nil search cells in the parse tree.
When matching, use the pre-processed info.  Fix the :test function for
the cache hash table.
---
 lisp/ox.el | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 7f90dc36f..4a9387519 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4346,17 +4346,27 @@ (defun org-export-resolve-fuzzy-link (link info &rest pseudo-types)
   (let* ((search-cells (org-export-string-to-search-cell
 			(org-element-property :path link)))
 	 (link-cache (or (plist-get info :resolve-fuzzy-link-cache)
-			 (let ((table (make-hash-table :test #'eq)))
+			 (let ((table (make-hash-table :test #'equal)))
+                           ;; Cache all the element search cells.
+                           (org-element-map (plist-get info :parse-tree)
+		               (append pseudo-types '(target) org-element-all-elements)
+	                     (lambda (datum)
+		               (dolist (cell (org-export-search-cells datum))
+		                 (if (gethash cell table)
+                                     (push datum (gethash cell table))
+                                   (puthash cell (list datum) table)))))
 			   (plist-put info :resolve-fuzzy-link-cache table)
 			   table)))
 	 (cached (gethash search-cells link-cache 'not-found)))
     (if (not (eq cached 'not-found)) cached
       (let ((matches
-	     (org-element-map (plist-get info :parse-tree)
-		 (append pseudo-types '(target) org-element-all-elements)
-	       (lambda (datum)
-		 (and (org-export-match-search-cell-p datum search-cells)
-		      datum)))))
+             (let (result)
+               (dolist (search-cell search-cells)
+                 (setq result
+                       (nconc
+                        result
+	                (gethash search-cell link-cache))))
+               (delq nil result))))
 	(unless matches
 	  (signal 'org-link-broken (list (org-element-property :path link))))
 	(puthash
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-org-export-resolve-id-link-Pre-cache-all-the-ids-in-.patch --]
[-- Type: text/x-patch, Size: 2832 bytes --]

From eca7a1d4dea1e8980aee6a02967d4c596114f759 Mon Sep 17 00:00:00 2001
Message-Id: <eca7a1d4dea1e8980aee6a02967d4c596114f759.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:32:35 +0800
Subject: [PATCH 3/8] org-export-resolve-id-link: Pre-cache all the ids in the
 parse tree

* lisp/ox.el (org-export-resolve-id-link): Pre-cache all the ids in
the parse tree for faster lookup.
---
 lisp/ox.el | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 4a9387519..b431d7119 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4393,15 +4393,27 @@ (defun org-export-resolve-id-link (link info)
 \"custom-id\".  Throw an error if no match is found."
   (let ((id (org-element-property :path link)))
     ;; First check if id is within the current parse tree.
-    (or (org-element-map (plist-get info :parse-tree) 'headline
-	  (lambda (headline)
-	    (when (or (equal (org-element-property :ID headline) id)
-		      (equal (org-element-property :CUSTOM_ID headline) id))
-	      headline))
-	  info 'first-match)
-	;; Otherwise, look for external files.
-	(cdr (assoc id (plist-get info :id-alist)))
-	(signal 'org-link-broken (list id)))))
+    (or (let ((local-ids (or (plist-get info :id-local-cache)
+                             (let ((table (make-hash-table :test #'equal)))
+                               (org-element-map
+                                   (plist-get info :parse-tree)
+                                   'headline
+                                 (lambda (headline)
+                                   (let ((id (org-element-property :ID headline))
+                                         (custom-id (org-element-property :CUSTOM_ID headline)))
+                                     (when id
+                                       (unless (gethash id table)
+                                         (puthash id headline table)))
+                                     (when custom-id
+                                       (unless (gethash custom-id table)
+                                         (puthash custom-id headline table)))))
+                                 info)
+                               (plist-put info :id-local-cache table)
+                               table))))
+          (gethash id local-ids))
+        ;; Otherwise, look for external files.
+        (cdr (assoc id (plist-get info :id-alist)))
+        (signal 'org-link-broken (list id)))))
 
 (defun org-export-resolve-radio-link (link info)
   "Return radio-target object referenced as LINK destination.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-org-export-as-Do-not-update-buffer-settings-when-not.patch --]
[-- Type: text/x-patch, Size: 3174 bytes --]

From fb8b921066ca84aaa7034c7f13e2cc62da40d7ad Mon Sep 17 00:00:00 2001
Message-Id: <fb8b921066ca84aaa7034c7f13e2cc62da40d7ad.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:03:18 +0800
Subject: [PATCH 4/8] org-export-as: Do not update buffer settings when not
 modified

* lisp/ox.el (org-export-as): Use `buffer-chars-modified-tick' and
avoid extra invocations of `org-set-regexps-and-options' and
`org-update-radio-target-regexp' when the buffer is not changed.
Also, disable folding checks.  Folding is irrelevant inside export
buffer.
---
 lisp/ox.el | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index b431d7119..a4512270c 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2956,11 +2956,12 @@ (defun org-export-as
 		    (mapcar (lambda (o) (and (eq (nth 4 o) 'parse) (nth 1 o)))
 			    (append (org-export-get-all-options backend)
 				    org-export-options-alist))))
-	     tree)
+	     tree modified-tick)
 	;; Update communication channel and get parse tree.  Buffer
 	;; isn't parsed directly.  Instead, all buffer modifications
 	;; and consequent parsing are undertaken in a temporary copy.
 	(org-export-with-buffer-copy
+         (font-lock-mode -1)
 	 ;; Run first hook with current back-end's name as argument.
 	 (run-hook-with-args 'org-export-before-processing-hook
 			     (org-export-backend-name backend))
@@ -2972,6 +2973,7 @@ (defun org-export-as
 	 ;; potentially invasive changes.
 	 (org-set-regexps-and-options)
 	 (org-update-radio-target-regexp)
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;;  Possibly execute Babel code.  Re-run a macro expansion
 	 ;;  specifically for {{{results}}} since inline source blocks
 	 ;;  may have generated some more.  Refresh buffer properties
@@ -2979,8 +2981,10 @@ (defun org-export-as
 	 (when org-export-use-babel
 	   (org-babel-exp-process-buffer)
 	   (org-macro-replace-all '(("results" . "$1")) parsed-keywords)
-	   (org-set-regexps-and-options)
-	   (org-update-radio-target-regexp))
+           (unless (eq modified-tick (buffer-chars-modified-tick))
+	     (org-set-regexps-and-options)
+	     (org-update-radio-target-regexp))
+           (setq modified-tick (buffer-chars-modified-tick)))
 	 ;; Run last hook with current back-end's name as argument.
 	 ;; Update buffer properties and radio targets one last time
 	 ;; before parsing.
@@ -2988,8 +2992,10 @@ (defun org-export-as
 	 (save-excursion
 	   (run-hook-with-args 'org-export-before-parsing-hook
 			       (org-export-backend-name backend)))
-	 (org-set-regexps-and-options)
-	 (org-update-radio-target-regexp)
+         (unless (eq modified-tick (buffer-chars-modified-tick))
+	   (org-set-regexps-and-options)
+	   (org-update-radio-target-regexp))
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;; Update communication channel with environment.
 	 (setq info
 	       (org-combine-plists
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-doc-Makefile-Disable-GC-during-export.patch --]
[-- Type: text/x-patch, Size: 1396 bytes --]

From 7f3ee582c1105c66a66fb61786c4e086e30460b4 Mon Sep 17 00:00:00 2001
Message-Id: <7f3ee582c1105c66a66fb61786c4e086e30460b4.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 07:29:06 +0800
Subject: [PATCH 5/8] doc/Makefile: Disable GC during export

* doc/Makefile (org.texi):
(orgguide.texi): Set `gc-cons-threshold` to `most-positive=fixnum' and
thus disable garbage collection while exporting manuals.  This reduces
the manual generation time.
---
 doc/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/Makefile b/doc/Makefile
index cb6d72bdc..5911bd08a 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -31,12 +31,14 @@ org.texi:	org-manual.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-manual)'
 
 orgguide.texi:	org-guide.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-guide)'
 
 org-version.inc:	org.texi
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-org-export-data-Concatenate-strings-in-temporary-buf.patch --]
[-- Type: text/x-patch, Size: 3492 bytes --]

From 4a466e28999bbaa5169b6fad414cf08c6d23cd7b Mon Sep 17 00:00:00 2001
Message-Id: <4a466e28999bbaa5169b6fad414cf08c6d23cd7b.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:01:53 +0800
Subject: [PATCH 6/8] org-export-data: Concatenate strings in temporary buffer
 for performance

* lisp/ox.el (org-export-data): Use temporary buffer to collect export
data instead of `mapconcat'.  Using buffer puts less load on garbage
collector.
---
 lisp/ox.el | 50 ++++++++++++++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 22 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index a4512270c..ae7e41e57 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1923,28 +1923,34 @@ (defun org-export-data (data info)
 			      (and (not greaterp)
 				   (memq type org-element-recursive-objects)))
 			     (contents
-			      (mapconcat
-			       (lambda (element) (org-export-data element info))
-			       (org-element-contents
-				(if (or greaterp objectp) data
-				  ;; Elements directly containing
-				  ;; objects must have their indentation
-				  ;; normalized first.
-				  (org-element-normalize-contents
-				   data
-				   ;; When normalizing first paragraph
-				   ;; of an item or
-				   ;; a footnote-definition, ignore
-				   ;; first line's indentation.
-				   (and
-				    (eq type 'paragraph)
-				    (memq (org-element-type parent)
-					  '(footnote-definition item))
-				    (eq (car (org-element-contents parent))
-					data)
-				    (eq (org-element-property :pre-blank parent)
-					0)))))
-			       "")))
+                              (let ((export-buffer (current-buffer)))
+                                (with-temp-buffer
+                                  (dolist (element (org-element-contents
+				                    (if (or greaterp objectp) data
+				                      ;; Elements directly containing
+				                      ;; objects must have their indentation
+				                      ;; normalized first.
+				                      (org-element-normalize-contents
+				                       data
+				                       ;; When normalizing first paragraph
+				                       ;; of an item or
+				                       ;; a footnote-definition, ignore
+				                       ;; first line's indentation.
+				                       (and
+				                        (eq type 'paragraph)
+				                        (memq (org-element-type parent)
+					                      '(footnote-definition item))
+				                        (eq (car (org-element-contents parent))
+					                    data)
+				                        (eq (org-element-property :pre-blank parent)
+					                    0))))))
+                                    (insert
+                                     ;; Use right local variable
+                                     ;; environment if there are, for
+                                     ;; example, #+BIND variables.
+                                     (with-current-buffer export-buffer
+                                       (org-export-data element info))))
+                                  (buffer-string)))))
 			(broken-link-handler
 			 (funcall transcoder data
 				  (if (not greaterp) contents
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-org-element-map-Avoid-repetitive-plist-get-call.patch --]
[-- Type: text/x-patch, Size: 1486 bytes --]

From efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc Mon Sep 17 00:00:00 2001
Message-Id: <efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 09:28:27 +0800
Subject: [PATCH 7/8] org-element-map: Avoid repetitive `plist-get' call

* lisp/org-element.el (org-element-map): Do not call `(plist-get info
:ignore-list)' on every iteration.
---
 lisp/org-element.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index 9db1406b3..20b5b0303 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4391,6 +4391,7 @@ (defun org-element-map
 		       ;; every element it encounters.
 		       (and (not (eq category 'elements))
 			    (setq category 'elements))))))))
+         (--ignore-list (plist-get info :ignore-list))
 	 --acc)
     (letrec ((--walk-tree
 	      (lambda (--data)
@@ -4400,7 +4401,7 @@ (defun org-element-map
 		  (cond
 		   ((not --data))
 		   ;; Ignored element in an export context.
-		   ((and info (memq --data (plist-get info :ignore-list))))
+		   ((and info (memq --data --ignore-list)))
 		   ;; List of elements or objects.
 		   ((not --type) (mapc --walk-tree --data))
 		   ;; Unconditionally enter parse trees.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-org-cite-list-citations-Cache-footnote-definition-se.patch --]
[-- Type: text/x-patch, Size: 3309 bytes --]

From 56830c1792b548e9157cdd95f4bb7c093e390a05 Mon Sep 17 00:00:00 2001
Message-Id: <56830c1792b548e9157cdd95f4bb7c093e390a05.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 10:43:29 +0800
Subject: [PATCH 8/8] org-cite-list-citations: Cache footnote-definition
 searches

* lisp/oc.el (org-cite-list-citations): Avoid quadratic complexity.
Pre-calculate list of all footnote definitions and cache the footnote
label search hits.  Do not make `org-element-map' accumulate unused
result.
---
 lisp/oc.el | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/lisp/oc.el b/lisp/oc.el
index eb5f519cb..c4cd0268c 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -808,6 +808,8 @@ (defun org-cite-list-citations (info)
   (or (plist-get info :citations)
       (letrec ((cites nil)
                (tree (plist-get info :parse-tree))
+               (definition-cache (make-hash-table :test #'equal))
+               (definition-list nil)
                (find-definition
                 ;; Find definition for standard reference LABEL.  At
                 ;; this point, it is impossible to rely on
@@ -816,11 +818,21 @@ (defun org-cite-list-citations (info)
                 ;; un-processed citation objects.  So we use
                 ;; a simplified version of the function above.
                 (lambda (label)
-                  (org-element-map tree 'footnote-definition
-                    (lambda (d)
-                      (and (equal label (org-element-property :label d))
-                           (or (org-element-contents d) "")))
-                    info t)))
+                  (or (gethash label definition-cache)
+                      (org-element-map
+                          (or definition-list
+                              (setq definition-list
+                                    (org-element-map
+                                        tree
+                                        'footnote-definition
+                                      #'identity info)))
+                          'footnote-definition
+                        (lambda (d)
+                          (and (equal label (org-element-property :label d))
+                               (puthash label
+                                        (or (org-element-contents d) "")
+                                        definition-cache)))
+                        info t))))
                (search-cites
                 (lambda (data)
                   (org-element-map data '(citation footnote-reference)
@@ -834,7 +846,8 @@ (defun org-cite-list-citations (info)
                         (_
                          (let ((label (org-element-property :label datum)))
                            (funcall search-cites
-                                    (funcall find-definition label))))))
+                                    (funcall find-definition label)))))
+                      nil)
                     info nil 'footnote-definition t))))
         (funcall search-cites tree)
         (let ((result (nreverse cites)))
-- 
2.35.1

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 15:03:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see no need for such significant changes in our merging practices.
> > Rather, I hoped that the speedup you achieved will be important enough
> > to make an exception and install just it in the Emacs master branch.
> 
> See the attached patches.

Thanks, I installed these, with one exception: I limited
gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
most-positive-fixnum.  The slowdown in producing the Org manual as
result of that is negligible, and I didn't feel comfortable with
disabling GC altogether, as it caused the Emacs memory footprint go up
to 0.5GB.

These changes slash the conversion from 2.5 min to just 35 sec on my
system, with an unoptimized build of Emacs, which is a terrific
speedup.  Thanks!

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Mattias Engdegård]

16 juni 2022 kl. 10.13 skrev Eli Zaretskii <eliz@gnu.org>:

> I limited
> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
> most-positive-fixnum.

Some timings for the export to .texi (old machine, optimised build, bytecode only):

|    gc-cons-threshold | time |
|----------------------+------|
|               800000 | 14.1 | (default value)
|             25000000 |  6.2 |
|             50000000 |  5.7 |
|            100000000 |  5.7 |
| most-positive-fixnum |  5.1 |

thus Eli's choice is very good, and we really should do something about that GC of ours.

> a terrific
> speedup.  Thanks!

Echoed!

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Mattias Engdegård <mattiase@acm.org> writes:

>> I limited
>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>> most-positive-fixnum.
>
> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>
> |    gc-cons-threshold | time |
> |----------------------+------|
> |               800000 | 14.1 | (default value)
> |             25000000 |  6.2 |
> |             50000000 |  5.7 |
> |            100000000 |  5.7 |
> | most-positive-fixnum |  5.1 |
>
> thus Eli's choice is very good, and we really should do something about that GC of ours.

I am wondering if there is a "universal" value suitable for one-off Emacs
batch invocations.

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-16 20:58:07] wrote:
> Mattias Engdegård <mattiase@acm.org> writes:
>>> I limited
>>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>>> most-positive-fixnum.
>>
>> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>>
>> |    gc-cons-threshold | time |
>> |----------------------+------|
>> |               800000 | 14.1 | (default value)
>> |             25000000 |  6.2 |
>> |             50000000 |  5.7 |
>> |            100000000 |  5.7 |
>> | most-positive-fixnum |  5.1 |
>>
>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>
> I am wondering if there is a "universal" value suitable for one-off Emacs
> batch invocations.

I doubt that's the case, but of course we should try and use values that
work "overall best" on average.  Maybe we should revisit the current
knobs and their default values.

Currently the two important knobs we have are:

    gc-cons-threshold
    gc-cons-percentage

Maybe it's time for some benchmarks with various values for these knobs
to see if the values we're using now are "good enough" or might need to
be changed.

We could also try and be bit more discerning.  E.g. currently when the
program is in a phase where it builds a lot of new data-structures, we
run the GC everytime some amount of new memory has been allocated, but
presumably almost none of those objects have died yet (we're still
building the overall datastructure) so the GC will be a pure waste of
time.  OTOH in other phases the code allocates objects which are used
only temporarily, in which case it's beneficial to run the GC somewhat
frequently to avoid growing our heap unnecessarily (and suffering
fragmentation as a result).

It's hard to know beforehand whether a GC will be useful or not, tho.
But maybe we can find good heuristics.  E.g. have something like
a `gc-cons-percentage` which depends on how much garbage we collected in
the last GC: if a GC doesn't collect any garbage (or very little of it),
it's a sign that we're in a phase where running the GC is not very
useful so we should wait a bit more until the next GC, whereas if the GC
collected a fair bit of garbage, it's a sign that we're in a phase where
running the GC is beneficial and we can run it a bit more often.


        Stefan

Larger GC thresholds for non-interactive Emacs (was: Org mode and Emacs)

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>>
>> I am wondering if there is a "universal" value suitable for one-off Emacs
>> batch invocations.
>
> I doubt that's the case, but of course we should try and use values that
> work "overall best" on average.  Maybe we should revisit the current
> knobs and their default values.
>
> Currently the two important knobs we have are:
>
>     gc-cons-threshold
>     gc-cons-percentage
>
> Maybe it's time for some benchmarks with various values for these knobs
> to see if the values we're using now are "good enough" or might need to
> be changed.

Note that I did not initially ask to change the GC defaults for normal
Emacs session.

What I am trying to suggest is:
1. Have some conservative GC settings in interactive Emacs session (as
   we do now; I do _not_ suggest changing it)
2. Increase GC thresholds in non-interactive --batch sessions, which
   tend to not run for a long time.

I am a bit afraid to touch the topic of changing the interactive GC
defaults. People tend to have contradicting opinions on this.

I'd like to ask one thing from the very beginning here: Please spin a
separate thread if you have objections on actual GC defaults applicable
to normal Emacs sessions.

> We could also try and be bit more discerning.  E.g. currently when the
> program is in a phase where it builds a lot of new data-structures, we
> run the GC everytime some amount of new memory has been allocated, but
> presumably almost none of those objects have died yet (we're still
> building the overall datastructure) so the GC will be a pure waste of
> time.  OTOH in other phases the code allocates objects which are used
> only temporarily, in which case it's beneficial to run the GC somewhat
> frequently to avoid growing our heap unnecessarily (and suffering
> fragmentation as a result).

AFAIK, there is no way to determine such stage automatically. It's
usually up to the program. For example, org-element-parse-buffer
(function that creates buffer AST in Org) does the following:

(let (... (gc-cons-threshold #x40000000))
      ...
      (org-element--parse-elements ...))

A possible useful thing Emacs could help with would be a macro dedicated
to let-binds like the above. Something like:

(with-reduced-gc
 ;; Do staff)

 with-reduced-gc could take care about determining the inner specifics
 on what alternative gc-cons-threshold value should be used (possibly
 depending on the system memory information).

> It's hard to know beforehand whether a GC will be useful or not, tho.
> But maybe we can find good heuristics.  E.g. have something like
> a `gc-cons-percentage` which depends on how much garbage we collected in
> the last GC: if a GC doesn't collect any garbage (or very little of it),
> it's a sign that we're in a phase where running the GC is not very
> useful so we should wait a bit more until the next GC, whereas if the GC
> collected a fair bit of garbage, it's a sign that we're in a phase where
> running the GC is beneficial and we can run it a bit more often.

FYI, I have played with this approach making use of
https://gitlab.com/koral/gcmh

In practice, relying on recent GC history is often misleading.

GC is normally not very frequent - tens of seconds to minutes. And the
practical Emacs memory usage patterns are often irregular: long periods
of "idleness"/low intensity usage vs. short bursts of high memory
allocations when some "heavy" commands are called.

The adjustment you propose will suffer from the following:
1. There is long "idle" period and GC gets less frequent
2. User runs "heavy" command, GC frequency not yet adjusted
3. Command allocates a lots of memory - GC is finally triggered and has
   to go through a lot of garbage; It often happens _after_ the command
   finishes.
4. GC re-adjusts the frequency, but Emacs is again "idle" with no
   "heavy" commands being used.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> Maybe it's time for some benchmarks with various values for these knobs
>> to see if the values we're using now are "good enough" or might need to
>> be changed.
>
> Note that I did not initially ask to change the GC defaults for normal
> Emacs session.

You're absolutely right, but we've had various reports of late that
indicate that we're probably spending too much time in the GC.
So I think we're due for some re-evaluation.

> What I am trying to suggest is:
> 1. Have some conservative GC settings in interactive Emacs session (as
>    we do now; I do _not_ suggest changing it)
> 2. Increase GC thresholds in non-interactive --batch sessions, which
>    tend to not run for a long time.

Agreed.  Then again, we occasionally also use long-running commands
interactively so it might be worthwhile to tune the settings for
interactive use as well.

> I am a bit afraid to touch the topic of changing the interactive GC
> defaults.  People tend to have contradicting opinions on this.

:-)

The issue is that it's easy to see "oh god, we're spending so much time
in the GC, let's disable it" but much harder to see the "long term"
impact of increasing the thresholds (which tend to increase the
fragmentation and the total heap size both of which tend to slow down
execution but the second additionally makes every GC cycle take longer,
again increasing the pressure from naive onlookers to disable the GC).

> I'd like to ask one thing from the very beginning here: Please spin a
> separate thread if you have objections on actual GC defaults applicable
> to normal Emacs sessions.

No, I think focusing on batch use is a good idea.  Also because it makes
it much easier to actually measure the impact.

> A possible useful thing Emacs could help with would be a macro dedicated
> to let-binds like the above. Something like:
>
> (with-reduced-gc
>  ;; Do staff)

Sounds about right, tho I think the name of the macro should be related
to what the code does rather than to what the author thinks the GC should
do about it.  Something like `this-code-does-not-generate-garbage`.

>  with-reduced-gc could take care about determining the inner specifics
>  on what alternative gc-cons-threshold value should be used (possibly
>  depending on the system memory information).

Sounds good.

>> It's hard to know beforehand whether a GC will be useful or not, tho.
>> But maybe we can find good heuristics.  E.g. have something like
>> a `gc-cons-percentage` which depends on how much garbage we collected in
>> the last GC: if a GC doesn't collect any garbage (or very little of it),
>> it's a sign that we're in a phase where running the GC is not very
>> useful so we should wait a bit more until the next GC, whereas if the GC
>> collected a fair bit of garbage, it's a sign that we're in a phase where
>> running the GC is beneficial and we can run it a bit more often.
>
> FYI, I have played with this approach making use of
> https://gitlab.com/koral/gcmh

But this one focuses on interactive use.  The kind of heuristic I'm
proposing above would only make sense within a single (long-running)
command, or in batch mode, I think.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

I tried this very advanced patch:

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index d28ec0be16..1bd0793097 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -5366,6 +5366,7 @@ batch-byte-compile
   ;; behavior.
   (setq attempt-stack-overflow-recovery nil
         attempt-orderly-shutdown-on-fatal-signal nil)
+  (setq gc-cons-threshold (* 100 gc-cons-threshold))
   (let ((error nil))
     (while command-line-args-left
       (if (file-directory-p (expand-file-name (car command-line-args-left)))

A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀

Most of this time is spent doing other things than building lisp files,
though, so the impact on byte compilation is huge.

rm lisp/*/*.elc
time make -j32

goes from 29s to 14s.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀

I wonder whether we should just have a command line option for this --
i.e., --no-garbage-collection to be used in combination with --batch for
jobs that aren't long-running (i.e., will exit before growing huge).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀
> I wonder whether we should just have a command line option for this --
> i.e., --no-garbage-collection to be used in combination with --batch for
> jobs that aren't long-running (i.e., will exit before growing huge).

"no-garbage-collection" would be either a bug (we'd crash or be killed[*]
on long-enough-running jobs) or a lie (we'd still perform GC sometimes).

Also I hope we can first try to simply provide better defaults before
requiring manual intervention from the user.  I haven't seen any
evidence yet that the current defaults are good for batch jobs.

BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
does not adapt to the heap size, so maybe we'd be better off
playing with `gc-cons-percentage`.


        Stefan


[*] Like the Org manual's build on elpa.gnu.org yesterday because its
gc-threshold was set to most-positive-fixnum and `elpa.gnu.org` only has
1GB of RAM.

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> "no-garbage-collection" would be either a bug (we'd crash or be killed[*]
> on long-enough-running jobs) or a lie (we'd still perform GC sometimes).

It's not a bug to do what the user says.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
> does not adapt to the heap size, so maybe we'd be better off
> playing with `gc-cons-percentage`.

I am not sure if `gc-cons-percentage' is reliable.
When we have a small heap (at the beginning of the batch command run),
the GC will be triggered frequently. Later, after initial memory
allocations for data structures is complete, the same
`gc-cons-percentage' will make GC fire less frequently and will have
nothing to do with actually allocated/deallocated memory objects in the
subsequent parts of the batch process.

Consider the following example:

(defun init () (allocate-huge-hash-map-object))
(defun run () (dolist (...) (let (this and that) (query the huge hash-map))))
(init)
(run)

The (init) part will trigger frequent GCs yielding no of very small
amount of garbage being collected.

The (run) part may use more temporary allocations. Yet, GC will be
triggered much less frequently simply because the atomic (let ...)
allocations are small compared to the initially allocated hash-map
object.

I believe that gc-cons-threshold will still be useful as _an upper
bound_ to trigger GC. gc-cons-percentage is probably more efficient when
it triggers below gc-cons-threshold.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-18 13:35:04] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
>> does not adapt to the heap size, so maybe we'd be better off
>> playing with `gc-cons-percentage`.
> I am not sure if `gc-cons-percentage' is reliable.

From where I stand it's the only one that's reliable.

It basically controls the percentage of *time* we spend in the GC: if we
assume that the time to perform a GC is proportional to the heap size,
and that the allocation rate (in MB/s) is more or less constant, then
`gc-cons-percentage' directly controls the percentage of time spent in
the GC.

Of course, those two assumptions don't hold exactly, but they're fairly
good approximations of reality.

Tweaking `gc-cons-threshold` on the other hand is very coarse and
brittle: if the value is significantly smaller than the heap, then we'll
spend too much time in the GC and if it's significantly larger then we
waste a lot of memory on garbage.  There simply can't be any "one good
value" for `gc-cons-threshold`.

> When we have a small heap (at the beginning of the batch command run),
> the GC will be triggered frequently.

Yes, but the GC cycles will finish quickly.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> rm lisp/*/*.elc
> time make -j32

IIRC this test can give surprising results because it's strongly
affected by the time when the byte-compiler files get compiled which is
somewhat arbitrary.

> goes from 29s to 14s.

Wow!

Here are the results of a similar test where I tried to avoid the above
problem by also removing src/boostrap-emacs (and I also rm'd
lisp/loaddefs.el for good measure tho it was probably not a great idea),
and I used

    BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":

make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total
make -j24 FOO=0.1    1338.36s user 60.60s system 1252% cpu 1:51.65 total
make -j24 FOO=0.1    1339.53s user 60.08s system 1252% cpu 1:51.76 total
make -j24 FOO=0.2    1266.54s user 59.75s system 1264% cpu 1:44.92 total
make -j24 FOO=0.2    1269.17s user 60.18s system 1256% cpu 1:45.83 total
make -j24 FOO=0.2    1266.10s user 60.42s system 1258% cpu 1:45.43 total
make -j24 FOO=0.3    1200.06s user 60.24s system 1251% cpu 1:40.73 total
make -j24 FOO=0.3    1198.80s user 60.06s system 1254% cpu 1:40.39 total
make -j24 FOO=0.3    1202.49s user 59.60s system 1252% cpu 1:40.80 total
make -j24 FOO=0.4    1134.26s user 59.75s system 1233% cpu 1:36.81 total
make -j24 FOO=0.4    1133.11s user 60.07s system 1226% cpu 1:37.25 total
make -j24 FOO=0.4    1132.18s user 59.94s system 1226% cpu 1:37.18 total
make -j24 FOO=0.5    1069.59s user 59.70s system 1211% cpu 1:33.22 total
make -j24 FOO=0.5    1070.84s user 60.06s system 1211% cpu 1:33.34 total
make -j24 FOO=0.5    1070.93s user 59.86s system 1210% cpu 1:33.43 total
make -j24 FOO=1.0     845.92s user 61.24s system 1126% cpu 1:20.55 total
make -j24 FOO=1.0     847.92s user 61.82s system 1130% cpu 1:20.46 total
make -j24 FOO=1.0     846.59s user 61.46s system 1133% cpu 1:20.10 total
make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total
make -j24 FOO=2.0     678.04s user 69.19s system 1042% cpu 1:11.64 total
make -j24 FOO=2.0     677.66s user 68.24s system 1041% cpu 1:11.59 total
make -j24 FOO=3.0     677.50s user 69.38s system 1047% cpu 1:11.28 total
make -j24 FOO=3.0     677.42s user 68.43s system 1047% cpu 1:11.19 total
make -j24 FOO=3.0     677.98s user 69.10s system 1044% cpu 1:11.50 total
make -j24 FOO=4.0     677.43s user 68.58s system 1051% cpu 1:10.96 total
make -j24 FOO=4.0     677.09s user 68.49s system 1056% cpu 1:10.57 total
make -j24 FOO=4.0     677.60s user 69.23s system 1049% cpu 1:11.17 total
make -j24 FOO=5.0     677.22s user 68.90s system 1043% cpu 1:11.52 total
make -j24 FOO=5.0     677.47s user 68.91s system 1044% cpu 1:11.49 total
make -j24 FOO=5.0     677.13s user 67.98s system 1049% cpu 1:10.98 total
make -j24 FOO=7.0     678.21s user 68.57s system 1044% cpu 1:11.52 total
make -j24 FOO=7.0     677.48s user 68.50s system 1044% cpu 1:11.42 total
make -j24 FOO=7.0     677.95s user 68.25s system 1050% cpu 1:11.00 total
make -j24 FOO=10.0    677.30s user 69.52s system 1045% cpu 1:11.45 total
make -j24 FOO=10.0    678.18s user 68.55s system 1042% cpu 1:11.62 total
make -j24 FOO=10.0    678.97s user 68.45s system 1043% cpu 1:11.64 total
make -j24 FOO=20.0    677.99s user 68.42s system 1041% cpu 1:11.67 total
make -j24 FOO=20.0    677.69s user 69.31s system 1046% cpu 1:11.36 total
make -j24 FOO=20.0    677.89s user 68.70s system 1045% cpu 1:11.39 total

Clearly for this test, the default (0.1) results in too low a threshold
for best performance and thus too much time spent in the GC.  We reach
a plateau around percentage=2.0.

This suggests that for batch jobs maybe we should bump up
`gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Here are the results of a similar test where I tried to avoid the above
> problem by also removing src/boostrap-emacs (and I also rm'd
> lisp/loaddefs.el for good measure tho it was probably not a great idea),
> and I used
>
>     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":
>
> make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total

[...]

> make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total

Nice!

> This suggests that for batch jobs maybe we should bump up
> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

Yup.  But do you mean in general?  I.e., -batch would set that variable
to 2.0?  Would there be any likely major repercussions -- i.e., jobs
that used to run fine would run out of memory?

As a quick test, I added a (setq gc-cons-percentage 2.0) to startup.el
here, and a

git clean -dfx
time make -j32

went from 1m29s to 1m4s.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Alan Mackenzie]

Hello, Lars.

On Fri, Jun 17, 2022 at 20:20:48 +0200, Lars Ingebrigtsen wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Here are the results of a similar test where I tried to avoid the above
> > problem by also removing src/boostrap-emacs (and I also rm'd
> > lisp/loaddefs.el for good measure tho it was probably not a great idea),
> > and I used

> >     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":

> > make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total

> [...]

> > make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total

> Nice!

> > This suggests that for batch jobs maybe we should bump up
> > `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?  Would there be any likely major repercussions -- i.e., jobs
> that used to run fine would run out of memory?

> As a quick test, I added a (setq gc-cons-percentage 2.0) to startup.el
> here, and a

> git clean -dfx
> time make -j32

> went from 1m29s to 1m4s.

Just as another independent test (as if any were needed), I put

    BYTE_COMPILE_EXTRA_FLAGS = --eval '(setq gc-cons-threshold 50000000)'

into lisp/Makefile.in, and my time for $ make -j17 bootstrap dropped
from 5m52s to 3m35s, a drop of 39%.  Watching the .elc's being built was
noticeably faster, even to the naked eye.  This was a build with native
compilation enabled.

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> Just as another independent test (as if any were needed), I put

Thanks anyway, but indeed, it would be good to see tests that do other
things than compilation.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Thanks anyway, but indeed, it would be good to see tests that do other
> things than compilation.

Perhaps

rm lisp/leim/ja-dic/ja-dic.el*; time make

It goes from around 9 seconds to around 6 seconds (with a
cons-percentage of 4.0).  "make -j32 check" goes from 18s to 15s (with a
value of 2.0).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> This suggests that for batch jobs maybe we should bump up
>> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.
> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?

Yup.

> Would there be any likely major repercussions

That's the question we should investigate as well.

> -- i.e., jobs that used to run fine would run out of memory?

I added the patch below to try and see a bit more what's going on:

    diff --git a/src/alloc.c b/src/alloc.c
    index 55e18ecd77e..d954e928eed 100644
    --- a/src/alloc.c
    +++ b/src/alloc.c
    @@ -6245,6 +6245,11 @@ garbage_collect (void)
       consing_until_gc = gc_threshold
         = consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0);

    +  fprintf (stderr, "GC-%d p=%.1f total=%.1fM thresold=%.1fM\n",
    +          getpid (), XFLOAT_DATA (Vgc_cons_percentage),
    +          total_bytes_of_live_objects () / 1048576.0,
    +          consing_until_gc / 1048576.0);
    +
       /* Unblock *after* re-setting `consing_until_gc` in case `unblock_input`
          signals an error (see bug#43389).  */
       unblock_input ();

and then I ran

    rm **/*.elc src/bootstrap-emacs lisp/loaddefs.el
    make -j24 BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage 2.0)'

The first thing I see is that with the number of GCs where p=0.1
outweighs those with p=2.0 by a factor 10, so clearly the number of GCs
is significantly reduced.  I also looked for the number of GCs performed
in a given process:

    % grep 'p=2.0' build-log | sed 's/ .*//' | sort | uniq -c
      1 GC-15817
      1 GC-15837
      1 GC-15841
      1 GC-15842
      1 GC-15985
      1 GC-15990
      1 GC-15991
      1 GC-15993
      1 GC-15995
      1 GC-15998
     18 GC-16009
     18 GC-16021
     18 GC-16023
     18 GC-16026
     18 GC-16028
     18 GC-16042
     18 GC-16045
     18 GC-16048
     18 GC-16056
     18 GC-16058
      1 GC-16062
      1 GC-16064
     18 GC-16067
     17 GC-16079

This says that only 24 processes performed a GC with p=2.0, so the
vast majority of the byte-compilation processes finished without
performing any GC at all (well, maybe not quite: see below).

It's also odd how most performed 0 GC, then 12 performed 1 GC, then
1 performed 17 GCs and 11 performed exactly 18 GCs.  What's so magical
about 18?

Then I looked at those that performed 18 GCs and they all look quite similar:

    % grep GC-16026 ./+make-2.0.log
    GC-16026 p=0.1 total=0.5M thresold=0.8M
    GC-16026 p=2.0 total=3.9M thresold=7.8M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M

The first GC seems to happen before we set percentage to 2.0 (so
apparently all compilation processes performed at least one GC before we
set percentage to 2.0 and then the majority of them performed no further
GC before exiting).  [ So if we set percentage to 2.0 a bit earlier than
what happens with `--eval` we may gain yet a bit more time.  ]

9.5M is actually exactly 10000000 bytes so I suspect these come from
either `cedet/semantic.el` or `international/mule-cmds.el` which both
set gc-cons-threshold to (max gc-cons-threshold 10000000).


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 17 Jun 2022 18:53:17 -0400
> 
> It's also odd how most performed 0 GC, then 12 performed 1 GC, then
> 1 performed 17 GCs and 11 performed exactly 18 GCs.  What's so magical
> about 18?

Divine intervention?  18 means "alive" (חי) in gematria.

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The first GC seems to happen before we set percentage to 2.0 (so
> apparently all compilation processes performed at least one GC before we
> set percentage to 2.0 and then the majority of them performed no further
> GC before exiting).

Hm, interesting...

> [ So if we set percentage to 2.0 a bit earlier than what happens with
> `--eval` we may gain yet a bit more time.  ]

This should probably be early enough, I guess?

diff --git a/src/emacs.c b/src/emacs.c
index 43b9901e08..33ad9c6b8d 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1620,6 +1620,7 @@ main (int argc, char **argv)
   if (argmatch (argv, argc, "-batch", "--batch", 5, NULL, &skip_args)
       || only_version)
     {
+      Vgc_cons_percentage = make_float (2.0);
       noninteractive = 1;
       Vundo_outer_limit = Qnil;
     }


> 9.5M is actually exactly 10000000 bytes so I suspect these come from
> either `cedet/semantic.el` or `international/mule-cmds.el` which both
> set gc-cons-threshold to (max gc-cons-threshold 10000000).

Ah, I see.  Perhaps those should increase that even more if it still
leads to 18 GCs?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> 9.5M is actually exactly 10000000 bytes so I suspect these come from
>> either `cedet/semantic.el` or `international/mule-cmds.el` which both
>> set gc-cons-threshold to (max gc-cons-threshold 10000000).
>
> Ah, I see.  Perhaps those should increase that even more if it still
> leads to 18 GCs?

As I just explained in another message, I think messing with
`gc-cons-threshold` is a bad idea because it depends on the heap size.
We see this here where we end up thinking that maybe we should update
this magical number `10000000` with some other magical number.

The values of `gc-cons-percentage` aren't nearly as magical and they
automatically adjust to changes in heap sizes.  We can see that the
proposed `gc-cons-percentage` of 2.0 would result in a heap size almost
equal to that magical `10000000`, so rather than increase this number we
should likely just remove those hacks.

As for reducing the number of those 18 GCs, the data we have doesn't
tell us that it would bring any benefit.  Instead the data says we already
use up more than 3 times as much memory as what's truly needed (13.6MB
vs 4.1MB), which suggests to me that `10000000` is already rather high.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?  Would there be any likely major repercussions -- i.e., jobs
> that used to run fine would run out of memory?

I've looked a bit further at some of the repercussions.  The most
obvious immediate one is that a program that needs 10MB of live data
will now use up 30MB of heap (10MB of live data at the last GC plus upto
20MB of data allocated since the last GC), so it will increase the
process sizes in a non-trivial way.

I'd lean towards 1.0 instead of 2.0 for that kind of reason.

I also took a look at related data.  E.g. comparing p=1.0 to the old
p-0.1 on the process that performs the highest number of GCs (188 cycles
with p=1.0 and 650 cycles with p=0.1) during an Emacs build.  Here are
the corresponding last few GC cycles:

    % grep GC-26227 ./+make-0.1.log | tail -n 10              
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.6M free=2.0M thresold=1.9M
    GC-26227 p=0.1 total=18.7M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.7M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.8M thresold=1.9M
    GC-26227 p=0.1 total=145.7M free=32.6M thresold=14.6M
    GC-26227 p=0.1 total=132.8M free=60.7M thresold=13.3M
    % grep GC-898 ./+make-1.0.log | tail -n 10              
    GC-898 p=1.0 total=18.5M free=7.5M thresold=18.5M
    GC-898 p=1.0 total=18.6M free=7.4M thresold=18.6M
    GC-898 p=1.0 total=18.6M free=7.4M thresold=18.6M
    GC-898 p=1.0 total=18.7M free=7.4M thresold=18.7M
    GC-898 p=1.0 total=18.7M free=7.3M thresold=18.7M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=145.7M free=32.6M thresold=145.7M
    % 

Obviously this process ends up with a very large one-step allocation
which is arguably interesting in itself (I suspect there's some
GC-inhibition going on there), but the more interesting point
is that with p=0.1 we get an amount of free space after GC
(i.e. blocks we can't release, because of fragmentation) that's about as
large as the next threshold, i.e. about 10%, whereas with p=1.0 this
amount of unreleasable free space is significantly higher.  Most likely
this is not wasted space: a lot of it will be re-used for new
allocations, but still with 19MB of live data, we end up with 7MB of
space we can't release back.

I think p=1.0 (and the corresponding implication that we use up about
twice as much memory as the minimum we need) might be an acceptable
tradeoff (for batch use), but I don't think I'd be comfortable going
beyond that.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Obviously this process ends up with a very large one-step allocation
> which is arguably interesting in itself (I suspect there's some
> GC-inhibition going on there), but the more interesting point
> is that with p=0.1 we get an amount of free space after GC
> (i.e. blocks we can't release, because of fragmentation) that's about as
> large as the next threshold, i.e. about 10%, whereas with p=1.0 this
> amount of unreleasable free space is significantly higher.

As far as I know, we never actually release back cons space to the OS
anyway -- even if we could.  (Because glibc's malloc implementation
doesn't do that unless we call that trim function, which we don't.)  So
I'm not sure that makes much difference -- especially in a batch process.

> I think p=1.0 (and the corresponding implication that we use up about
> twice as much memory as the minimum we need) might be an acceptable
> tradeoff (for batch use), but I don't think I'd be comfortable going
> beyond that.

Right.  We can still have the Emacs makefiles increase the threshold for
build purposes even if we settle on a lower general level for --batch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> Obviously this process ends up with a very large one-step allocation
>> which is arguably interesting in itself (I suspect there's some
>> GC-inhibition going on there), but the more interesting point
>> is that with p=0.1 we get an amount of free space after GC
>> (i.e. blocks we can't release, because of fragmentation) that's about as
>> large as the next threshold, i.e. about 10%, whereas with p=1.0 this
>> amount of unreleasable free space is significantly higher.
> As far as I know, we never actually release back cons space to the OS
> anyway -- even if we could.

But we do release to the malloc library, which means it can be reused
for other things (instead of being constrained to only contain cons
cells, for example).   Whether that library in turn releases it back to
the OS is out of our control (and apparently current glibc never does
so).

>> I think p=1.0 (and the corresponding implication that we use up about
>> twice as much memory as the minimum we need) might be an acceptable
>> tradeoff (for batch use), but I don't think I'd be comfortable going
>> beyond that.
>
> Right.  We can still have the Emacs makefiles increase the threshold for
> build purposes even if we settle on a lower general level for --batch.

We could, I guess, tho the performance difference between p=1.0 and
p=2.0 isn't very significant.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But we do release to the malloc library, which means it can be reused
> for other things (instead of being constrained to only contain cons
> cells, for example).

Yes, that's true.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Sat, 18 Jun 2022 14:49:49 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > Obviously this process ends up with a very large one-step allocation
> > which is arguably interesting in itself (I suspect there's some
> > GC-inhibition going on there), but the more interesting point
> > is that with p=0.1 we get an amount of free space after GC
> > (i.e. blocks we can't release, because of fragmentation) that's about as
> > large as the next threshold, i.e. about 10%, whereas with p=1.0 this
> > amount of unreleasable free space is significantly higher.
> 
> As far as I know, we never actually release back cons space to the OS
> anyway -- even if we could.  (Because glibc's malloc implementation
> doesn't do that unless we call that trim function, which we don't.)

AFAIK, that's not really true.  We call 'free' and glibc does release
to the OS when it can.  And don't forget that Emacs calls 'malloc' a
lot not just for conses and other Lisp data.

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIK, that's not really true.  We call 'free' and glibc does release
> to the OS when it can.  And don't forget that Emacs calls 'malloc' a
> lot not just for conses and other Lisp data.

Wasn't there a huge discussion about this a couple months ago and the
conclusion was that glibc rarely releases memory back to the OS?  That's
why we added `malloc-trim'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  yantar92@gmail.com,  mattiase@acm.org,
>   theophilusx@gmail.com,  rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 19 Jun 2022 13:02:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIK, that's not really true.  We call 'free' and glibc does release
> > to the OS when it can.  And don't forget that Emacs calls 'malloc' a
> > lot not just for conses and other Lisp data.
> 
> Wasn't there a huge discussion about this a couple months ago and the
> conclusion was that glibc rarely releases memory back to the OS?  That's
> why we added `malloc-trim'.

Well, "rarely" is not the same as "never".

How rare is "rarely" really depends on the specific use patterns, and
I don't think we have ever established what kinds of uses cause us to
release memory more frequently than others.

Maybe it's worth our while to collect some data about this?  For
example, frequent contributors who tend to run Emacs for many days
could perhaps take a snapshot of the total VM of the Emacs process and
see how it behaves.  Then we'd know how frequently the footprint goes
down and by how much.

I can tell from my own experience that, when I purge my mail INBOX,
typically leaving about 2500 email messages in it down from ~4500, the
memory footprint of the Emacs process goes down from about 620 MB to
about 520-550 MB.  But this is not with glibc.

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Sun, Jun 19, 2022, 7:05 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > AFAIK, that's not really true.  We call 'free' and glibc does release
> > to the OS when it can.  And don't forget that Emacs calls 'malloc' a
> > lot not just for conses and other Lisp data.
>
> Wasn't there a huge discussion about this a couple months ago and the
> conclusion was that glibc rarely releases memory back to the OS?  That's
> why we added `malloc-trim'.
>

Was there?  I see:.  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71
but it's a little more than a couple of months ago.
I'm pretty curious because if I accumulate a large buffer of trace output
(running memory up to 100s of MB), killing the buffer doesn't seem to
impact gc time substantially.
I would think using mmap-based allocation would make release of memory to
the system more likely.  But mmap is only used for 64+KiB size blocks, and
the trim threshold is 128KiB blocks (shouldn't it be lower than the
allocation size?)
I'm also interested in the improvement in performance reported in that
thread from using the jemalloc implementation.  Has that been investigated
further?

Thanks,
Lynn

[-- Attachment #2: Type: text/html, Size: 2007 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

On Mon, Jun 20, 2022 at 10:01 PM Lynn Winebarger <owinebar@gmail.com> wrote:

> On Sun, Jun 19, 2022, 7:05 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> > AFAIK, that's not really true.  We call 'free' and glibc does release
>> > to the OS when it can.  And don't forget that Emacs calls 'malloc' a
>> > lot not just for conses and other Lisp data.
>>
>> Wasn't there a huge discussion about this a couple months ago and the
>> conclusion was that glibc rarely releases memory back to the OS?  That's
>> why we added `malloc-trim'.
>>
>
> Was there?  I see:.
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71
> but it's a little more than a couple of months ago.
> I'm pretty curious because if I accumulate a large buffer of trace output
> (running memory up to 100s of MB), killing the buffer doesn't seem to
> impact gc time substantially.
> I would think using mmap-based allocation would make release of memory to
> the system more likely.  But mmap is only used for 64+KiB size blocks, and
> the trim threshold is 128KiB blocks (shouldn't it be lower than the
> allocation size?)
> I'm also interested in the improvement in performance reported in that
> thread from using the jemalloc implementation.  Has that been investigated
> further?
>
> I tried jemalloc per Ihor's description, using LD_PRELOAD to have it
replace the system malloc without
having to change or even recompile any emacs code.

My test was to start a shell in a buffer, and then cat about 32MB of csv
data to the buffer.  It seemed
like a reasonable approximation of generating large buffers of tracing
output for this purpose.
I started emacs 28.1 from a build directory with
MALLOC_CONF=background_thread:true,retain:false ./src/emacs

As expected, the emacs process slowly added about 50MB of memory to RSS.
After the cat had completed,
I killed the buffer and ran (garbage-collect).
The RSS and VSZ of the process dropped about 40MB.
Not complete reclamation, but I expect that it would solve the problem of a
permanent bump in gc pause time.
For some reason, the configure script chooses not to use mmap for malloc.
jemalloc is designed to use mmap
as much as possible so it isn't limited to freeing only the uppermost
regions.
I can't tell what (if any) option allows me to overrule the configure
scripts decision to use sbrk instead of mmap.
That might help make the mallopt knobs more effective.

Are there any formal memory-allocation/reclamation benchmarks that I should
use instead of this ad hoc task?

Lynn

[-- Attachment #2: Type: text/html, Size: 3735 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Tue, 21 Jun 2022 20:01:43 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Ihor Radchenko <yantar92@gmail.com>, Mattias Engdegård <mattiase@acm.org>, 
> 	Tim Cross <theophilusx@gmail.com>, rms@gnu.org, Alan Mackenzie <acm@muc.de>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> For some reason, the configure script chooses not to use mmap for malloc.

The configure script chooses not to use mmap directly because glibc's
malloc already does that internally where it decides that to be
beneficial, and because using mmap directly has some disadvantages
we'd like to avoid if possible.

> jemalloc is designed to use mmap as much as possible

So is glibc.

> so it isn't limited to freeing only the uppermost regions.

Neither is glibc, although it uses different algorithms.

> I can't tell what (if any) option allows me to overrule the configure
> scripts decision to use sbrk instead of mmap.

You don't want that, not on GNU/Linux systems.  We switched away of
sbrk and mmap for several good reasons.

> Are there any formal memory-allocation/reclamation benchmarks that I should
> use instead of this ad hoc task?

Not that I know of.

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

On Wed, Jun 22, 2022 at 8:59 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Tue, 21 Jun 2022 20:01:43 -0400
> > Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <
> monnier@iro.umontreal.ca>,
> >       Ihor Radchenko <yantar92@gmail.com>, Mattias Engdegård <
> mattiase@acm.org>,
> >       Tim Cross <theophilusx@gmail.com>, rms@gnu.org, Alan Mackenzie <
> acm@muc.de>,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > For some reason, the configure script chooses not to use mmap for malloc.
>
> The configure script chooses not to use mmap directly because glibc's
> malloc already does that internally where it decides that to be
> beneficial, and because using mmap directly has some disadvantages
> we'd like to avoid if possible.
>
> > jemalloc is designed to use mmap as much as possible
>
> So is glibc.
>

Wow, so it is - when I tried the same test without jemalloc, both 28.1 and
24.3 gave
similar results.  I must have misinterpreted the behavior of edebug's
trace-buffer as
being similar to issues I've had previously (probably due to not using an
explicit
garbage-collect,  or killing rather than deleting text with undo disabled).
Maybe I just buried the trace buffer instead of killing it.

> so it isn't limited to freeing only the uppermost regions.
>
> Neither is glibc, although it uses different algorithms.
>
> > I can't tell what (if any) option allows me to overrule the configure
> > scripts decision to use sbrk instead of mmap.
>
> You don't want that, not on GNU/Linux systems.  We switched away of
> sbrk and mmap for several good reasons.
>

Not that I don't enjoy trawling through old threads in the emacs-devel
archive
(searching for mmap led to a thread from the introduction of the portable
dumper),
but it would be a lot easier if there were design notes somewhere recording
the rationale of the decisions reflected in the current code.  The best I
can do now
is comb through the arguments on the list (and maybe in the bug tracker?)
and
try to guess which points ended up getting reflected in the code.

I will say jemalloc appears to have a lot of heap profiling bells and
whistles.

[-- Attachment #2: Type: text/html, Size: 3532 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 22 Jun 2022 19:30:20 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Ihor Radchenko <yantar92@gmail.com>, Mattias Engdegård <mattiase@acm.org>, 
> 	Tim Cross <theophilusx@gmail.com>, rms@gnu.org, Alan Mackenzie <acm@muc.de>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
>  > I can't tell what (if any) option allows me to overrule the configure
>  > scripts decision to use sbrk instead of mmap.
> 
>  You don't want that, not on GNU/Linux systems.  We switched away of
>  sbrk and mmap for several good reasons.
> 
> Not that I don't enjoy trawling through old threads in the emacs-devel archive 
> (searching for mmap led to a thread from the introduction of the portable dumper),
> but it would be a lot easier if there were design notes somewhere recording
> the rationale of the decisions reflected in the current code.

We lack someone in the role of the "project historian", who would then
summarize and publish the design discussions in the form of such
notes.  Volunteers are most welcome.

> The best I can do now
> is comb through the arguments on the list (and maybe in the bug tracker?) and
> try to guess which points ended up getting reflected in the code.

A better method is to use Git logs to find when the GNU/Linux build
stopped using mmap, and then search the archives around that time.

> I will say jemalloc appears to have a lot of heap profiling bells and whistles. 

AFAIU, so does glibc.  Some of them where mentioned in the discussions
you already unearthed, AFAIR.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Not that I don't enjoy trawling through old threads in the emacs-devel archive 
>> (searching for mmap led to a thread from the introduction of the portable dumper),
>> but it would be a lot easier if there were design notes somewhere recording
>> the rationale of the decisions reflected in the current code.
>
> We lack someone in the role of the "project historian", who would then
> summarize and publish the design discussions in the form of such
> notes.  Volunteers are most welcome.

What about simply following a convention to link relevant discussion in
the commit message?

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Lynn Winebarger <owinebar@gmail.com>,  larsi@gnus.org,
>   monnier@iro.umontreal.ca,  mattiase@acm.org,  theophilusx@gmail.com,
>   rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 23 Jun 2022 15:18:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Not that I don't enjoy trawling through old threads in the emacs-devel archive 
> >> (searching for mmap led to a thread from the introduction of the portable dumper),
> >> but it would be a lot easier if there were design notes somewhere recording
> >> the rationale of the decisions reflected in the current code.
> >
> > We lack someone in the role of the "project historian", who would then
> > summarize and publish the design discussions in the form of such
> > notes.  Volunteers are most welcome.
> 
> What about simply following a convention to link relevant discussion in
> the commit message?

That convention does exist, and the results of its use are before our
eyes ;-).  It doesn't help that Git doesn't support retroactive
modifications of commit logs, except via cumbersome workarounds that
no one seems to like using.

And that won't help with past discussions whose commits are in the
tree for many years.

But yes, using such links is definitely helpful, and I'd encourage
people to use that more.  (That's one reason why changes should
preferably be discussed on the bug tracker: the bug number -- if not
forgotten to be mentioned in the log message -- serves as a pointer to
the discussions.)

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> What about simply following a convention to link relevant discussion in
>> the commit message?
>
> That convention does exist, and the results of its use are before our
> eyes ;-).  It doesn't help that Git doesn't support retroactive
> modifications of commit logs, except via cumbersome workarounds that
> no one seems to like using.

Good to know.

> And that won't help with past discussions whose commits are in the
> tree for many years.

Can't the past discussions be directly updated by sending an extra email
that links back to the new commit?

> But yes, using such links is definitely helpful, and I'd encourage
> people to use that more.  (That's one reason why changes should
> preferably be discussed on the bug tracker: the bug number -- if not
> forgotten to be mentioned in the log message -- serves as a pointer to
> the discussions.)

What about the message ID? It should also be unique enough to link to
the relevant ML thread. (it is a bit annoying that lists.gnu.org does
not support easy search via link id, unlike e.g.
https://yhetil.org/emacs-devel/87r13n5muk.fsf@gnus.org/).

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: owinebar@gmail.com,  larsi@gnus.org,  monnier@iro.umontreal.ca,
>   mattiase@acm.org,  theophilusx@gmail.com,  rms@gnu.org,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Thu, 23 Jun 2022 16:02:36 +0800
> 
> > And that won't help with past discussions whose commits are in the
> > tree for many years.
> 
> Can't the past discussions be directly updated by sending an extra email
> that links back to the new commit?

You mean, rely on search engines to find an addition to a discussion
made many years later?  My experience with searching the archives is
that it many times fails when the discussion crosses the year mark.
For starters, display of the archives by thread doesn't work in that
case as expected: you are given the illusion that the thread ends.

Searching the archives also has the disadvantage that in many cases
it's hard to know what are the keywords that would find the discussion
efficiently (i.e. without drowning it in thousands of irrelevant
messages).

Otherwise, I don't see how sending a message to a thread long dead
would help in this matter.  Am I missing something?

> > But yes, using such links is definitely helpful, and I'd encourage
> > people to use that more.  (That's one reason why changes should
> > preferably be discussed on the bug tracker: the bug number -- if not
> > forgotten to be mentioned in the log message -- serves as a pointer to
> > the discussions.)
> 
> What about the message ID? It should also be unique enough to link to
> the relevant ML thread. (it is a bit annoying that lists.gnu.org does
> not support easy search via link id, unlike e.g.
> https://yhetil.org/emacs-devel/87r13n5muk.fsf@gnus.org/).

That's okay, but is it really better than a URL of a message in the
archives?  It isn't much shorter.

The basic problem is to remember to mention a discussion, in any
form.  Even just a date and the name of the forum are usually enough
to quickly find the relevant thread.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Can't the past discussions be directly updated by sending an extra email
>> that links back to the new commit?
>
> You mean, rely on search engines to find an addition to a discussion
> made many years later?  My experience with searching the archives is
> that it many times fails when the discussion crosses the year mark.
> For starters, display of the archives by thread doesn't work in that
> case as expected: you are given the illusion that the thread ends.

Are you referring to lists.gnu.org thread display interface?
If it is as buggy as you are describing, why don't you file a bug
report the relevant ML and get the problem fixed?

There are alternatives. I personally prefer to search using local
maildir copy via notmuch. I believe that gnus should also be able to
search across the maildir.

Also, there is https://yhetil.org/ with somewhat better search and
display interface (IMHO). See
https://yhetil.org/emacs-devel/_/text/help/

> Searching the archives also has the disadvantage that in many cases
> it's hard to know what are the keywords that would find the discussion
> efficiently (i.e. without drowning it in thousands of irrelevant
> messages).

When I was saying "links back to the new commit", I was referring to the
unique commit number. Searching exact match will not give false
positives then.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: owinebar@gmail.com,  larsi@gnus.org,  monnier@iro.umontreal.ca,
>   mattiase@acm.org,  theophilusx@gmail.com,  rms@gnu.org,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Thu, 23 Jun 2022 17:03:35 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can't the past discussions be directly updated by sending an extra email
> >> that links back to the new commit?
> >
> > You mean, rely on search engines to find an addition to a discussion
> > made many years later?  My experience with searching the archives is
> > that it many times fails when the discussion crosses the year mark.
> > For starters, display of the archives by thread doesn't work in that
> > case as expected: you are given the illusion that the thread ends.
> 
> Are you referring to lists.gnu.org thread display interface?

I mean it, but also every other mailing list software I ever used.
They all have these problems, and some more than others.

> If it is as buggy as you are describing, why don't you file a bug
> report the relevant ML and get the problem fixed?

Because I don't believe this is fixable in practice.

> There are alternatives. I personally prefer to search using local
> maildir copy via notmuch. I believe that gnus should also be able to
> search across the maildir.

I have local search set up as well, but that only works on messages I
decided to archive locally.  If you think I'm archiving every forum in
which I participate, think again ;-)

> Also, there is https://yhetil.org/ with somewhat better search and
> display interface (IMHO). See
> https://yhetil.org/emacs-devel/_/text/help/

How is it significantly better?

> > Searching the archives also has the disadvantage that in many cases
> > it's hard to know what are the keywords that would find the discussion
> > efficiently (i.e. without drowning it in thousands of irrelevant
> > messages).
> 
> When I was saying "links back to the new commit", I was referring to the
> unique commit number. Searching exact match will not give false
> positives then.

What do you do with this when we change the VCS (which happened twice
already)?

And even if you are only talking about Git SHA signatures, the "no
false positives" is not guaranteed, when messages provide only the
7-digit SHA.

And in any case, the trigger for this discussion was the situation
where you want to answer questions like "why did Emacs stop using sbrk
on GNU/Linux", in which case there's no commit ID to search for.

Moreover, I don't understand the proposal in general: are you
suggesting that every commit related to a discussion would somehow
send a message to the thread with the commit ID?  If so, how would
this work better than our (constantly failing) practice of mentioning
the bug number in the commit log messages?

These measures don't really work without a gatekeeper.  Which we don't
have, and probably never will.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > You mean, rely on search engines to find an addition to a discussion
>> > made many years later?  My experience with searching the archives is
>> > that it many times fails when the discussion crosses the year mark.
>> > For starters, display of the archives by thread doesn't work in that
>> > case as expected: you are given the illusion that the thread ends.
>> 
>> Are you referring to lists.gnu.org thread display interface?
>
> I mean it, but also every other mailing list software I ever used.
> They all have these problems, and some more than others.
>
>> If it is as buggy as you are describing, why don't you file a bug
>> report the relevant ML and get the problem fixed?
>
> Because I don't believe this is fixable in practice.

Then it raises a bigger question. Do we have anything better than email
archives to archive emacs-devel?

>> There are alternatives. I personally prefer to search using local
>> maildir copy via notmuch. I believe that gnus should also be able to
>> search across the maildir.
>
> I have local search set up as well, but that only works on messages I
> decided to archive locally.  If you think I'm archiving every forum in
> which I participate, think again ;-)

I did expect that you at least archive emacs-devel (:

>> Also, there is https://yhetil.org/ with somewhat better search and
>> display interface (IMHO). See
>> https://yhetil.org/emacs-devel/_/text/help/
>
> How is it significantly better?

Search is more flexible because it at least allows search by date and by
some more email fields.
Compare search section of https://yhetil.org/emacs-devel/_/text/help/
and https://lists.gnu.org/archive/cgi-bin/namazu.cgi?idxname=bug-gnu-emacs

>> > Searching the archives also has the disadvantage that in many cases
>> > it's hard to know what are the keywords that would find the discussion
>> > efficiently (i.e. without drowning it in thousands of irrelevant
>> > messages).
>> 
>> When I was saying "links back to the new commit", I was referring to the
>> unique commit number. Searching exact match will not give false
>> positives then.
>
> What do you do with this when we change the VCS (which happened twice
> already)?

Of course, things will break if we change VCS, but it should not stop us
from linking to commits - not linking at all is still worse.

Note that I can also ask "what if we change debbugs?". Would it mean
that we should not link to bug#?

> And even if you are only talking about Git SHA signatures, the "no
> false positives" is not guaranteed, when messages provide only the
> 7-digit SHA.

Sure. But false positives will not be numerous. You can expect a few max
- not a big deal to check one by one.

> And in any case, the trigger for this discussion was the situation
> where you want to answer questions like "why did Emacs stop using sbrk
> on GNU/Linux", in which case there's no commit ID to search for.

I implied using git log search to identify the relevant commits.

> Moreover, I don't understand the proposal in general: are you
> suggesting that every commit related to a discussion would somehow
> send a message to the thread with the commit ID?  If so, how would
> this work better than our (constantly failing) practice of mentioning
> the bug number in the commit log messages?

I do not imply that we should do it _instead_ of mentioning bug number.
It's complementary and can be very useful (I know because I did have a
hard time finding commits that fixed particular bugs).
Also, bugs already need to be closed manually - that closing email
directive could as well include the commit hash.

> These measures don't really work without a gatekeeper.  Which we don't
> have, and probably never will.

I do not think that it is so much of an issue. Your argument would also
apply, e.g. to using double space between sentences in the commit
messages or to following log entry format. Yet people do follow these
conventions because they are documented in CONTRIBUTE file and because
non-compliant commits are being scolded. Not 100%, but frequently enough
to make the conventions useful.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: owinebar@gmail.com,  larsi@gnus.org,  monnier@iro.umontreal.ca,
>   mattiase@acm.org,  theophilusx@gmail.com,  rms@gnu.org,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Sat, 25 Jun 2022 13:10:19 +0800
> 
> >> > For starters, display of the archives by thread doesn't work in that
> >> > case as expected: you are given the illusion that the thread ends.
> >> 
> >> Are you referring to lists.gnu.org thread display interface?
> >
> > I mean it, but also every other mailing list software I ever used.
> > They all have these problems, and some more than others.
> >
> >> If it is as buggy as you are describing, why don't you file a bug
> >> report the relevant ML and get the problem fixed?
> >
> > Because I don't believe this is fixable in practice.
> 
> Then it raises a bigger question. Do we have anything better than email
> archives to archive emacs-devel?

I don't know of any.

> >> https://yhetil.org/emacs-devel/_/text/help/
> >
> > How is it significantly better?
> 
> Search is more flexible because it at least allows search by date and by
> some more email fields.
> Compare search section of https://yhetil.org/emacs-devel/_/text/help/
> and https://lists.gnu.org/archive/cgi-bin/namazu.cgi?idxname=bug-gnu-emacs

I don't see anything that would make the search more efficient.  In
particular, all the different method of searching for matches in
various parts of the message or in practice of no help at all, since I
basically have no idea where the keyword appeared when I start
searching.

> Note that I can also ask "what if we change debbugs?". Would it mean
> that we should not link to bug#?

The bug number gives you a quick way of reaching the bug discussion in
the email archives of bug-gnu-emacs, and in debbugs itself.  These
will remain available even if we switch away from debbugs as our issue
tracker.  So the bug number is definitely a good thing to have in the
logs.

> > And in any case, the trigger for this discussion was the situation
> > where you want to answer questions like "why did Emacs stop using sbrk
> > on GNU/Linux", in which case there's no commit ID to search for.
> 
> I implied using git log search to identify the relevant commits.

That doesn't work well IME.  Would you mind walking me though using
that when trying to answer the above question about sbrk?

> > These measures don't really work without a gatekeeper.  Which we don't
> > have, and probably never will.
> 
> I do not think that it is so much of an issue. Your argument would also
> apply, e.g. to using double space between sentences in the commit
> messages or to following log entry format. Yet people do follow these
> conventions because they are documented in CONTRIBUTE file and because
> non-compliant commits are being scolded. Not 100%, but frequently enough
> to make the conventions useful.

Two spaces between sentences is simple enough that we have several
eyes looking for that and fixing problems post-mortem.  And yet we
still have quite a few of them, just try searching in the Emacs tree.
Conventions that are harder to spot are followed even less.

As a matter of fact, I have hard time making sure each commit that has
a bug number mentions that number in the commit log message.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Then it raises a bigger question. Do we have anything better than email
>> archives to archive emacs-devel?
>
> I don't know of any.

What about Mailman 3 frontends like Hyperkitty?
https://wiki.archlinux.org/title/Hyperkitty#Xapian_search_backend

>> Note that I can also ask "what if we change debbugs?". Would it mean
>> that we should not link to bug#?
>
> The bug number gives you a quick way of reaching the bug discussion in
> the email archives of bug-gnu-emacs, and in debbugs itself.  These
> will remain available even if we switch away from debbugs as our issue
> tracker.  So the bug number is definitely a good thing to have in the
> logs.

Will the commit hash not be available e.g. on savannah? Do you intend to
remove the git history alltogether during the switch?

>> > And in any case, the trigger for this discussion was the situation
>> > where you want to answer questions like "why did Emacs stop using sbrk
>> > on GNU/Linux", in which case there's no commit ID to search for.
>> 
>> I implied using git log search to identify the relevant commits.
>
> That doesn't work well IME.  Would you mind walking me though using
> that when trying to answer the above question about sbrk?

1. I listed all the commits that mentioned sbrk in the messages using
   git log, displaying the statistics. (actually, I used Magit)
2. I noticed that some of the commits changed large number of lines and
   removed sbrk-related staff. In particular, d12e5d003d Add portable
   dumper
3. I searched https://yhetil.org/emacs-devel/?q=linux+sbrk and noticed
   two candidate threads: Re: When should ralloc.c be used?; and Re:
   Dumper issue, revisited; invalid realloc/free;
4. Looking through the threads; it appears that the pdumper thread had a
   lot of occurrences of sbrk word:
   https://yhetil.org/emacs-devel/?q=%22sbrk%22
5. I skimmed through the thread and possibly found relevant message
   root:
   https://yhetil.org/emacs-devel/20140625212421.GD179@brightrain.aerifal.cx/

It would have been even easier if I had an idea what sbrk does and where
its Linux support is supposed to be in the code.

>> > These measures don't really work without a gatekeeper.  Which we don't
>> > have, and probably never will.
>> 
>> I do not think that it is so much of an issue. Your argument would also
>> apply, e.g. to using double space between sentences in the commit
>> messages or to following log entry format. Yet people do follow these
>> conventions because they are documented in CONTRIBUTE file and because
>> non-compliant commits are being scolded. Not 100%, but frequently enough
>> to make the conventions useful.
>
> Two spaces between sentences is simple enough that we have several
> eyes looking for that and fixing problems post-mortem.  And yet we
> still have quite a few of them, just try searching in the Emacs tree.
> Conventions that are harder to spot are followed even less.
>
> As a matter of fact, I have hard time making sure each commit that has
> a bug number mentions that number in the commit log message.

On Org side, I do not find tracking such things too difficult. Probably
due to lesser volume of commits.

Note that things like double space, or even bug number checks could be
automated. For example, one can write a make target that checks recent
commit messages to follow specific rules.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: owinebar@gmail.com,  larsi@gnus.org,  monnier@iro.umontreal.ca,
>   mattiase@acm.org,  theophilusx@gmail.com,  rms@gnu.org,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Mon, 27 Jun 2022 17:32:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Then it raises a bigger question. Do we have anything better than email
> >> archives to archive emacs-devel?
> >
> > I don't know of any.
> 
> What about Mailman 3 frontends like Hyperkitty?
> https://wiki.archlinux.org/title/Hyperkitty#Xapian_search_backend

Not sure what that means in practice.  Do you mean I should install
that locally?  Or what else?

You asked, above, do we have a better way of archiving a mailing
list, and the solution you propose seems to be yet another way of
keeping email archives?  I must be missing something.

> >> Note that I can also ask "what if we change debbugs?". Would it mean
> >> that we should not link to bug#?
> >
> > The bug number gives you a quick way of reaching the bug discussion in
> > the email archives of bug-gnu-emacs, and in debbugs itself.  These
> > will remain available even if we switch away from debbugs as our issue
> > tracker.  So the bug number is definitely a good thing to have in the
> > logs.
> 
> Will the commit hash not be available e.g. on savannah? Do you intend to
> remove the git history alltogether during the switch?

No, but I don't want to look in more than one repository, either.

> >> > And in any case, the trigger for this discussion was the situation
> >> > where you want to answer questions like "why did Emacs stop using sbrk
> >> > on GNU/Linux", in which case there's no commit ID to search for.
> >> 
> >> I implied using git log search to identify the relevant commits.
> >
> > That doesn't work well IME.  Would you mind walking me though using
> > that when trying to answer the above question about sbrk?
> 
> 1. I listed all the commits that mentioned sbrk in the messages using
>    git log, displaying the statistics. (actually, I used Magit)
> 2. I noticed that some of the commits changed large number of lines and
>    removed sbrk-related staff. In particular, d12e5d003d Add portable
>    dumper
> 3. I searched https://yhetil.org/emacs-devel/?q=linux+sbrk and noticed
>    two candidate threads: Re: When should ralloc.c be used?; and Re:
>    Dumper issue, revisited; invalid realloc/free;

How did you decide that these threads are relevant?

> 4. Looking through the threads; it appears that the pdumper thread had a
>    lot of occurrences of sbrk word:
>    https://yhetil.org/emacs-devel/?q=%22sbrk%22
> 5. I skimmed through the thread and possibly found relevant message
>    root:
>    https://yhetil.org/emacs-devel/20140625212421.GD179@brightrain.aerifal.cx/

How much time did this take you?

> It would have been even easier if I had an idea what sbrk does and where
> its Linux support is supposed to be in the code.

sbrk is a system call, so I guess by "its Linux support" you mean
where we used to use it Emacs?  The answer to that is not always easy
to find in practice, unless you happen to know.

> > As a matter of fact, I have hard time making sure each commit that has
> > a bug number mentions that number in the commit log message.
> 
> On Org side, I do not find tracking such things too difficult. Probably
> due to lesser volume of commits.

What do you mean by "tracking"?  By "making sure" I meant the job of
having people who have write access mention the bug number.  Detecting
they missed after the commit is pushed is too late.

> Note that things like double space, or even bug number checks could be
> automated. For example, one can write a make target that checks recent
> commit messages to follow specific rules.

Commit log messages in Git are immutable, so looking up such problems
post-factum is not really useful.  But that is a tangent; the main
point here is that we have quite a few good practices and conventions
in place, but have problems living up to them.  So any procedure that
relies too heavily on these conventions and practices will not be too
successful, unless what it relies on is completely automated.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> Then it raises a bigger question. Do we have anything better than email
>> >> archives to archive emacs-devel?
>> >
>> > I don't know of any.
>> 
>> What about Mailman 3 frontends like Hyperkitty?
>> https://wiki.archlinux.org/title/Hyperkitty#Xapian_search_backend
>
> Not sure what that means in practice.  Do you mean I should install
> that locally?  Or what else?
>
> You asked, above, do we have a better way of archiving a mailing
> list, and the solution you propose seems to be yet another way of
> keeping email archives?  I must be missing something.

I am trying to challenge your statement that thread display is bad in
mailing list web frontends. I fail to believe that the problem with year
crossing is impossible to solve. So, I suggested that a newer Mailman
frontend might not have the described problem. Can we use it in
lists.gnu.org instead of the older web interface?

>> Will the commit hash not be available e.g. on savannah? Do you intend to
>> remove the git history alltogether during the switch?
>
> No, but I don't want to look in more than one repository, either.

My initial suggestion was bi-directional linking between the commit
messages and the relevant ML discussions. There is already a convention
to link from commit messages to debbugs, but it would be useful to have
the backlinks from debbugs/ML to the relevant commits. Via commit hash
or possibly via direct savannah link (whatever is more permanent).

I do not think that there is any way to have the commits directly in
debbugs. (Or am I wrong?)

>> > That doesn't work well IME.  Would you mind walking me though using
>> > that when trying to answer the above question about sbrk?
>> 
>> 1. I listed all the commits that mentioned sbrk in the messages using
>>    git log, displaying the statistics. (actually, I used Magit)
>> 2. I noticed that some of the commits changed large number of lines and
>>    removed sbrk-related staff. In particular, d12e5d003d Add portable
>>    dumper
>> 3. I searched https://yhetil.org/emacs-devel/?q=linux+sbrk and noticed
>>    two candidate threads: Re: When should ralloc.c be used?; and Re:
>>    Dumper issue, revisited; invalid realloc/free;
>
> How did you decide that these threads are relevant?

ralloc.c and pdumper where the keywords used both in the commits and in
the threads. And there were significant number of matching messages in
the threads.

>> 4. Looking through the threads; it appears that the pdumper thread had a
>>    lot of occurrences of sbrk word:
>>    https://yhetil.org/emacs-devel/?q=%22sbrk%22
>> 5. I skimmed through the thread and possibly found relevant message
>>    root:
>>    https://yhetil.org/emacs-devel/20140625212421.GD179@brightrain.aerifal.cx/
>
> How much time did this take you?

The total time I spent replying to your email was 42 minutes (recorded).
Out of those 42 minutes I subjectively spent 10-15 minutes trying to
answer "Would you mind walking me though using that when trying to
answer the above question about sbrk?". Not more than 25 minutes max.

>> > As a matter of fact, I have hard time making sure each commit that has
>> > a bug number mentions that number in the commit log message.
>> 
>> On Org side, I do not find tracking such things too difficult. Probably
>> due to lesser volume of commits.
>
> What do you mean by "tracking"?  By "making sure" I meant the job of
> having people who have write access mention the bug number.  Detecting
> they missed after the commit is pushed is too late.

>> Note that things like double space, or even bug number checks could be
>> automated. For example, one can write a make target that checks recent
>> commit messages to follow specific rules.
>
> Commit log messages in Git are immutable, so looking up such problems
> post-factum is not really useful.  But that is a tangent; the main
> point here is that we have quite a few good practices and conventions
> in place, but have problems living up to them.  So any procedure that
> relies too heavily on these conventions and practices will not be too
> successful, unless what it relies on is completely automated.

You are right. I mostly had to deal with people who do not have write
access.

As for automatic checks for all the committers, what about git pre-push
hooks (https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks)?
If one can write make commit_check target, it can be configured to run
before pushing to Emacs upstream.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Yuri Khan]

On Sat, 25 Jun 2022 at 12:10, Ihor Radchenko <yantar92@gmail.com> wrote:

> > What do you do with this when we change the VCS (which happened twice
> > already)?
>
> Of course, things will break if we change VCS, but it should not stop us
> from linking to commits - not linking at all is still worse.

Identifying VCS commits in a way that is stable across VCS migration
is at least theoretically a solved problem.
http://esr.ibiblio.org/?p=3872

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sat, 25 Jun 2022 14:51:14 +0700
> Cc: Eli Zaretskii <eliz@gnu.org>, owinebar@gmail.com, 
> 	Lars Magne Ingebrigtsen <larsi@gnus.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Mattias Engdegård <mattiase@acm.org>, theophilusx@gmail.com, 
> 	Richard Stallman <rms@gnu.org>, Alan Mackenzie <acm@muc.de>, Emacs developers <emacs-devel@gnu.org>
> 
> On Sat, 25 Jun 2022 at 12:10, Ihor Radchenko <yantar92@gmail.com> wrote:
> 
> > > What do you do with this when we change the VCS (which happened twice
> > > already)?
> >
> > Of course, things will break if we change VCS, but it should not stop us
> > from linking to commits - not linking at all is still worse.
> 
> Identifying VCS commits in a way that is stable across VCS migration
> is at least theoretically a solved problem.
> http://esr.ibiblio.org/?p=3872

We considered that in the past, and decided to use a different method,
see CONTRIBUTE.  And in any case, the results of following that
recommendation are before our eyes...

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 3503 bytes --]

On Thu, Jun 23, 2022 at 3:09 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> > Not that I don't enjoy trawling through old threads in the emacs-devel
> archive
> > (searching for mmap led to a thread from the introduction of the
> portable dumper),
> > but it would be a lot easier if there were design notes somewhere
> recording
> > the rationale of the decisions reflected in the current code.
>
> We lack someone in the role of the "project historian", who would then
> summarize and publish the design discussions in the form of such
> notes.  Volunteers are most welcome.
>

It's not the discussion (or history) per se, but the factors that decided
the current
design choices.  An example would be the #ifdef DOUG_LEA_MALLOC blocks in
lisp_align_malloc
of alloc.c that prevent allocating via mmap.  So far what I've read
indicates
that step is there to enable dumping by unexec (assuming I've understood
correctly).
Ideally there would be a design note (possibly generated from comments,
possibly free standing)
that would explain what the purpose of the restriction is, so you could
tell if it's still
relevant, and any negative impacts it has.  Also, if there are plans to
remove it at
some point, what (if any) the requirements/test for removing it would be,
etc.  By "it",
I mean any hypothetical design point, not just this particular one.
Alternative designs could also be discussed along with the factors that led
to their rejection, if it was significant enough.
Now, those could be derived from mailing list discussions, but I wouldn't
consider them
"history".  Per subsequent mails in this thread, if a developer or code
reviewer made
a practice of citing particular mail messages (or even other sources), some
extraction
process might even be automated, but then there would be the question of
copyright
assignment.  It would help if there were some taxonomy of features/design
points for
reference.  Is the bug database being used for that?

Actually, a good example (even though it's more post facto description than
prospective
design & cost-benefit tradeoff analysis) would be
https://github.com/rocky/elisp-bytecode
That is really useful documentation that would ideally be in the emacs docs
or etc directory.
Andrea's development log and paper on the compiler are another example.
The development log would benefit (in terms of use as design notes) from
de-chronologizing
them, figuring out the right taxonomy of the various topics, and gathering
the material accordingly.

I put in a request to assign@gnu.org a few days ago, but I have not
received any response yet.


>
> > The best I can do now
> > is comb through the arguments on the list (and maybe in the bug
> tracker?) and
> > try to guess which points ended up getting reflected in the code.
>
> A better method is to use Git logs to find when the GNU/Linux build
> stopped using mmap, and then search the archives around that time.
>
> > I will say jemalloc appears to have a lot of heap profiling bells and
> whistles.
>
> AFAIU, so does glibc.  Some of them where mentioned in the discussions
> you already unearthed, AFAIR.
>

I figured it would.  jemalloc's maintainers just spend a lot of effort
highlighting those features in the
documentation.  I wouldn't be surprised if it was 3/4 of the material on
the package's site
is dedicated to how to use jemalloc's profiling for debugging or optimizing
memory allocation in the calling program.  It makes an impression.

Lynn

[-- Attachment #2: Type: text/html, Size: 4654 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Thu, 23 Jun 2022 13:07:44 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Ihor Radchenko <yantar92@gmail.com>, Mattias Engdegård <mattiase@acm.org>, 
> 	Tim Cross <theophilusx@gmail.com>, rms@gnu.org, Alan Mackenzie <acm@muc.de>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> > We lack someone in the role of the "project historian", who would then
> > summarize and publish the design discussions in the form of such
> > notes.  Volunteers are most welcome.
> >
> 
> It's not the discussion (or history) per se, but the factors that decided
> the current
> design choices.  An example would be the #ifdef DOUG_LEA_MALLOC blocks in
> lisp_align_malloc
> of alloc.c that prevent allocating via mmap.  So far what I've read
> indicates
> that step is there to enable dumping by unexec (assuming I've understood
> correctly).
> Ideally there would be a design note (possibly generated from comments,
> possibly free standing)
> that would explain what the purpose of the restriction is, so you could
> tell if it's still
> relevant, and any negative impacts it has.  Also, if there are plans to
> remove it at
> some point, what (if any) the requirements/test for removing it would be,
> etc.  By "it",
> I mean any hypothetical design point, not just this particular one.
> Alternative designs could also be discussed along with the factors that led
> to their rejection, if it was significant enough.
> Now, those could be derived from mailing list discussions, but I wouldn't
> consider them
> "history".  Per subsequent mails in this thread, if a developer or code
> reviewer made
> a practice of citing particular mail messages (or even other sources), some
> extraction
> process might even be automated, but then there would be the question of
> copyright
> assignment.

What you describe is a full-time job, more or less.  I'd love to have
someone who'd volunteer to record all those points, but we don't have
him/her yet.

The only places where design is document are the large comments in
some source files.

> It would help if there were some taxonomy of features/design
> points for
> reference.  Is the bug database being used for that?

I don't think so (IIUC what you mean by that), and I don't really see
how bugs could serve in that role.  Most of the design changes and
redesigns in Emacs were developed without any bug report, simply
because those who did the job knew that a particular group of problems
needs to be taken care of.

> Actually, a good example (even though it's more post facto description than
> prospective
> design & cost-benefit tradeoff analysis) would be
> https://github.com/rocky/elisp-bytecode
> That is really useful documentation that would ideally be in the emacs docs
> or etc directory.

That's not design description, though.

> I put in a request to assign@gnu.org a few days ago, but I have not
> received any response yet.

Please ping them and CC me and Lars.

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

On Thu, Jun 23, 2022 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
>

> What you describe is a full-time job, more or less.  I'd love to have
> someone who'd volunteer to record all those points, but we don't have
> him/her yet.
>
> The only places where design is document are the large comments in
> some source files.
>
> > It would help if there were some taxonomy of features/design
> > points for
> > reference.  Is the bug database being used for that?
>
> I don't think so (IIUC what you mean by that), and I don't really see
> how bugs could serve in that role.

I agree, but some of the bug reports are classified as something along
the lines of "feature request" or "wishlist".  I haven't found any
other database tracking anything corresponding to features, proposed
or otherwise.  In more formal environments, feature implementation
would be preceded by some kind of ticket describing the problem to be
solved or functionality to be implemented.  I don't expect any
strictly enforced workflow in this kind of project, but that kind of
referent is still useful for the same reasons having an entry in a bug
database to reference is useful.

A crude taxonomy wouldn't be that hard to come up with, then
elaborated with finer points:
   Value Representation
   ELisp Abstract Machine
   Memory Management
   Intermediate Representations
      Core Lisp
      Lisp Assembly
      LIMPLE
   Compiler Transformations
   Serialized Lisp Program Formats
        ELC
        ELN
        PDMP
   etc
Some of these points are already documented in the Elisp manual, of
course, and some in the code comments, and others in the emacs-devel
archives.

Once something is in place, no matter how bare-bones, the key would be
maintainers mandating that any substantive introduction or revision of
features be tied to appropriate entries for the corresponding
design/spec documentation, in the same way and for the same reason
coding standards are enforced.  That's assuming the maintainers
consider such documentation important enough for enabling future
potential contributors and maintainers to hold otherwise useful
contributions in limbo.  And probably that doing so is convenient to
do.

>  Most of the design changes and
> redesigns in Emacs were developed without any bug report, simply
> because those who did the job knew that a particular group of problems
> needs to be taken care of.

It's not like there isn't any discussion or justification of features
offered prior to code being integrated into the main branch. It's more
a challenge of how to weave what's there into a coherent account of
what's going on in the code.

It would just be easier to automate some sort of design note
extraction from the git log if references to mails could be associated
with relevant features.  I've never used org, but maybe there's some
syntax that would be useful?  Or maybe some notation from supercite
for precise pulling of relevant text from list archives?

I feel like if I were a more savvy Emacs user, I would know all of
this functionality is already implemented in some combination of
CEDET-originated tools, one or more tagging systems, the various
interfaces for using the tagging systems, org, etc.

>
> > Actually, a good example (even though it's more post facto description than
> > prospective
> > design & cost-benefit tradeoff analysis) would be
> > https://github.com/rocky/elisp-bytecode
> > That is really useful documentation that would ideally be in the emacs docs
> > or etc directory.
>
> That's not design description, though.

You probably have a more nuanced view than me on this.  It's true,
that document is focused on the specification (the "what") rather than
the (detailed) "how" and "why" - is that what you mean?  Either way,
if you want to understand how the operational semantics of emacs lisp
work in practice, a document of that sort is invaluable.  Without
that, a document explaining the "why" isn't going to be able to be
very concrete.

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Thu, 23 Jun 2022 18:08:53 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Ihor Radchenko <yantar92@gmail.com>, Mattias Engdegård <mattiase@acm.org>, 
> 	Tim Cross <theophilusx@gmail.com>, rms@gnu.org, Alan Mackenzie <acm@muc.de>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> On Thu, Jun 23, 2022 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> 
> > > It would help if there were some taxonomy of features/design
> > > points for
> > > reference.  Is the bug database being used for that?
> >
> > I don't think so (IIUC what you mean by that), and I don't really see
> > how bugs could serve in that role.
> 
> I agree, but some of the bug reports are classified as something along
> the lines of "feature request" or "wishlist".

If you look at those, you will see that almost all of them request
minor features that have no impact on the design whatsoever.  It is
very rare to have a feature request on the bug tracker that changes
the design or introduces new designs.  I would even dare to say it
never happened, definitely not on my watch.

> I haven't found any other database tracking anything corresponding
> to features, proposed or otherwise.

We do have etc/TODO.  Did you look there?

> Once something is in place, no matter how bare-bones, the key would be
> maintainers mandating that any substantive introduction or revision of
> features be tied to appropriate entries for the corresponding
> design/spec documentation, in the same way and for the same reason
> coding standards are enforced.

I think you are greatly underestimating the efforts required to
maintain documentation of this kind, let alone the effort for creating
it basically from scratch.

Programmers usually don't like writing documentation, and most of them
cannot write good documentation even if they did want, in large part
because that take years of practice actually doing it.  In a project
like Emacs, it is nigh impossible to force contributors and developers
do what they are reluctant to do in the first place.  Without everyone
updating the documentation as they change code, any documentation
about design and internals will quickly become outdated, and thus not
just useless, but downright harmful.  We have trouble keeping even our
user documentation up-to-date.

Are you aware of _any_ Free Software project that has any useful
documentation of this kind?  Every project I ever knew or was involved
with which tried ended up abandoning that.  A case in point is GDB,
which once had a "GDB Internals" manual -- it was always outdated, and
was eventually scrapped because the maintainers decided they could not
and didn't want to invest the effort.  XEmacs tried in its time to
have the internals documented, but that was basically a one-man
effort, and even in it whole chapters were never written.  Etc. etc.

I think there's some conclusion waiting here.

> That's assuming the maintainers consider such documentation
> important enough for enabling future potential contributors and
> maintainers to hold otherwise useful contributions in limbo.

And you think we don't?  Of course we do!  We also want to release
Emacs much more frequently, and we want it to have no bugs, and a few
other things.  But somehow, each such goal cannot be reached for some
boring practical reasons, none of them related to their importance in
our eyes.

We do in general consider it somewhat more important to develop Emacs
and accept contributions than to document its internals, that's true.
Which is why I said up-thread that without someone who'd volunteer to
do this job (thus dedicating his/her time almost exclusively to it),
this will probably never happen, given the current state of our
resources, which are barely enough to maintain the status quo and move
forward at some reasonable pace.

> >  Most of the design changes and
> > redesigns in Emacs were developed without any bug report, simply
> > because those who did the job knew that a particular group of problems
> > needs to be taken care of.
> 
> It's not like there isn't any discussion or justification of features
> offered prior to code being integrated into the main branch. It's more
> a challenge of how to weave what's there into a coherent account of
> what's going on in the code.

IME, you are wrong in that assumption.  The significant changes in
Emacs design are almost never publicly discussed, not in the way that
would allow someone to glean the design from them.

E.g., take the bidirectional display example.  That one I'm intimately
familiar with, so I can tell you with 100% confidence: the current
design cannot be possibly gleaned from any discussions.  Quite the
contrary: the special emacs-bidi mailing list, created back there for
such discussions, holds many ideas expressed by people, none of them
actually relevant to what we have in Emacs today.  At some point I
realized that I should stop wasting my time on those discussions, and
instead sit down and code this stuff, or the job will never be done.
It is no coincidence that the discussions on emacs-bidi died right
about the time I started implementing.

I did document the main design points in bidi.c, but if you look for
design decisions explained there, you won't find such explanations,
only the description of the design's end point.

And this example is somewhat exceptional, because the amount of
documentation of the design and the implementation there is relatively
large.  In other areas the situation is more bleak.

> It would just be easier to automate some sort of design note
> extraction from the git log if references to mails could be associated
> with relevant features.  I've never used org, but maybe there's some
> syntax that would be useful?  Or maybe some notation from supercite
> for precise pulling of relevant text from list archives?

I wish this were true.  It isn't.  The discussions and the commit log
messages rarely describe the design, and in many cases barely describe
even the particular implementation.  In a project where people with
write access can commit changes without any review, I don't know how
can anything else to be expected.  We basically rely on each
individual to do the job perfectly and contribute to what you want to
see documented.  The results are before our eyes, and they shouldn't
surprise anyone.

> > > https://github.com/rocky/elisp-bytecode
> > > That is really useful documentation that would ideally be in the emacs docs
> > > or etc directory.
> >
> > That's not design description, though.
> 
> You probably have a more nuanced view than me on this.  It's true,
> that document is focused on the specification (the "what") rather than
> the (detailed) "how" and "why" - is that what you mean?

Of course.

> Either way, if you want to understand how the operational semantics
> of emacs lisp work in practice, a document of that sort is
> invaluable.  Without that, a document explaining the "why" isn't
> going to be able to be very concrete.

I agree, but the "what" is usually already available in the comments
to the code, though not everywhere and for every significant feature.
The "why" and "how", by contrast are almost completely missing.

To summarize: I'm not sure we should continue this discussion, because
I don't see where is it going and what could it possibly change in
practice.  I agree with the value of having all of that documented,
I'm just saying it's a large job that needs dedicated individuals, and
I don't see how that could be replaced by any semi-automated means.

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 13547 bytes --]

On Fri, Jun 24, 2022, 2:54 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Lynn Winebarger <owinebar@gmail.com>
> >
> > > > It would help if there were some taxonomy of features/design
> > > > points for
> > > > reference.  Is the bug database being used for that?
> > >
> > > I don't think so (IIUC what you mean by that), and I don't really see
> > > how bugs could serve in that role.
> >
> > I agree, but some of the bug reports are classified as something along
> > the lines of "feature request" or "wishlist".
>
> If you look at those, you will see that almost all of them request
> minor features that have no impact on the design whatsoever.  It is
> very rare to have a feature request on the bug tracker that changes
> the design or introduces new designs.  I would even dare to say it
> never happened, definitely not on my watch.
>



> > I haven't found any other database tracking anything corresponding
> > to features, proposed or otherwise.
>
> We do have etc/TODO.  Did you look there?
>

Yes, but it's not indexed so there's no good way to reference particular
items the way bugs are.


> > Once something is in place, no matter how bare-bones, the key would be
> > maintainers mandating that any substantive introduction or revision of
> > features be tied to appropriate entries for the corresponding
> > design/spec documentation, in the same way and for the same reason
> > coding standards are enforced.
>
> I think you are greatly underestimating the efforts required to
> maintain documentation of this kind, let alone the effort for creating
> it basically from scratch.
>

I just have a lot of sympathy for the points you made in that portable
dumper thread, regarding the cost of accepting substantial contributions
from transitory contributors.
Anyone can create a new feature or improvement and maintain their own
variant, but it's a lot easier for them if the project takes over
maintenance and breaking it becomes a rejection test for future changes.
Everything you say below is sensible.  You and the other (current and
future) core maintainers are the ones who have to make the trade-off, after
all.  I'm going to add some comments below, though, because I think my
perspective on what is useful is different.  If they are obvious or too
idiosyncratic to be useful in the context of attracting new
contributors/maintainers, feel free to ignore them.

>
> Programmers usually don't like writing documentation, and most of them
> cannot write good documentation even if they did want, in large part
> because that take years of practice actually doing it.  In a project
> like Emacs, it is nigh impossible to force contributors and developers
> do what they are reluctant to do in the first place.  Without everyone
> updating the documentation as they change code, any documentation
> about design and internals will quickly become outdated, and thus not
> just useless, but downright harmful.  We have trouble keeping even our
> user documentation up-to-date.
>

Is it harmful?  I suppose it depends on who's relying on it.  Diving into
the source of a project with Emacs's complexity with a particular design
change in mind,  personally I prefer to have some guideposts knowing that
they may be dated, then ask knowledgeable people if there seems to be a
disagreement between the doc and the code.  When that happens on a public
forum like this list, then there's some reference.

It's true, I have a low bar.  It might not seem like it, but my preference
is to explore a question as much as I can on my own before bothering the
experts.  In the best case, the question that stumps me also stumps the
expert - then I know I've done adequate study.   With Emacs (and often
other free software projects), it's just hard to tell what I should be
responsible for studying prior to asking questions.  Beyond the code
itself, of course.


Are you aware of _any_ Free Software project that has any useful
>
documentation of this kind?


I know some "Open Source" projects (eg chromium) that have useful docs.
Also projects with academic roots (Larceny Scheme is still my go-to) tend
to have them.  I'm assuming that in this context, the OSS projects, at
least, are not "Free Software project[s]" in the sense you intended, even
if the software produced by those projects happens to be Free.
I don't do development for a living, so when I first looked into the code
of those corporate projects, the general culture around demanding
uniformity of build systems and supported configurations was quite a
culture shock after being accustomed to the approach of Free Software
projects.  So I'm assuming you intend to refer to projects that have
freedom as an organizing principle, and hence exhibit a self-selection bias
toward diverse and idiosyncratic preferences and goals among their
contributors.  Or something along those lines?

Every project I ever knew or was involved
> with which tried ended up abandoning that.  A case in point is GDB,
> which once had a "GDB Internals" manual -- it was always outdated, and
> was eventually scrapped because the maintainers decided they could not
> and didn't want to invest the effort.


I remember thinking it was a shame.  But, my bar for usefulness is fairly
low.

XEmacs tried in its time to
> have the internals documented, but that was basically a one-man
> effort, and even in it whole chapters were never written.  Etc. etc.
>
> I think there's some conclusion waiting here.
>

I think we have different problem scopes in mind.  I'd take some kind of
indexed wiki as a substantial improvement, particularly if there were links
between tags in the code as well as files and directories cross-referencing
features and VC logs, with the primary archive of discussion happening
there rather than an email list (as with discussion around bugs).  I'd just
like to be able to determine answers to questions like:  why is preventing
the allocation of lisp objects by mmap necessary?  What purpose does
ralloc.c serve?  Do they still serve a purpose?, and know whether I've done
enough self-study to justify asking the experts.

> That's assuming the maintainers consider such documentation
> > important enough for enabling future potential contributors and
> > maintainers to hold otherwise useful contributions in limbo.
>
> And you think we don't?  Of course we do!  We also want to release
> Emacs much more frequently, and we want it to have no bugs, and a few
> other things.  But somehow, each such goal cannot be reached for some
> boring practical reasons, none of them related to their importance in
> our eyes.
>

I did not mean to imply you don't consider them important (I can see it
might read that way).  I meant "important enough [relative to all the
competing priorities]".  I don't know of any other way of expressing
importance except relative to competing priorities subject to
resource constraints.


> We do in general consider it somewhat more important to develop Emacs
> and accept contributions than to document its internals, that's true.
> Which is why I said up-thread that without someone who'd volunteer to
> do this job (thus dedicating his/her time almost exclusively to it),
> this will probably never happen, given the current state of our
> resources, which are barely enough to maintain the status quo and move
> forward at some reasonable pace.
>
> > >  Most of the design changes and
> > > redesigns in Emacs were developed without any bug report, simply
> > > because those who did the job knew that a particular group of problems
> > > needs to be taken care of.
> >
> > It's not like there isn't any discussion or justification of features
> > offered prior to code being integrated into the main branch. It's more
> > a challenge of how to weave what's there into a coherent account of
> > what's going on in the code.
>
> IME, you are wrong in that assumption.  The significant changes in
> Emacs design are almost never publicly discussed, not in the way that
> would allow someone to glean the design from them.
>

Then,  I should understand Daniel Colascione's approach with the portable
dumper preview thread and the more recent one on a revised gc are either
outliers - or that I'm misreading their scope relative to the discussion of
design issues they provide? [ That's a straight question to resolve my
ignorance, not intended to be snarky. ]

> It would just be easier to automate some sort of design note
> > extraction from the git log if references to mails could be associated
> > with relevant features.  I've never used org, but maybe there's some
> > syntax that would be useful?  Or maybe some notation from supercite
> > for precise pulling of relevant text from list archives?
>
> I wish this were true.  It isn't.  The discussions and the commit log
> messages rarely describe the design, and in many cases barely describe
> even the particular implementation.  In a project where people with
> write access can commit changes without any review, I don't know how
> can anything else to be expected.  We basically rely on each
> individual to do the job perfectly and contribute to what you want to
> see documented.  The results are before our eyes, and they shouldn't
> surprise anyone.
>

True enough.


>
> > > > https://github.com/rocky/elisp-bytecode
> > > > That is really useful documentation that would ideally be in the
> emacs docs
> > > > or etc directory.
> > >
> > > That's not design description, though.
> >
> > You probably have a more nuanced view than me on this.  It's true,
> > that document is focused on the specification (the "what") rather than
> > the (detailed) "how" and "why" - is that what you mean?
>
> Of course.
>
> > Either way, if you want to understand how the operational semantics
> > of emacs lisp work in practice, a document of that sort is
> > invaluable.  Without that, a document explaining the "why" isn't
> > going to be able to be very concrete.
>
> I agree, but the "what" is usually already available in the comments
> to the code, though not everywhere and for every significant feature.
> The "why" and "how", by contrast are almost completely missing.
>

In the particular case of the bytecode spec, at least it gives me the sense
of the invariants that are being maintained by the implementation.  Trying
to reverse engineer what those invariants are from a giant C switch
statement is always tricky, because they tend to be expressed with
boilerplate code, and then sometimes you have clever use of the fact that
cases are non-exclusive and "fall through" without an explicit break, etc.
 That's where it intersects with "design" for me.

OTOH, the same switch statement could be written as a dispatch table of
higher-order functions exploiting proper tail-recursion in a way I would
feel comfortable enough with to not feel the need to lean on a spec.  But
that assumes those higher-order functions clearly express the invariants
they impose.  It's a personal preference, I know.

To summarize: I'm not sure we should continue this discussion, because
> I don't see where is it going and what could it possibly change in
> practice.  I agree with the value of having all of that documented,
> I'm just saying it's a large job that needs dedicated individuals, and
> I don't see how that could be replaced by any semi-automated means.
>

I was surprised my off-the-cuff remark about trawling the archives
generated any response in the first place, to be honest.

One thing (I think) some of those OSS IDE projects do well is eat their own
dogfood in terms of features for projects/documentation/collaborative
development models generally.  AFAICT, Emacs has support (either in core or
in non-core packages) for a lot of different tools/approaches/etc to these
issues, but doesn't seem to take advantage of them for emacs development
itself, beyond the integrated bug reporting and version control systems.
There's support for all these project formats and tags - why isn't there a
standard choice of those imposed on the emacs projects itself, e.g. with a
centralized repo of pre-generated tags and auto-generated extraction of
documentation/data structure definitions?  Maybe I just haven't read that
section of the manual or the right document in "etc" yet - I freely admit
my ignorance.  As it is, I've just assumed that the core maintainer(s) have
weighed the trade-offs and decided imposing that kind of constraint, even
among the core maintainers, would cost more than it is worth (in attracting
contributors and maintainers).

If there was an "emacs-devel" mode where the tags (and files would be tags
considered as entries in a directory "document") linked to a database
cross-referenced with the feature database, the VC logs, and a wiki-ish
interface that could record both documentation and "talk".  Sort of like
"help" mode, but with a wikipedia-ish feel.  The contributors to that piece
of code and maintainers covering the related features would be
auto-subscribed to the entry, and any discussion would at least get
recorded in a way linked to the thing being discussed.  I'm not sure
contributors would want to be auto-subscribed like that, but then maybe
their contribution should be reviewed by someone who would be.

It's just an idea - whether it's useful or practical for the purpose of
facilitating the recruitment of new contributors/maintainers, given that
some culture change would likely be required, is not something I have the
perspective to judge.  I do know I would find something like what I've
described very useful.

Lynn

[-- Attachment #2: Type: text/html, Size: 17766 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> Was there?  I see:.  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71
> but it's a little more than a couple of months ago.
> I'm pretty curious because if I accumulate a large buffer of trace output
> (running memory up to 100s of MB), killing the buffer doesn't seem to
> impact gc time substantially.

The GC does not really look at a buffer's content, so unless there is
a non-trivial amount of text-properties (e.g. because of font-lock and
such), the GC will take about the same amount of time handling a large
buffer as handling a tiny buffer.

Similarly, whether the malloc library returns the memory to the system
or not shouldn't affect GC times very much if at all.  What does affect
GC time is the amount of live data as well as the total amount of data
that's under the GC's control (i.e. live data, garbage that still needs
to be collected, as well as data that's free but that we haven't been
able to return to malloc (via `free`)).


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

On Wed, Jun 22, 2022 at 7:26 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Was there?  I see:.
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38345#71
> > but it's a little more than a couple of months ago.
> > I'm pretty curious because if I accumulate a large buffer of trace output
> > (running memory up to 100s of MB), killing the buffer doesn't seem to
> > impact gc time substantially.
>
> The GC does not really look at a buffer's content, so unless there is
> a non-trivial amount of text-properties (e.g. because of font-lock and
> such), the GC will take about the same amount of time handling a large
> buffer as handling a tiny buffer.
>
> Similarly, whether the malloc library returns the memory to the system
> or not shouldn't affect GC times very much if at all.  What does affect
> GC time is the amount of live data as well as the total amount of data
> that's under the GC's control (i.e. live data, garbage that still needs
> to be collected, as well as data that's free but that we haven't been
> able to return to malloc (via `free`)).
>
> That's why I found it so confusing - as though the gc was traversing
malloc's
free list.  Even if the text were killed and saved in the kill ring, it
would be a
lisp string, so the GC shouldn't be traversing it.

I'm not particularly eager to recreate the problem, as I spent (at least)
as
much time trying to understand edebug's behavior as the lisp macro I was
debugging.

It's possible there's something special about the edebug trace buffer, or
(I suspect
more likely) one of those packages I set up made "kill-buffer" bury them
instead.  I
just remember being very frustrated that the gc time seemed to grow
proportional to
the size of the trace buffer and not be improved by running garbage-collect
after
killing that buffer.

Lynn

[-- Attachment #2: Type: text/html, Size: 2505 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Sat, Jun 18, 2022 at 8:52 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> As far as I know, we never actually release back cons space to the OS
> anyway -- even if we could.  (Because glibc's malloc implementation
> doesn't do that unless we call that trim function, which we don't.)  So
> I'm not sure that makes much difference -- especially in a batch process.
>
>
Does this mean that, once the heap size of the process has been increased
by a large gc threshold, every subsequent gc will be proportional to the
largest heap size ever required, regardless of whether some of it could be
permanently freed?  So, if the process terminates before collection, then
it will run faster, but if it keeps running, the gc "pause" time will be
permanently
longer for each subsequent gc?

[-- Attachment #2: Type: text/html, Size: 1292 bytes --]

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I also took a look at related data.  E.g. comparing p=1.0 to the old
> p-0.1 on the process that performs the highest number of GCs (188 cycles
> with p=1.0 and 650 cycles with p=0.1) during an Emacs build.  Here are
> the corresponding last few GC cycles:
>
>     % grep GC-26227 ./+make-0.1.log | tail -n 10              
>     GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
> ...
>     GC-26227 p=0.1 total=145.7M free=32.6M thresold=14.6M
>     GC-26227 p=0.1 total=132.8M free=60.7M thresold=13.3M
>     % grep GC-898 ./+make-1.0.log | tail -n 10              
>     GC-898 p=1.0 total=18.5M free=7.5M thresold=18.5M
> ...
>     GC-898 p=1.0 total=145.7M free=32.6M thresold=145.7M
>     % 

I feel that I am confused about this data.

We have gc-cons-threshold and gc-cons-percentage

According to the gc-cons-percentage docstring:

>> If this portion is smaller than `gc-cons-threshold', this is ignored.

which implies that gc-cons-percentage only takes effect when
heap_size * gc-cons-percentage >= gc-cons-threshold (= 0.8M)

So, heap_size should be >= 80M to take effect at p=1.0
and heap_size should be >= 800M to take effect at p=0.1

yet, you do see an effect at total=18.5M.

What am I missing?

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 19 Jun 2022 15:25:15 +0800
> 
> I feel that I am confused about this data.
> 
> We have gc-cons-threshold and gc-cons-percentage
> 
> According to the gc-cons-percentage docstring:
> 
> >> If this portion is smaller than `gc-cons-threshold', this is ignored.
> 
> which implies that gc-cons-percentage only takes effect when
> heap_size * gc-cons-percentage >= gc-cons-threshold (= 0.8M)
> 
> So, heap_size should be >= 80M to take effect at p=1.0
> and heap_size should be >= 800M to take effect at p=0.1
> 
> yet, you do see an effect at total=18.5M.
> 
> What am I missing?

gc-cons-percentage is measured in units of 1, not in percents.  So the
value of 1.0 means that the condition

   heap_size * gc-cons-percentage >= gc-cons-threshold (= 0.8M)

will become true whenever heap_size is 0.8MB, not 80MB.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> gc-cons-percentage is measured in units of 1, not in percents.  So the
> value of 1.0 means that the condition
>
>    heap_size * gc-cons-percentage >= gc-cons-threshold (= 0.8M)
>
> will become true whenever heap_size is 0.8MB, not 80MB.

Thanks! I was confused by "percentage" in the variable name.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>
>     % grep GC-26227 ./+make-0.1.log | tail -n 10              
>     GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
> ...
>     GC-26227 p=0.1 total=18.8M free=1.8M thresold=1.9M
>     GC-26227 p=0.1 total=145.7M free=32.6M thresold=14.6M
>     GC-26227 p=0.1 total=132.8M free=60.7M thresold=13.3M
>     % grep GC-898 ./+make-1.0.log | tail -n 10              
>     GC-898 p=1.0 total=18.5M free=7.5M thresold=18.5M
> ...
>     GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
>     GC-898 p=1.0 total=145.7M free=32.6M thresold=145.7M
>     % 

Now, when I cleared my confusion about the units of gc-cons-percentage,
let me share one thought on the above results.

In the first series of benchamarks with p=0.1 and total=18.8M, free is
roughly equal to threshold.

When we increase the threshold, free increases up to some value, but
that value is limited by the underlying code:
>     GC-26227 p=0.1 total=145.7M free=32.6M thresold=14.6M
>     GC-898 p=1.0 total=145.7M free=32.6M thresold=145.7M

In the above two cases, it appears that 32.6M is required to store all
the transient values, regardless of the threshold.

Hypothesis:

1. When threshold < equilibrium free, we perform redundant GC passes
2. When threshold > equilibrium free, we allocate extra unnecessary
   memory
3. The optimum memory usage/GC frequency happens at free=threshold

Can we test this hypothesis?

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> When we increase the threshold, free increases up to some value, but
> that value is limited by the underlying code:

A clarification: when you see

    [...]
    GC-898 p=1.0 total=18.7M free=7.3M thresold=18.7M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    [...]

what it means is that after the first of the two GCs, the threshold
until the next GC was set at 18.7M, so when the next GC is called, the
heap has grown to at least 18.7M+18.7M=37.4M (possibly more), then GC
took place, it found 18.8M of live data and of the remaining ≥18.6M it
managed to return ≥11.3M back to the malloc library but it had to keep
7.3M as "free memory" because these are contained in malloc'd blocks
which still also contain *some* live data (typically, we malloc blocks
of say 16kB and then split them into the corresponding number of cons
cells, or floats, or strings, so we can only free a block of floats if
all the floats it contains are dead).

> In the first series of benchamarks with p=0.1 and total=18.8M, free is
> roughly equal to threshold.

This likely means that we basically neither called `malloc` in between
the two GCs (we instead managed to satisfy all the allocations from the
set of free objects we still had around) nor called `free` at the end of
the GC (because those 1.9M of free memory are spread around in such
a way that we can't return any of that memory to the malloc library).

The difference between the two cases is that when you have half of the
heap made up of dead objects, there's a higher probability that some of
those objects are clustered such that they make up a few complete
16kB blocks, whereas when only the dead objects represent only 10% of
the heap, almost all 16kB blocks will contain a mis of dead and live
objects and hence can't be returned to the malloc library.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> When we increase the threshold, free increases up to some value, but
>> that value is limited by the underlying code:
>
> A clarification: when you see
>
>     [...]
>     GC-898 p=1.0 total=18.7M free=7.3M thresold=18.7M
>     GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
>     [...]
> ...

Thanks! I was totally misreading the results.

> The difference between the two cases is that when you have half of the
> heap made up of dead objects, there's a higher probability that some of
> those objects are clustered such that they make up a few complete
> 16kB blocks, whereas when only the dead objects represent only 10% of
> the heap, almost all 16kB blocks will contain a mis of dead and live
> objects and hence can't be returned to the malloc library.

I am again confused.
Does what you say imply that frequent GC with small threshold is not
good because it will have less dead objects and have higher chance of
fragmentation?

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

>> The difference between the two cases is that when you have half of the
>> heap made up of dead objects, there's a higher probability that some of
>> those objects are clustered such that they make up a few complete
>> 16kB blocks, whereas when only the dead objects represent only 10% of
>> the heap, almost all 16kB blocks will contain a mis of dead and live
>> objects and hence can't be returned to the malloc library.
>
> I am again confused.
> Does what you say imply that frequent GC with small threshold is not
> good because it will have less dead objects and have higher chance of
> fragmentation?

Not at all.  The general rule of thumb is: the sooner you recover free
space, the better you can avoid fragmentation.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The difference between the two cases is that when you have half of the
>>> heap made up of dead objects, there's a higher probability that some of
>>> those objects are clustered such that they make up a few complete
>>> 16kB blocks, whereas when only the dead objects represent only 10% of
>>> the heap, almost all 16kB blocks will contain a mis of dead and live
>>> objects and hence can't be returned to the malloc library.
>>
>> I am again confused.
>> Does what you say imply that frequent GC with small threshold is not
>> good because it will have less dead objects and have higher chance of
>> fragmentation?
>
> Not at all.  The general rule of thumb is: the sooner you recover free
> space, the better you can avoid fragmentation.

Sure. But how does the probability to get fragmentation scale with the
GC threshold? Is it linear scaling or is it nonlinear (I suspect that it
is non-linear)? Can someone quantify it? If we do, it would help to
adjust the thresholds more optimally to balance between GC time and
fragmentation.

Best,
Ihor

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Sat, 25 Jun 2022 13:13:22 +0800
> 
> > Not at all.  The general rule of thumb is: the sooner you recover free
> > space, the better you can avoid fragmentation.
> 
> Sure. But how does the probability to get fragmentation scale with the
> GC threshold? Is it linear scaling or is it nonlinear (I suspect that it
> is non-linear)? Can someone quantify it? If we do, it would help to
> adjust the thresholds more optimally to balance between GC time and
> fragmentation.

The answer depends on the C library being used.  If you are talking
about glibc, there's still the question of the statistics of
allocation size distribution, because AFAIK glibc's malloc
implementation uses several separate lists to manage the arena, each
list for allocations between X and Y bytes, where X and Y are
different for each list.  And for allocations larger than 64KB we
instruct glibc to use mmap, where there should be no fragmentation at
all.

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> Sure.  But how does the probability to get fragmentation scale with the
> GC threshold?

No, fragmentation is too complex an effect to be characterized this way.
It heavily depends on the application's allocation patterns as well.


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>  emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 17 Jun 2022 13:48:46 -0400
> 
>     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":
> 
> make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total
> make -j24 FOO=0.1    1338.36s user 60.60s system 1252% cpu 1:51.65 total
> make -j24 FOO=0.1    1339.53s user 60.08s system 1252% cpu 1:51.76 total
> make -j24 FOO=0.2    1266.54s user 59.75s system 1264% cpu 1:44.92 total
> make -j24 FOO=0.2    1269.17s user 60.18s system 1256% cpu 1:45.83 total
> make -j24 FOO=0.2    1266.10s user 60.42s system 1258% cpu 1:45.43 total
> make -j24 FOO=0.3    1200.06s user 60.24s system 1251% cpu 1:40.73 total
> make -j24 FOO=0.3    1198.80s user 60.06s system 1254% cpu 1:40.39 total
> make -j24 FOO=0.3    1202.49s user 59.60s system 1252% cpu 1:40.80 total
> make -j24 FOO=0.4    1134.26s user 59.75s system 1233% cpu 1:36.81 total
> make -j24 FOO=0.4    1133.11s user 60.07s system 1226% cpu 1:37.25 total
> make -j24 FOO=0.4    1132.18s user 59.94s system 1226% cpu 1:37.18 total
> make -j24 FOO=0.5    1069.59s user 59.70s system 1211% cpu 1:33.22 total
> make -j24 FOO=0.5    1070.84s user 60.06s system 1211% cpu 1:33.34 total
> make -j24 FOO=0.5    1070.93s user 59.86s system 1210% cpu 1:33.43 total
> make -j24 FOO=1.0     845.92s user 61.24s system 1126% cpu 1:20.55 total
> make -j24 FOO=1.0     847.92s user 61.82s system 1130% cpu 1:20.46 total
> make -j24 FOO=1.0     846.59s user 61.46s system 1133% cpu 1:20.10 total
> make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total
> make -j24 FOO=2.0     678.04s user 69.19s system 1042% cpu 1:11.64 total
> make -j24 FOO=2.0     677.66s user 68.24s system 1041% cpu 1:11.59 total
> make -j24 FOO=3.0     677.50s user 69.38s system 1047% cpu 1:11.28 total
> make -j24 FOO=3.0     677.42s user 68.43s system 1047% cpu 1:11.19 total
> make -j24 FOO=3.0     677.98s user 69.10s system 1044% cpu 1:11.50 total
> make -j24 FOO=4.0     677.43s user 68.58s system 1051% cpu 1:10.96 total
> make -j24 FOO=4.0     677.09s user 68.49s system 1056% cpu 1:10.57 total
> make -j24 FOO=4.0     677.60s user 69.23s system 1049% cpu 1:11.17 total
> make -j24 FOO=5.0     677.22s user 68.90s system 1043% cpu 1:11.52 total
> make -j24 FOO=5.0     677.47s user 68.91s system 1044% cpu 1:11.49 total
> make -j24 FOO=5.0     677.13s user 67.98s system 1049% cpu 1:10.98 total
> make -j24 FOO=7.0     678.21s user 68.57s system 1044% cpu 1:11.52 total
> make -j24 FOO=7.0     677.48s user 68.50s system 1044% cpu 1:11.42 total
> make -j24 FOO=7.0     677.95s user 68.25s system 1050% cpu 1:11.00 total
> make -j24 FOO=10.0    677.30s user 69.52s system 1045% cpu 1:11.45 total
> make -j24 FOO=10.0    678.18s user 68.55s system 1042% cpu 1:11.62 total
> make -j24 FOO=10.0    678.97s user 68.45s system 1043% cpu 1:11.64 total
> make -j24 FOO=20.0    677.99s user 68.42s system 1041% cpu 1:11.67 total
> make -j24 FOO=20.0    677.69s user 69.31s system 1046% cpu 1:11.36 total
> make -j24 FOO=20.0    677.89s user 68.70s system 1045% cpu 1:11.39 total
> 
> Clearly for this test, the default (0.1) results in too low a threshold
> for best performance and thus too much time spent in the GC.  We reach
> a plateau around percentage=2.0.
> 
> This suggests that for batch jobs maybe we should bump up
> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

It is important to track the memory footprint of the Emacs process in
each run (like, show the maximum VM of all the compilations, for
example), because that's the other side of the tradeoff.

Re: Larger GC thresholds for non-interactive Emacs

[Stefan Monnier]

> It is important to track the memory footprint of the Emacs process in
> each run (like, show the maximum VM of all the compilations, for
> example), because that's the other side of the tradeoff.

I measured the amount of free data in a later measurement, which is not
quite the same but gives some of that info.

Do you happen to know how to get the current process's RSS or VM size
from C so I can try and add this data to my log?


        Stefan

Re: Larger GC thresholds for non-interactive Emacs

[Eli Zaretskii]

On June 18, 2022 4:46:14 PM GMT+03:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote
> 
> Do you happen to know how to get the current process's RSS or VM size
> from C so I can try and add this data to my log?


We do that in process-attributes, so you could just use that.

Re: Larger GC thresholds for non-interactive Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> FYI, I have played with this approach making use of
>> https://gitlab.com/koral/gcmh
>
> But this one focuses on interactive use.  The kind of heuristic I'm
> proposing above would only make sense within a single (long-running)
> command, or in batch mode, I think.

Got it.
But are there actually long-running batch mode commands over there?
The byte-compilation benchmarks in the other replies revealed that the
total number of GCs is not very high over the course of single batch
invocation.

Best,
Ihor

Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs)

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> A possible useful thing Emacs could help with would be a macro dedicated
>> to let-binds like the above. Something like:
>>
>> (with-reduced-gc
>>  ;; Do staff)
>
> Sounds about right, tho I think the name of the macro should be related
> to what the code does rather than to what the author thinks the GC should
> do about it.  Something like `this-code-does-not-generate-garbage`.
>
>>  with-reduced-gc could take care about determining the inner specifics
>>  on what alternative gc-cons-threshold value should be used (possibly
>>  depending on the system memory information).
>
> Sounds good.

This part of the discussion appears to be missed from the following
replies. So, I would like to bring it up again.

To clarify a bit on the usefulness of the proposed macro idea, consider
the following code:

(defvar lst-data)
(benchmark-progn
(let (result)
  (dotimes (i 1000000)
    (push i result))
  (setq lst-data result)
  nil))

This code does not really generate any garbage. Yet GC is triggered
frequently:

Elapsed time: 0.619852s (0.426893s in 11 GCs)

If I instead do

(defvar lst-data)
(benchmark-progn
  (let ((gc-cons-threshold most-positive-fixnum)) ;; the value is just for demo purposes 
(let (result)
  (dotimes (i 1000000)
    (push i result))
  (setq lst-data result)
  nil)))

Elapsed time: 0.216380s (0.031766s in 1 GCs)

The difference is obvious.

More generally, any code that generates/returns large data structures is
going to trigger frequent GCs regardless whether such code generates any
garbage.

On the other hand, we cannot, in general terms, predict if real-life
code, which allocates large permanent data structures, also produces a
lot of actual valid garbage that should be collected.

Having some way to prevent excessive garbage collection that is also
smarter than simply setting gc-cons-threshold to large value would be
useful.

As one idea, a lisp program may mark some of the variables to be skipped
by GC and to not contribute to GC threshold checks (that is, allocating
memory into the marked variables will not increase the memory counter
used by GC).

WDYT?

Best,
Ihor

Re: Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Mattias Engdegård <mattiase@acm.org>,  Eli Zaretskii
>  <eliz@gnu.org>,
>   Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie
>  <acm@muc.de>,  emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 01 Jul 2022 10:34:34 +0800
> 
> (defvar lst-data)
> (benchmark-progn
> (let (result)
>   (dotimes (i 1000000)
>     (push i result))
>   (setq lst-data result)
>   nil))
> 
> This code does not really generate any garbage. Yet GC is triggered
> frequently:
> 
> Elapsed time: 0.619852s (0.426893s in 11 GCs)
> 
> If I instead do
> 
> (defvar lst-data)
> (benchmark-progn
>   (let ((gc-cons-threshold most-positive-fixnum)) ;; the value is just for demo purposes 
> (let (result)
>   (dotimes (i 1000000)
>     (push i result))
>   (setq lst-data result)
>   nil)))
> 
> Elapsed time: 0.216380s (0.031766s in 1 GCs)
> 
> The difference is obvious.

Please don't forget that GC doesn't only collects unused Lisp objects,
it also does other useful memory-management related tasks.  It
compacts buffer text and strings, and it frees unused slots in various
caches (font cache, image cache, etc.).  You can find in the archives
discussions where innocently-looking code could cause Emacs run out of
memory because it used too many fonts without flushing the font cache
(any program that works on the list of fonts returned by the likes of
x-list-fonts is in danger of bumping into that).

> More generally, any code that generates/returns large data structures is
> going to trigger frequent GCs regardless whether such code generates any
> garbage.
> 
> On the other hand, we cannot, in general terms, predict if real-life
> code, which allocates large permanent data structures, also produces a
> lot of actual valid garbage that should be collected.

Yes, that's the conundrum.

> Having some way to prevent excessive garbage collection that is also
> smarter than simply setting gc-cons-threshold to large value would be
> useful.
> 
> As one idea, a lisp program may mark some of the variables to be skipped
> by GC and to not contribute to GC threshold checks (that is, allocating
> memory into the marked variables will not increase the memory counter
> used by GC).
> 
> WDYT?

I'm not sure I understand how this idea can be implemented.  The
counting of how much Lisp data since last GC was produced is done
_before_ variables are bound to the produced data as values.  So by
the time we know the data is bound to such "special" variables, it's
already too late, and the only way to do what you suggest would be to
increase consing_until_gc back after we realize this fact.  Which
would mean computing how much consing was done for the value of these
variables, and that would probably slow down the generation of Lisp
data, wouldn't it?

Or what am I missing?

Re: Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> Please don't forget that GC doesn't only collects unused Lisp objects,
> it also does other useful memory-management related tasks.  It
> compacts buffer text and strings, and it frees unused slots in various
> caches (font cache, image cache, etc.).  You can find in the archives
> discussions where innocently-looking code could cause Emacs run out of
> memory because it used too many fonts without flushing the font cache
> (any program that works on the list of fonts returned by the likes of
> x-list-fonts is in danger of bumping into that).

Then, if we decide to implement the macro I am suggesting, such macro
should not affect memory allocation of such sensitive objects: font
cache, image cache, etc. Just "safe" memory allocations.

>> As one idea, a lisp program may mark some of the variables to be skipped
>> by GC and to not contribute to GC threshold checks (that is, allocating
>> memory into the marked variables will not increase the memory counter
>> used by GC).
>> 
>> WDYT?
>
> I'm not sure I understand how this idea can be implemented.  The
> counting of how much Lisp data since last GC was produced is done
> _before_ variables are bound to the produced data as values.  So by
> the time we know the data is bound to such "special" variables, it's
> already too late, and the only way to do what you suggest would be to
> increase consing_until_gc back after we realize this fact.  Which
> would mean computing how much consing was done for the value of these
> variables, and that would probably slow down the generation of Lisp
> data, wouldn't it?
>
> Or what am I missing?

We can do the following:

1. In addition to directly bumping the TOTAL counter of newly allocated
   memory, we can introduce a new LOCAL counter holding recent
   allocations within current sexp.
2. Every time we return from a sexp/self-quoting object into assignment,
   if we are inside the proposed macro and also assigning value to one
   of the pre-defined symbols, increase the upper LOCAL counter in the
   parent sexp. Otherwise, do not change the upper LOCAL counter.
3. Perform GC according to TOTAL-LOCAL threshold value.
4. When exiting the macro, set LOCAL to 0, unless inside another such
   macro.

Example (I intentionally avoid using dolist because macros using
temporary symbols complicate things):

(defvar lst-value)
(with-no-gc '(i return lst-value)
  (let (return (i 0))
    (while (< i 1000000)
      (setq return (cons i return))
      (make-string 1 ?a)
      (setq i (1+ i)))
    (setq lst-value return)))

Let TOTAL be the global counter and LOCAL be the local counter.

In the above code will:

1. Eval "0" to bind initial value of i. LOCAL=size_of_int;
   GLOBAL+=size_of_int. GC code will not count this allocation when
   deciding whether to perform GC.
2. Return 0 into (i 0) assignment and bump parent LOCAL because i is
   declared by the macro: LOCAL(let)+=LOCAL == size_of_int
3. Eval "1000000" in (< i 1000000); LOCAL=size_of_int; GLOBAL+=size_of_int.
4. Now, we are not inside an assignment, so LOCAL(<) == 0 (unchanged),
   TOTAL is increased, and TOTAL-LOCAL will be bumped by size_of_int.
5. Now, we are inside (while ...) and not inside assignment;
   LOCAL(while)+=LOCAL(<)  == size_of_int
5. Eval (cons i return); LOCAL(cons)=cons_size; GLOBAL+=cons_size;
6. Return the new cons into the assignment to return. Because return is
   declared by the macro, the outer value of local is bumped:
   LOCAL(while) == size_of_int+cons_size
7. Eval (make-string ...). This will allocate a new string;
   LOCAL(make-string)=string_length; GLOBAL+=string_length;
8. Return to (while ...) sexp; Because we are not inside the assignment
   LOCAL(while) is unchanged and the string allocation will contribute
   to the GC threshold.
9. Eval (1+ i), which will allocate a new integer;
   LOCAL(1+)=size_of_int; GLOBAL+=size_of_int;
10. The newly allocated integer is assigned to i symbol, declared by the
   macro; thus LOCAL(while)+=LOCAL(1+) ==
   size_of_int+cons_size+size_of_int; GLOBAL-LOCAL is unchanged and do
   not count towards next GC.
[second iteration]
11. Eval "1000000" in (< i 1000000), which will not allocate any memory
   because integers are immutable (AFAIK). GLOBAL and LOCAL remain
   unchanged.
... (all other iterations)
    LOCAL(while) == size_of_int * 1000000 + cons_size * (1- 1000000)
12. Return from (while ...). It is not an assignment, so
   LOCAL(let)+=LOCAL(while)
13. Assign lst_value, which will not allocate any extra memory.
14. Return from let: LOCAL(with-no-gc)+=LOCAL(let)
15. Return from with-no-gc. LOCAL=0 and the whole allocated object will
    now be able to contribute to the GC threshold, unless the example
   snippet is not by itself wrapped into parent with-no-gc call.
   In any case, while inside with-no-gc, only a single 1000000 symbol +
   all the string allocations can trigger GC. Assignments to i,result,
   and lst_value will only be counted upon exiting the macro.

Best,
Ihor
   

Re: Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  mattiase@acm.org,  theophilusx@gmail.com,
>   rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 01 Jul 2022 15:52:53 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please don't forget that GC doesn't only collects unused Lisp objects,
> > it also does other useful memory-management related tasks.  It
> > compacts buffer text and strings, and it frees unused slots in various
> > caches (font cache, image cache, etc.).  You can find in the archives
> > discussions where innocently-looking code could cause Emacs run out of
> > memory because it used too many fonts without flushing the font cache
> > (any program that works on the list of fonts returned by the likes of
> > x-list-fonts is in danger of bumping into that).
> 
> Then, if we decide to implement the macro I am suggesting, such macro
> should not affect memory allocation of such sensitive objects: font
> cache, image cache, etc.

Why not?  The same could happen with those cached objects.  And doing
these jobs does take CPU time.

> 1. In addition to directly bumping the TOTAL counter of newly allocated
>    memory, we can introduce a new LOCAL counter holding recent
>    allocations within current sexp.
> 2. Every time we return from a sexp/self-quoting object into assignment,
>    if we are inside the proposed macro and also assigning value to one
>    of the pre-defined symbols, increase the upper LOCAL counter in the
>    parent sexp. Otherwise, do not change the upper LOCAL counter.
> 3. Perform GC according to TOTAL-LOCAL threshold value.
> 4. When exiting the macro, set LOCAL to 0, unless inside another such
>    macro.

Is this aligned with what the implementation of the Lisp interpreter
actually does?  I'm not sure we know, where we allocate memory for
Lisp data, whether we are going to bind some variable to the produced
data.  Thus "count recent allocations within the current sexp" sounds
like non-trivial to implement.

And what do you mean by "pre-defined symbols"? what makes a symbol
"pre-defined"?

I also am not sure we have the "parent sexp" in hand, but I'll let
experts on the Lisp interpreter to comment on that.

Re: Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Please don't forget that GC doesn't only collects unused Lisp objects,
>> > it also does other useful memory-management related tasks.  It
>> > compacts buffer text and strings, and it frees unused slots in various
>> > caches (font cache, image cache, etc.).  You can find in the archives
>> > discussions where innocently-looking code could cause Emacs run out of
>> > memory because it used too many fonts without flushing the font cache
>> > (any program that works on the list of fonts returned by the likes of
>> > x-list-fonts is in danger of bumping into that).
>> 
>> Then, if we decide to implement the macro I am suggesting, such macro
>> should not affect memory allocation of such sensitive objects: font
>> cache, image cache, etc.
>
> Why not?  The same could happen with those cached objects.  And doing
> these jobs does take CPU time.

I do not have enough knowledge on the topic to argue on this. What I
wanted to say is that we can make some objects to be not affected by the
suggested macro if there are reasons to believe that it could be too
dangerous.

>> 1. In addition to directly bumping the TOTAL counter of newly allocated
>>    memory, we can introduce a new LOCAL counter holding recent
>>    allocations within current sexp.
>> 2. Every time we return from a sexp/self-quoting object into assignment,
>>    if we are inside the proposed macro and also assigning value to one
>>    of the pre-defined symbols, increase the upper LOCAL counter in the
>>    parent sexp. Otherwise, do not change the upper LOCAL counter.
>> 3. Perform GC according to TOTAL-LOCAL threshold value.
>> 4. When exiting the macro, set LOCAL to 0, unless inside another such
>>    macro.
>
> Is this aligned with what the implementation of the Lisp interpreter
> actually does?  I'm not sure we know, where we allocate memory for
> Lisp data, whether we are going to bind some variable to the produced
> data.  Thus "count recent allocations within the current sexp" sounds
> like non-trivial to implement.

I am not 100% sure, but AFAIK lisp interpreter always knows the current
lisp nesting level. The extra C-level variable I suggest may be an array
of max-lisp-eval-depth size. Then, at every given sexp nesting level, if
we know the current level, we automatically gain access to the
appropriate extra in-sexp memory allocation info.

It is also trivial to ensure that all the deeper nesting levels in the
array are set to 0 (after appropriate update of the value corresponding
to the current nesting)

> And what do you mean by "pre-defined symbols"? what makes a symbol
> "pre-defined"?

I was referring to

(with-no-gc '(pre-defined-symbol1 symbol2 ...) ...)

I am not sure if my description above is 100% accurate, but the general
idea is:
1. Introduce LOCAL variable holding total memory used to allocate
   objects within current lisp nesting. This variable can be represented
   by an array of max-lisp-eval-depth size.
2. GC is triggered based on TOTAL-LOCAL instead of currently used TOTAL.
3. LOCAL[i]+=LOCAL[i+1]; LOCAL[i+1]=0 should be done after anything
   potentially involving recursive sexp call, except symbol slot
   assignments to non-"pre-defined" symbols.
4. If we are assigning symbol value slot and the symbol value is not
   "pre-defined" in the macro, LOCAL is not incremented in 3, which will
   make all the memory allocated inside that sexp to be counted towards
   next GC threshold.

The above effectively allows GCs only upon returning from symbol
assignments (in let/setq/setf/etc). But I believe that it should not be
a big deal.

> I also am not sure we have the "parent sexp" in hand, but I'll let
> experts on the Lisp interpreter to comment on that.

Agree. I am nowhere expert. Mostly tried to look into setq
implementation I come up with something.

Hopefully people more familiar with the interpreter chime in.

Best,
Ihor

Re: Lisp-level macro to avoid excessive GC in memory-allocating code

[Stefan Monnier]

>> Is this aligned with what the implementation of the Lisp interpreter
>> actually does?  I'm not sure we know, where we allocate memory for
>> Lisp data, whether we are going to bind some variable to the produced
>> data.  Thus "count recent allocations within the current sexp" sounds
>> like non-trivial to implement.

That's also my impression.

> I am not 100% sure, but AFAIK lisp interpreter always knows the current
> lisp nesting level.

Not really, no.  There is some related info, but that's a non-trivial
part of the cost of interpretation so any increase to that info will
tend to slow us down.

A simple `with-mostly-allocating-code` macro that increases
`gc-cons-percentage` to 1.0 sounds a lot simpler and won't impose any
extra burden on "normal" code.


        Stefan

Re: Lisp-level macro to avoid excessive GC in memory-allocating code

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  mattiase@acm.org,  theophilusx@gmail.com,
>   rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 01 Jul 2022 09:56:52 -0400
> 
> A simple `with-mostly-allocating-code` macro that increases
> `gc-cons-percentage` to 1.0 sounds a lot simpler and won't impose any
> extra burden on "normal" code.

I agree, but if we are going to introduce such an implementation, I
think we should make our handling of memory-full situation smarter.
Like perhaps keep somewhat larger memory reserve, so that if we hit
the memory limit, we could still call GC and continue, instead of
signaling a fatal error.

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

I think we settled on 1.0 being a promising new default value for
-batch, so I've now pushed a change that makes -batch default to that.
This does change the memory profile (making batch processes use more
memory), so please give feedback on any regressions you see in real
world usage -- if it turns out to be problematic, we'll have to revert
it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Larger GC thresholds for non-interactive Emacs

[Lars Ingebrigtsen]

I've tried doing builds on various machines.  Here's from my M1 Apple
laptop.  Before:

make -j8 bootstrap  717.57s user 64.69s system 482% cpu 2:42.11 total

After:

make -j8 bootstrap  457.93s user 62.46s system 415% cpu 2:05.25 total

Nice reduction in time spent (and impressive lowering of the CPU work
done).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs

[Pankaj Jangid]

Ihor Radchenko <yantar92@gmail.com> writes:

>> This is not needed for the emacs-28 branch,
>> so I meant master.
>
> I am not too familiar with how Emacs handles
> Org updates. AFAIK, Emacs master only tracks
> our stable tagged releases. Not the working
> branches (both bugfix and main). I do not
> know the motivation behind it and I do not
> know if it is a good idea to change it. Do
> you think that Emacs master should instead
> track our Org dev branch?
>

Emacs master is the development branch of
Emacs. I have always wanted Org dev branch to
be part of that. But may be that Org’s dev
branch is not as stable as Emacs’. Hence it is
kept that way.

I have complained about this branch confusion
once on the emacs-devel mailing list. Only to
learn that there is a separate mailing list for
org development related things.

Even for reporting bugs there was a different
mailing list for Org. I am not sure what is the
situation now. I use whichever version comes
with master branch of Emacs. And I report bugs
using ‘M-x report-emacs-bug’ only.

Re: Org mode and Emacs

[Visuwesh]

[வியாழன் ஜூன் 16, 2022 08:49] Pankaj Jangid wrote:

> Even for reporting bugs there was a different
> mailing list for Org. I am not sure what is the
> situation now. I use whichever version comes
> with master branch of Emacs. And I report bugs
> using ‘M-x report-emacs-bug’ only.

The situation is still the same and you're supposed to report org-mode
bugs with M-x org-submit-bug-report.

Re: Org mode and Emacs

[Bastien]

Here is the email I sent when I accepted to switch to using .org as
the native format for Org documentation:

https://list.orgmode.org/orgmode/87371gfas7.fsf@bzg.fr/

As you can read, I wanted to make the switch as an experiment to see
if we were really solving a problem here.

I believe we didn't get more contributions to the manual by switching
to .org, so I'd be in favor of switching back to using .texi as the
native format for Org's documentation.  (Not for 9.6, obviously, more
probably for 10.0 -- I'll discuss this with other Org maintainers.)

The other topics in this thread (make Org's Texinfo exporter provide
good .texi manuals, make Org more modular, etc.) are interesting, but
they are really separate questions IMHO.

-- 
 Bastien

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:45 +0800
> 
> >> It may also help if you try to profile org-make-manuals from
> >> mk/org-fixup.el and share the results.
> >
> > If profiling can help, wouldn't it be simpler to invoke the same
> > commands from an interactive Emacs session, then show the profile?
> 
> This is exactly what I meant. To run org-make-manuals from interactive
> Emacs session.

I've run org-texinfo-export-to-texinfo instead.  Is that OK?

Here's the profile, collected after loading ox-texinfo.el (NOT .elc):

          35  54% - ...
          31  48%    Automatic GC
           4   6%  - minibuffer-complete
           4   6%   - completion-in-region
           4   6%    - completion--in-region
           4   6%     - #<compiled -0x1b56346374fd962>
           4   6%      - apply
           4   6%       - #<compiled 0x1145271b5beccb90>
           4   6%        - completion--in-region-1
           4   6%         - completion--do-completion
           4   6%          - completion-try-completion
           4   6%           - completion--nth-completion
           4   6%            - completion--some
           4   6%             - #<compiled -0x1c897b8c342611de>
           4   6%              - completion-basic-try-completion
           4   6%               - try-completion
           4   6%                - #<compiled -0x1144c96014867ed9>
           4   6%                   complete-with-action
          29  45% - command-execute
          29  45%  - call-interactively
          25  39%   - funcall-interactively
          25  39%    - execute-extended-command
          25  39%     - command-execute
          25  39%      - call-interactively
          25  39%       - funcall-interactively
          25  39%        - org-texinfo-export-to-texinfo
          25  39%         - let
          23  35%          - org-export-to-file
          23  35%           - org-export-as
          10  15%            - org-macro-initialize-templates
          10  15%             - org-macro--collect-macros
           5   7%                org-macro--find-keyword-value
           3   4%              - org-collect-keywords
           3   4%               - org--collect-keywords-1
           1   1%                - org--collect-keywords-1
           1   1%                 - org-element-at-point
           1   1%                  - org-element--parse-to
           1   1%                     org-element--current-element
           2   3%              - org-macro--find-date
           2   3%                 org-macro--find-keyword-value
           6   9%              org-export--delete-comment-trees
           4   6%              org-export-expand-include-keyword
           3   4%            - org-macro-replace-all
           3   4%             - org-macro-expand
           3   4%              - apply
           3   4%               - #<lambda 0xf2f300dd4>
           3   4%                - save-current-buffer
           3   4%                 - set-buffer
           3   4%                  - find-file-noselect
           2   3%                   - find-file-noselect-1
           1   1%                    - after-find-file
           1   1%                     - normal-mode
           1   1%                      - set-auto-mode
           1   1%                       - set-buffer-major-mode
           1   1%                        - fundamental-mode
           1   1%                         - run-mode-hooks
           1   1%                          - hack-local-variables
           1   1%                           - hack-dir-local-variables
           1   1%                            - hack-dir-local--get-variables
           1   1%                             - dir-locals-find-file
           1   1%                              - locate-dominating-file
           1   1%                                 dir-locals--all-files
           1   1%                     find-buffer-visiting
           2   3%            org-export-output-file-name
           4   6%   - byte-code
           4   6%    - read-extended-command
           4   6%     - completing-read
           4   6%      - completing-read-default
           2   3%         read-from-minibuffer

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> This is exactly what I meant. To run org-make-manuals from interactive
>> Emacs session.
>
> I've run org-texinfo-export-to-texinfo instead.  Is that OK?

It is a valid option. org-make-manuals just opens the file and runs
org-texinfo-export-to-texinfo.

> Here's the profile, collected after loading ox-texinfo.el (NOT .elc):
>
>           35  54% - ...
>           31  48%    Automatic GC
>           10  15%             - org-macro--collect-macros
>            5   7%                org-macro--find-keyword-value
>            2   3%              - org-macro--find-date
>            2   3%                 org-macro--find-keyword-value
>            6   9%              org-export--delete-comment-trees
>            4   6%              org-export-expand-include-keyword

The common feature of all the above functions is that they are
structured as:
(goto-char 1) (while (re-search-forward regexp) (do-stuff))

The fact that no-sublevels are revealed by the profiler signifies that
it is regexp search that is likely taking a long time.

Note that I currently experience a significant degradation in
re-search-forward performance on some Org file using Emacs 29 master,
but not Emacs 28 and earlier.

What will happen if you run
(progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
right after exporting. Does it take siginficant amount of time?
What if you run M-: (org-macro--collect-macros) directly in the buffer?
Is it also slow? Is the profiler data more detailed in such case?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:48:05 +0800
> 
> What will happen if you run
> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
> right after exporting. Does it take siginficant amount of time?

It's instantaneous.

> What if you run M-: (org-macro--collect-macros) directly in the buffer?

Also instantaneous.

> Is the profiler data more detailed in such case?

Sorry, I don't understand what you mean here.  What should I do to get
that possibly more detailed profiler data?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> What will happen if you run
>> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
>> right after exporting. Does it take siginficant amount of time?
>
> It's instantaneous.
>
>> What if you run M-: (org-macro--collect-macros) directly in the buffer?
>
> Also instantaneous.

And yet it took 15% of CPU time according to your profile. Something
fishy is going on here. My guess that it is GC.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 17 Jun 2022 14:11:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> What will happen if you run
> >> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
> >> right after exporting. Does it take siginficant amount of time?
> >
> > It's instantaneous.
> >
> >> What if you run M-: (org-macro--collect-macros) directly in the buffer?
> >
> > Also instantaneous.
> 
> And yet it took 15% of CPU time according to your profile. Something
> fishy is going on here. My guess that it is GC.

That'd be strange, because AFAIK the profiler accounts for GC
separately.

Anyway, I think your latest changes close this issue for good.
Thanks!

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2022-06-12 21:00]:
> You are free to open discussion in Org ML about turning back to texi
> format, but it will, for example, make things harder for me and other
> people who are not familiar with texinfo format. I am not sure if build
> time justifies extra maintenance burden.

If Org exports to texi, export that file, and how I understand it
emacs build process was using maybe Org file to convert it to texi.

Why not export texi internally in Org project, then let Emacs build
process use texi file only.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> Date: Sun, 12 Jun 2022 22:25:47 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Eli Zaretskii <eliz@gnu.org>, theophilusx@gmail.com, rms@gnu.org,
>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> 
> Why not export texi internally in Org project, then let Emacs build
> process use texi file only.

Because the real source is in Org, and we in the GNU Project are
always providing the real source files, not the products of their
conversion.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So, now we have our manual written in Org mode and we never had reasons
  > to come back to texi.

I suspect that it doesn't fully follow the markup conventions
for GNU manuals.  That's because we designed Texinfo to have markup commands
to make all the proper semantic distinctions.  If the manual source
is written in a language which doesn't have the full gamut of markup
distinctions, there is no way to do the markup correctly.

It would be useful for someone who understands these conventions
to check the Org manual and see.

I don't think the slowness of processing nowadays is the crucial issue
here.  Computers are much faster now than in the 1990s.  It used
to take a long time for TeX to process the Emacs Lisp Reference Manual.
Now it is perhaps 10 times as fast.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

I am CCing Org ML from now signifying that this branch of the thread is
directly relevent to Org mode and might be of interest for other Org
contributors.

Richard Stallman <rms@gnu.org> writes:

>   > So, now we have our manual written in Org mode and we never had reasons
>   > to come back to texi.
>
> I suspect that it doesn't fully follow the markup conventions
> for GNU manuals.  That's because we designed Texinfo to have markup commands
> to make all the proper semantic distinctions.  If the manual source
> is written in a language which doesn't have the full gamut of markup
> distinctions, there is no way to do the markup correctly.
>
> It would be useful for someone who understands these conventions
> to check the Org manual and see.

Yes, it would certainly help!
The work on manual has been done a long time ago and we also extended
our texinfo exported to suit the manual at that time. Assumingly, just
enough to handle the Org manual use-cases.

Note that we have doc/Documentation_Standards.org explaining some of the
conventions.

Here is a possibly relevant note inside it:

 - Only two of the standard Texinfo indexes are used; those for
   concepts and keys.  This has some implications:

   + The preference is to document commands by key rather than by name

   + Texinfo commands such as @var and @defoption are not used.  The
     preference for this type of thing is that the user browses the
     customize groups.  If you want or need to refer to, say, a
     variable then document it as "the variable
     @code{org-startup-folded}"
 
   + Entries in the concept index are normally all lower case unless
     some other rule dictates otherwise.

Without knowing texinfo, the above paragraphs do not make a whole lot of
sense for me. So, if someone points out any omissions, it would be
helpful for future Org contributors.

> I don't think the slowness of processing nowadays is the crucial issue
> here.  Computers are much faster now than in the 1990s.  It used
> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> Now it is perhaps 10 times as fast.

Generally, there is no way Org export to .info gets any faster than
texinfo. Org is only able to export to other text formats: org->texi;
org->tex; org->html; etc. Convertion to more low-level formats is left
to the external tools like texinfo and pdflatex.

As for reports on the slow performance, they are still useful as long as
they reveal some bottlenecks in Org exporter. Org is being used to
export large documents and whole websites. Hence, improving performance
in this area is generally helpful, even if it is not strictly a critical
blocking issue.

What I want to say is: do not expect Org export to be faster than native
binary tools, but do not hesitate to report performance issues either.

Best,
Ihor

Org syntax compatibility with texinfo syntax (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))

[Ihor Radchenko]

I am forwarding yet another email from the emacs-devel thread to here.
This thread includes both emacs-devel and Org ML.
I will still keep emacs-devel in the loop because people from there can
provide a useful insight about texinfo capabilities.

Richard Stallman wrote:

> I don't know for certain that every possible nesting "does the right
> thing".  I do know that @var{} is used inside many other constructs.
> By contrast, @dfn{} would not be nested inside or around other
> contructs very much.  @key can be nested inside @kbd, and it behaves
> a little differently when nested.

signifying (in addition to previous message), that
1. Texinfo has some software documentation-specific markup elements
2. Those elements can be nested in non-trivial ways, similar to Org's
org-element-object-restrictions, the that nesting can affect formatted
export (again, similar to Org).

With regards to Org syntax, I do not think that we must include all the
possible texinfo elements into Org standard.
@dfn, @key, and @var constructs are extremely specific to software
documentation and have no sense to maintain as a part of general-purpose
markup syntax.

However, what we can do is to allow extending Org syntax to support such
elements. We do support custom syntax elements already (or are
discussing such support):

We have the following, potentially customizeable syntax elements:
- Org links (extensible via org-link-set-parameters)
- Special blocks (de-facto extensible in export backends; also,
  https://github.com/alhassy/org-special-block-extras/);
  https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af
- [FR] Inline special blocks idea being discussed in
  https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net

The links are somewhat limited wrt nesting: links cannot be nested
inside links thus limiting their usefulness as custom markup. Hence, we
may need to look more closely into the idea of inline special blocks
(discuss it in https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net).

Special blocks already support custom :type, but lack convenient Elisp
interface like org-link-set-parameters. (let's keep this part in
https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af)

I'd like to keep discussion on the specifics of customizeable Org syntax
in the relevant threads.

In here, I'd like to hear feedback on possible additional
incompatibilities between texinfo and Org. Is there anything (not
covered by the above list) that is present in texinfo, but impossible in
Org?

For starters, Org does not have a full support for generating index
(apart from ox-texinfo and ox-publish). Though see
https://github.com/tecosaur/org-glossary

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, theophilusx@gmail.com, monnier@iro.umontreal.ca,
>  acm@muc.de, emacs-devel@gnu.org
> Date: Mon, 13 Jun 2022 18:37:04 -0400
> 
> I don't think the slowness of processing nowadays is the crucial issue
> here.  Computers are much faster now than in the 1990s.  It used
> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> Now it is perhaps 10 times as fast.

The Emacs Lisp Reference manual gets processed in about 15 seconds
here.  So 2.5 minutes required to process org.org into org.texi is
quite annoying.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think the slowness of processing nowadays is the crucial issue
>> here.  Computers are much faster now than in the 1990s.  It used
>> to take a long time for TeX to process the Emacs Lisp Reference Manual.
>> Now it is perhaps 10 times as fast.
>
> The Emacs Lisp Reference manual gets processed in about 15 seconds
> here.  So 2.5 minutes required to process org.org into org.texi is
> quite annoying.

To be fair, 2.5 minutes are likely caused by unoptimized Emacs build
because Org has to use the same unoptimized Emacs executable. In
contrast, texinfo is an external tool with release build.

So, while some generic improvements can likely be made on Org side, do
not expect orders of magnitude improvements.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 10:45:02 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I don't think the slowness of processing nowadays is the crucial issue
> >> here.  Computers are much faster now than in the 1990s.  It used
> >> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> >> Now it is perhaps 10 times as fast.
> >
> > The Emacs Lisp Reference manual gets processed in about 15 seconds
> > here.  So 2.5 minutes required to process org.org into org.texi is
> > quite annoying.
> 
> To be fair, 2.5 minutes are likely caused by unoptimized Emacs build
> because Org has to use the same unoptimized Emacs executable.

As an Emacs maintainer, I must have the development version built
without optimizations in at least one branch, because I need to be
able to debug the latest Emacs code at all times.  So for me (and I
presume other frequent contributors to the development) building and
using an unoptimized binary is a must, part of my daily routine.

An optimized build of Emacs produces org.texi in about 30 to 40
seconds here, and that is also quite annoying for a production build.
Lars recently made significant changes in our build scripts and
support code because he didn't like much shorter delays.  We build
Emacs many times a week, so these delays add up.

> So, while some generic improvements can likely be made on Org side, do
> not expect orders of magnitude improvements.

Any improvement will be welcome, thank you.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> An optimized build of Emacs produces org.texi in about 30 to 40
> seconds here, and that is also quite annoying for a production build.
> Lars recently made significant changes in our build scripts and
> support code because he didn't like much shorter delays.  We build
> Emacs many times a week, so these delays add up.

Is there any practical reason to re-build Org manual every time you
rebuild Emacs, even if the source did not change? I though that make
would skip building unchanged sources.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 19:18:57 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > An optimized build of Emacs produces org.texi in about 30 to 40
> > seconds here, and that is also quite annoying for a production build.
> > Lars recently made significant changes in our build scripts and
> > support code because he didn't like much shorter delays.  We build
> > Emacs many times a week, so these delays add up.
> 
> Is there any practical reason to re-build Org manual every time you
> rebuild Emacs, even if the source did not change?

No, and it doesn't happen unless I bootstrap.

> I though that make would skip building unchanged sources.

Indeed, it does (except as part of "make bootstrap", which rebuilds
everything).  But even without a bootstrap, when org.org changes,
building several branches needs to process it once per branch.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > some relating to org, some relating to the fact I'm not convinced there
  > > is a need to replace texinfo, especially given that in some areas, I
  > > think texinfo is more feature rich and consistent and some relating to the feeling I have
  > > that the result is hard to justify given the amount of work it would
  > > require.

Why replace Texinfo?  Because it has some tricky syntax which it
inherits directly from TeX.  And some syntax which is cumbersome,
again imposed directly by TeX.

Nowadays nobody can maintain the TeX-based part which is used to
generate printed manuals.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Werner LEMBERG]

> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.

Well, the thing is backward compatibility – for the sake of the GNU
project.  If Texinfo was allowed to drop that (in particular, the many
unfortunate restrictions of the `.info` file format), everything would
be much easier.

> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.

How do you come to this conclusion?  `texinfo.tex` is actively
maintained, and bug reports (which I submitted a lot) are usually
fixed within days.

Of course, there are some areas where more development would be
desirable (in particular, a better font interface based on ideas from
LaTeX).  On the other hand, the next Texinfo version will contain a
converter from Texinfo source to LaTeX source, which should help
overcome (at least some of) the font problems.

> (In the Info version of the file, the title of the node for @code is
> actually formatted wrong.  It has single quotes around `@code' in
> the title itself.  Texinfo does have some problems that need fixing,
> and it is hard to fix anything in it.)

I strongly suggest that you write to bug-texinfo, discussing this
issue and your other concerns.


    Werner

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Monday, June 13, 2022 at 10:38 AM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Eli Zaretskii" <eliz@gnu.org>
> Cc: theophilusx@gmail.com, yantar92@gmail.com, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > some relating to org, some relating to the fact I'm not convinced there
>   > > is a need to replace texinfo, especially given that in some areas, I
>   > > think texinfo is more feature rich and consistent and some relating to the feeling I have
>   > > that the result is hard to justify given the amount of work it would
>   > > require.
>
> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.
>
> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.
>

Correct.  You got to move on beyond tex no matter the amount of work.

>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> One of Texinfo's crucial features is a wide range of semantic markup
> constructs, each of which can generate different output depending on
> the output format.  For instance, Texinfo has @var, @emph and @dfn,
> all of which generate italics in printed output, but they differ in
> what they generate for other output formats.  There are probably 15
> other such constructs.

Could you please point to the place where I can read about the different
generated output?

> Does Org format have the ability to make all these distinctions?
> If not, I suspect that the Org mode documentation isn't following
> all our style conventions for documentation.

Org manual does not AFAIK, other than various index entries.
However, I am not sure what exactly you are referring to.
Could you please provide a link describing the conventions you are
referring to?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  eliz@gnu.org,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 09:27:57 +0800
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > One of Texinfo's crucial features is a wide range of semantic markup
> > constructs, each of which can generate different output depending on
> > the output format.  For instance, Texinfo has @var, @emph and @dfn,
> > all of which generate italics in printed output, but they differ in
> > what they generate for other output formats.  There are probably 15
> > other such constructs.
> 
> Could you please point to the place where I can read about the different
> generated output?

In the Texinfo manual.  It describes, for each markup, what that
produces in every major output format.

> > Does Org format have the ability to make all these distinctions?
> > If not, I suspect that the Org mode documentation isn't following
> > all our style conventions for documentation.
> 
> Org manual does not AFAIK, other than various index entries.

That is strange to hear.  I was under the impression that Org can
export to various format, and each such output format has its own
export support.  So at least in principle, the distinction Richard is
talking about should be not only possible, but quite easy to
implement, if not already implemented.

> However, I am not sure what exactly you are referring to.
> Could you please provide a link describing the conventions you are
> referring to?

Most of them are in the Texinfo manual, and some are in the GNU Coding
Standards manual.

To take one example, @dfn produces italics in HTML and in printed
formats (PDF etc.), and quoted text in Info.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   For instance, Texinfo has @var, @emph and @dfn,
  > > all of which generate italics in printed output, but they differ in
  > > what they generate for other output formats.  There are probably 15
  > > other such constructs.

  > Could you please point to the place where I can read about the different
  > generated output?

They are in the Texinfo manual.  Look at node "Indicating", which is
section 7.1 in the version I have in Info.

(In the Info version of the file, the title of the node for @code is
actually formatted wrong.  It has single quotes around `@code' in the
title itself.  Texinfo does have some problems that need fixing, and
it is hard to fix anything in it.)

  > One key reason I worry about going down that road is that I suspect it
  > would complicate org's syntax. Two key benefits of org mode is that the
  > basic syntax is simple and it maps reasonably consistently acorss
  > different output formats. However, this flexibility does come at a cost.
  > To provide consistency across export formats, the basic formatting
  > 'concepts' need to be somewhat 'generalised', which means at times you
  > will loose some of the more advanced or sophisticated formatting power
  > of some export back-ends. 

I suspect we are slightly miscommunicating, because Texinfo already
generates several different formats of output, and each markup construct
is carefully defined about how it should appear in each output format.

So I'm sure it is possible to define additional markup constucts and
make each one do, in each output format, what Texinfo would (or does)
do with it.

The only hard part is finding syntax for them.

  > As pointed out elswhere in this thread, org could support missing
  > texinfo syntax using texinfo specific blocks. However, that isn't a
  > great experience from an authoring perspective. It is fine if you only
  > need to do it occasionally, but if you have to constantly add such
  > blocks in order to get really well formatted texinfo output,

This is rather vague.  Why do you thik this would be difficult to use
Can we find a solution to make it easy to use?

We don't have to be limited to whichever escape syntaxes Org already
satisfies.  This use is _important_.  For this, it is worth making new
ones.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

>   > One key reason I worry about going down that road is that I suspect it
>   > would complicate org's syntax. Two key benefits of org mode is that the
>   > basic syntax is simple and it maps reasonably consistently acorss
>   > different output formats. However, this flexibility does come at a cost.
>   > To provide consistency across export formats, the basic formatting
>   > 'concepts' need to be somewhat 'generalised', which means at times you
>   > will loose some of the more advanced or sophisticated formatting power
>   > of some export back-ends. 
>
> I suspect we are slightly miscommunicating, because Texinfo already
> generates several different formats of output, and each markup construct
> is carefully defined about how it should appear in each output format.
>
> So I'm sure it is possible to define additional markup constucts and
> make each one do, in each output format, what Texinfo would (or does)
> do with it.
>
> The only hard part is finding syntax for them.
>

Perhaps, though I tend to feel you may have misconceptions regarding the
purpose of org and its main design goals. 

The org mode syntax, especially the syntax for markup (markdown), is
very small and concise. This is one of its big strengths. It is also one
of the main reasons various markdown dialects have become so popular and
why we have so many different markdown dialects - people don't want a
rich syntax supporting numerous different sematic elements. They just
want a very few. In fact, despite the strong arguments in favour of
semantic markup, people tend to prefer typographical markup - italics,
bold, underline, etc and they want a markup where they don't have to
constantly wonder if they have selected the correct semantic tag for a
certain bit of text. They want something which is easy to write in,
which does not require referring to a manual to lookup different
semantic tags and something which is easier than HTML, LaTeX, TexInfo,
odt, xml, etc.

I also think the statement "The only hard part is finding syntax for
them." is a huge understatement. There are already a couple of requested
syntax changes for markup which IMO have far greater merit than
supporting more texinfo sematnic markup, which have not been implemented
primarily due to the difficulty of ding so in a consistent manner which
will not also break the large amount of existing org data out there. 

You mentioned in another thread that one of the big issues with texinfo
was that maintenance of the tex part, used primarily for the printed
representation, was difficult to maintain these days (assuming lack of
people with expertise in tex). I just wanted to point out that much of
the output formatting org supports is also achieved via tex and latex.
Chances are that if we extended org syntax to support the broader set of
texinfo markup, we would also inherit those same issues. At the very
least, we would not be escaping from tex simply because we moved to org
mode. 

Where I think there is a misunderstanding of org is with the emphasis
which appears to be occurring on the markup and formatting aspects of
org. Org's ability to support a small markdown dialect and generate
output in various formats is not org's main emphasis. Org's primary role
is not that of being a documentation authoring tool. Org is about a lot
more, it is about information gathering and management, task and time
management, information capture, 'executable' notes and literate
programming and lab books with convenient outline editing support. As
the name suggests, it is about helping to organise your life. It isn't
about writing documentation. The fact it can support multiple output
formats and produce reasonable looking documents is certainly part of
it, but in some ways, that was the low hanging fruit which was
relatively easy to achieve once all the other parts were in place.

If a replacement for texinfo is required, then lets take the lessons we
have learned from texinfo, latex/tex, xml, html, markdown and org and
implement something appropriate. This might seem like too much of a
task, but that is probably what people said when Carsten started
developing org mode.

There are many ways org mode can be improved. Current work to clarify
and refine the syntax, provide a solid and efficient parser, improved
font-locking and formatting and more consistent and rich API for both
extensions and new export backends are far more critical than adding
additional markup/markdown syntax to enable org mode to replace texinfo.
IMO such changes are not only misguided, they have the real potential to
adversely impact org by making things more complicated and harder to
maintain, destroying the advantage of a very small and easy to use
markup and moving the focus towards becoming an authoring tool rather
than a personal data organisation aid.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> There are many ways org mode can be improved. Current work to clarify
> and refine the syntax, provide a solid and efficient parser, improved
> font-locking and formatting and more consistent and rich API for both
> extensions and new export backends are far more critical than adding
> additional markup/markdown syntax to enable org mode to replace texinfo.
> IMO such changes are not only misguided, they have the real potential to
> adversely impact org by making things more complicated and harder to
> maintain, destroying the advantage of a very small and easy to use
> markup and moving the focus towards becoming an authoring tool rather
> than a personal data organisation aid.

Tim, I afraid that we a slipping to overgeneralisation again.
We may or may not need to make changes to Org syntax for the purpose of
writing documentation using Org.

Can someone please prepare a small example texi file containing examples
of various texi features that are commonly needed and possibly also
demonstrating some concrete instances of problems that exist in texinfo?

If we have such an example and know the expected outputs, someone can
then create an equivalent Org file. Then, the discussion will hopefuly
become more constructive.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> There are many ways org mode can be improved. Current work to clarify
>> and refine the syntax, provide a solid and efficient parser, improved
>> font-locking and formatting and more consistent and rich API for both
>> extensions and new export backends are far more critical than adding
>> additional markup/markdown syntax to enable org mode to replace texinfo.
>> IMO such changes are not only misguided, they have the real potential to
>> adversely impact org by making things more complicated and harder to
>> maintain, destroying the advantage of a very small and easy to use
>> markup and moving the focus towards becoming an authoring tool rather
>> than a personal data organisation aid.
>
> Tim, I afraid that we a slipping to overgeneralisation again.
> We may or may not need to make changes to Org syntax for the purpose of
> writing documentation using Org.
>
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?
>
> If we have such an example and know the expected outputs, someone can
> then create an equivalent Org file. Then, the discussion will hopefuly
> become more constructive.
>

Well, I think Richard was pretty clear when he asked for the additional
semantic tags listed in the Indicating Definitions, Commands & etc
section of the texinfo manual (6.1 in my copy). Looking at just that
section and it is clear current org markup syntax can handle less than
half. Some of those not handled are the types of semantic elements you
would tend to encounter in software manuals and that is just one section. 

However, I think I've made my position clear and have little more to
add, so I will bow out of any further discussion. 

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> Well, I think Richard was pretty clear when he asked for the additional
> semantic tags listed in the Indicating Definitions, Commands & etc
> section of the texinfo manual (6.1 in my copy). Looking at just that
> section and it is clear current org markup syntax can handle less than
> half. Some of those not handled are the types of semantic elements you
> would tend to encounter in software manuals and that is just one section. 

To clarify, all those extra semantics boil down to either quoting the
text inside in ASCII export or transforming them to /italics/ or
~fixed-width~ in LaTeX/HTML export. Moreover, the majority of extra
semantics is simply aliases (for example, @command is an alias to
@code).

Two exceptions are @acronym and @abbr, which can be easily added as
special link types.

In general, I so nothing in the 6.1 Indicating Definitions, Commands, etc.
that cannot be replicated using Org links or existing emphasis objects:
@code{sample} = ~sample~
@kbd{M-a}     = [[kbd:M-a]]  (with @kbdinputstyle being document keyword)
@key{RET}     = \RET (new entity type)
@samp{text}   = [[samp:text]]
@verb{text}   = =text=
@var{name}    = [[var:name]]
@env{variable}= [[env:variable]]
@file{path}   = [[file:path]]
@command{cmd} = [[command:cmd]]
@option{opt}  = [[option:opt]]
@dfn{def}     = [[dfn:def]]
@abbr{abbrev, meaning} = [[abbr:meaning][abbrev]]
same for @acronym.
@email{address,text} = [[mailto:address][text]]
@indicateurl{url} = [[url]]
@emph{text} = /text/
@strong{text} = *text*
@sc{text} = [[sc:][text]]

Note that for abbrevs, we have WIP
https://github.com/tecosaur/org-glossary

7 Quotations and Examples
@quotation: Block Quotations                    #+begin_verbatim
@indentedblock: Indented text blocks            N/A but can be a special block
@example: Example Text                          #+begin_example
@verbatim: Literal Text                         #+begin_verse
@lisp: Marking a Lisp Example                   #+begin_src
@display: Examples Using the Text Font          N/A (not sure here) but can also be a special block
@format: Examples Using the Full Line Width     I am a bit lost here, but again, anything can be a special block basically
@exdent: Undoing a Line’s Indentation           ...
@flushleft and @flushright                      ...
@raggedright: Ragged Right Text                 ...
@noindent: Omitting Indentation                 \\ I think
@indent: Forcing Indentation                    Not sure here
@cartouche: Rounded Rectangles                  Can be a block option (I am not sure if we can make it universal for all possible backends)
@small… Block Commands                          Can be a block option

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Monday, June 13, 2022 at 12:39 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Tim Cross" <theophilusx@gmail.com>
> Cc: rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Tim Cross <theophilusx@gmail.com> writes:
>
> > There are many ways org mode can be improved. Current work to clarify
> > and refine the syntax, provide a solid and efficient parser, improved
> > font-locking and formatting and more consistent and rich API for both
> > extensions and new export backends are far more critical than adding
> > additional markup/markdown syntax to enable org mode to replace texinfo.
> > IMO such changes are not only misguided, they have the real potential to
> > adversely impact org by making things more complicated and harder to
> > maintain, destroying the advantage of a very small and easy to use
> > markup and moving the focus towards becoming an authoring tool rather
> > than a personal data organisation aid.
>
> Tim, I afraid that we a slipping to overgeneralisation again.
> We may or may not need to make changes to Org syntax for the purpose of
> writing documentation using Org.
>
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?

Does org accept latex structures?  Because texinfo is just plain tex, and as
consequence remained limited (e.g. no colouring, no bold mathematical expressions'
no boxes for formal end of proof).

> If we have such an example and know the expected outputs, someone can
> then create an equivalent Org file. Then, the discussion will hopefuly
> become more constructive.
>
> Best,
> Ihor
>
>

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Christopher Dimech <dimech@gmx.com> writes:

> Does org accept latex structures?  Because texinfo is just plain tex, and as
> consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> no boxes for formal end of proof).

Yes, latex is supported natively. You can mix it into Org text. For
non-latex export, it will be either converted to image of left as-is.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Monday, June 13, 2022 at 2:47 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: "Tim Cross" <theophilusx@gmail.com>, rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Christopher Dimech <dimech@gmx.com> writes:
>
> > Does org accept latex structures?  Because texinfo is just plain tex, and as
> > consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> > no boxes for formal end of proof).
>
> Yes, latex is supported natively. You can mix it into Org text. For
> non-latex export, it will be either converted to image of left as-is.
>
> Best,
> Ihor

That would be very good development, using org for Official Gnu Documentation.
Had discussed with Gavin Smith about latex a couple of years ago, who explained
that texinfo would have to be rewritten completely if we are to have a latex engine
for it.  Lot of work  and unlikely to happen.

Re: Org mode and Emacs

[Werner LEMBERG]

> Had discussed with Gavin Smith about latex a couple of years ago,
> who explained that texinfo would have to be rewritten completely if
> we are to have a latex engine for it.  Lot of work and unlikely to
> happen.

By the way, has someone informed the Texinfo maintainers that people
are whetting knives here on the list, discussing quite fundamental
changes about the future of the project?


    Werner

Re: Org mode and Emacs

[Eli Zaretskii]

On June 13, 2022 8:22:57 AM GMT+03:00, Werner LEMBERG <wl@gnu.org> wrote:
> 
> > Had discussed with Gavin Smith about latex a couple of years ago,
> > who explained that texinfo would have to be rewritten completely if
> > we are to have a latex engine for it.  Lot of work and unlikely to
> > happen.
> 
> By the way, has someone informed the Texinfo maintainers that people
> are whetting knives here on the list, discussing quite fundamental
> changes about the future of the project?

Please don't over-dramatize.  No one is "wetting any knives" here, we are just talking.  If you followed the discussion, supplanting Texinfo is nowhere in sight yet.  This is Emacs, not GCC.

And yes, this discussion should move to the Texinfo list, as has been suggested already.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  eliz@gnu.org,  monnier@iro.umontreal.ca,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Mon, 13 Jun 2022 08:39:42 +0800
> 
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?

Given that you already produced a list of markups, is such an example
still wanted?

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Can someone please prepare a small example texi file containing examples
>> of various texi features that are commonly needed and possibly also
>> demonstrating some concrete instances of problems that exist in texinfo?
>
> Given that you already produced a list of markups, is such an example
> still wanted?

It would certainly help with development. For example, I did not look
close into allowed markup nesting. File with tricky examples will be
helpful as a benchmark for any attempted implementation.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 21:32:33 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can someone please prepare a small example texi file containing examples
> >> of various texi features that are commonly needed and possibly also
> >> demonstrating some concrete instances of problems that exist in texinfo?
> >
> > Given that you already produced a list of markups, is such an example
> > still wanted?
> 
> It would certainly help with development. For example, I did not look
> close into allowed markup nesting. File with tricky examples will be
> helpful as a benchmark for any attempted implementation.

I'm afraid that would take someone who knows more than I do about Org,
because I don't have a clear idea what examples to provide to be
useful for this purpose.  In general the Texinfo manual has examples
of every Texinfo feature, of course.  But it is a large manual.

Specifically wrt nested markups, the most frequent ones I can think
off the top of my head are:

  . @var inside @code
  . @key inside @kbd
  . any in-line markup inside @w
  . @r, which is "escape to regular typeface" and can be used inside
    any markup that produces other typefaces

HTH

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would certainly help with development. For example, I did not look
  > close into allowed markup nesting. File with tricky examples will be
  > helpful as a benchmark for any attempted implementation.

I don't know for certain that every possible nesting "does the right
thing".  I do know that @var{} is used inside many other constructs.
By contrast, @dfn{} would not be nested inside or around other
contructs very much.  @key can be nested inside @kbd, and it behaves
a little differently when nested.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > It would certainly help with development. For example, I did not look
>   > close into allowed markup nesting. File with tricky examples will be
>   > helpful as a benchmark for any attempted implementation.
>
> I don't know for certain that every possible nesting "does the right
> thing".  I do know that @var{} is used inside many other constructs.
> By contrast, @dfn{} would not be nested inside or around other
> contructs very much.  @key can be nested inside @kbd, and it behaves
> a little differently when nested.

Thanks! I replied in a different thread, that includes Org ML in CC.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> They are in the Texinfo manual.  Look at node "Indicating", which is
> section 7.1 in the version I have in Info.

Thanks!

>   > One key reason I worry about going down that road is that I suspect it
>   > would complicate org's syntax. Two key benefits of org mode is that the
>   > basic syntax is simple and it maps reasonably consistently acorss
>   > different output formats. However, this flexibility does come at a cost.
>   > To provide consistency across export formats, the basic formatting
>   > 'concepts' need to be somewhat 'generalised', which means at times you
>   > will loose some of the more advanced or sophisticated formatting power
>   > of some export back-ends. 
>
> I suspect we are slightly miscommunicating, because Texinfo already
> generates several different formats of output, and each markup construct
> is carefully defined about how it should appear in each output format.
>
> So I'm sure it is possible to define additional markup constucts and
> make each one do, in each output format, what Texinfo would (or does)
> do with it.
>
> The only hard part is finding syntax for them.

As I stated in another message, we may not need to define additional
markup constructs. texinfo-specific markup appears to be a little bit
too specific for general-purpose formatting aimed by Org. Instead of
modifying Org syntax (and forcing all third-party packages adapt), we
can leverage the fact that parts of Org syntax are programmable in
Elisp.

Org provides links as a flexible markup element - its export to
different formats can be arbitrarily customized (via function).
See A.3 Adding Hyperlink Types of Org manual.

For paragraph-level constructs, there is
org-export-filter-parse-tree-functions and external
https://github.com/alhassy/org-special-block-extras
We may implement something more reminiscent to link customization in
future. org-export-filter-parse-tree-functions is a bit too generic (one
function for everything).

In summary, Org does not really need to change syntax in order to
support full texinfo markup. AFAIU, the missing parts are
some of the texinfo-specific markup elements, abbreviations, and
generating index. They all can be implemented as new libraries.
Abbreviations and index are WIP at
https://github.com/tecosaur/org-glossary
The texinfo-specific markup can be another (optionally loaded) Org
library.

>   > As pointed out elswhere in this thread, org could support missing
>   > texinfo syntax using texinfo specific blocks. However, that isn't a
>   > great experience from an authoring perspective. It is fine if you only
>   > need to do it occasionally, but if you have to constantly add such
>   > blocks in order to get really well formatted texinfo output,
>
> This is rather vague.  Why do you thik this would be difficult to use
> Can we find a solution to make it easy to use?

This was a reference to Org export blocks:

#+begin_export latex
This text will only appear when exported to \LaTeX, not to any other format.
#+end_export

They may be not great experience what one needs to do something like

#+begin_export texinfo
@var{test} will only appear in .texi output.
#+end_export

#+begin_export latex
\emph{test} will only appear in .tex output.
#+end_export

many times in the same .org document.

However, we have special blocks for this purpose. They are extendable,
as I descibed above. Though some improvements in Org core might be
needed if we have to use this extensibility more frequently.

Best,
Ihor

Re: Org mode and Emacs

[Robert Pluim]

>>>>> On Tue, 14 Jun 2022 21:18:58 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> #+begin_export latex
    Ihor> This text will only appear when exported to \LaTeX, not to any other format.
    Ihor> #+end_export


    Ihor> They may be not great experience what one needs to do something like

    Ihor> #+begin_export texinfo
    Ihor> @var{test} will only appear in .texi output.
    Ihor> #+end_export

    Ihor> #+begin_export latex
    Ihor> \emph{test} will only appear in .tex output.
    Ihor> #+end_export

    Ihor> many times in the same .org document.

    Ihor> However, we have special blocks for this purpose. They are extendable,
    Ihor> as I descibed above. Though some improvements in Org core might be
    Ihor> needed if we have to use this extensibility more frequently.

My first thought here was Org macros, but as far as I can tell they canʼt
be multi-line, which would make them a bit cumbersome to use. Tell us
more about special blocks, the documentation on them is a bit sparse.

Robert
-- 

Cusom special block export, similar org org-link :export parameter (was: Org mode and Emacs)

[Ihor Radchenko]

Robert Pluim <rpluim@gmail.com> writes:

>     Ihor> However, we have special blocks for this purpose. They are extendable,
>     Ihor> as I descibed above. Though some improvements in Org core might be
>     Ihor> needed if we have to use this extensibility more frequently.
>
> My first thought here was Org macros, but as far as I can tell they canʼt
> be multi-line, which would make them a bit cumbersome to use. Tell us
> more about special blocks, the documentation on them is a bit sparse.

Macros are meant to configure per-document level of export. Not global,
usually.

Special blocks are Org blocks with the following syntax:

#+begin_environment [optional parameters]
#+end_environment

With "environment" being any word except "center", "quote", "comment",
"example", "export", "src", and "verse".

Currently, latex export directly translates such blocks into
\begin{environment}
\end{environment}

plain text export supports special handling of
environment=justifyleft, justifyright.

html export supports
'("article" "aside" "audio" "canvas" "details" "figcaption"
    "figure" "footer" "header" "menu" "meter" "nav" "output"
    "progress" "section" "summary" "video")
special blocks

texinfo export translates special blocks into
@environment [options]
@end

odt exprot supports "annotation" and "textbox" blocks.

etc.

Because "environment" can be pretty much anything, we can have a
pre-processor in org-export-filter-parse-tree-functions that transforms
different special blocks into appropriate form for a given export
backend (texinfo, latex, ascii, html, etc). This way, we will not have
to rely on backend-specific handling.

Such pre-processor currently does not exist, but it would be fairly
trivial to implement. We may even introduce an interface similar to
org-link-set-parameters for easier configuration of various types of
special blocks.

WDYT?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They may be not great experience what one needs to do something like

  > #+begin_export texinfo
  > @var{test} will only appear in .texi output.
  > #+end_export

  > #+begin_export latex
  > \emph{test} will only appear in .tex output.
  > #+end_export

  > many times in the same .org document.

  > However, we have special blocks for this purpose. They are extendable,
  > as I descibed above. Though some improvements in Org core might be
  > needed if we have to use this extensibility more frequently.

Maybe that is sufficient.  It would be ok to use, for manuals, a
submode which extend the usual Org facilities.

But it would be good for that submode to define shorter command names
for these.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > However, we have special blocks for this purpose. They are extendable,
>   > as I descibed above. Though some improvements in Org core might be
>   > needed if we have to use this extensibility more frequently.
>
> Maybe that is sufficient.  It would be ok to use, for manuals, a
> submode which extend the usual Org facilities.

Submode is what I am inclined to. All these @var and @dfn constructs
only make sense for software documentation. Org is aiming much more
general writeup.

> But it would be good for that submode to define shorter command names
> for these.

Could you please elaborate? Which parts of the Org syntax you find too
long?

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Making org mode syntax equivalent to texinfo syntax seems like a
  > mistake to me.

If this succeeds, it would be an important advance for the GNU system.
We would replace Texinfo with a much cleaner system, easier to use and
more maintainable.  Not solely for Emacs, but for ALL our
documentation!

                   The original idea was to have a light weight syntax which
  > is easyh to learn, not create a clone of texinfo. Besides, I suspect it
  > would be very difficult to maintain backwards compatibility. 

These are real concerns, but they are not real certainties.  If we look
for solutions, we may find good ones.

I hope someone will give it a try.

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

These constructs make it possible to carry out our documentation style
constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

There is nothing fundamentally hard about supporting these
distinctions.  Can someone please examine which ones Org supports and
which ones not, and propose exensions for the ones not yet supported?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Tim Cross]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Making org mode syntax equivalent to texinfo syntax seems like a
>   > mistake to me.
>
> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!
>

One key reason I worry about going down that road is that I suspect it
would complicate org's syntax. Two key benefits of org mode is that the
basic syntax is simple and it maps reasonably consistently acorss
different output formats. However, this flexibility does come at a cost.
To provide consistency across export formats, the basic formatting
'concepts' need to be somewhat 'generalised', which means at times you
will loose some of the more advanced or sophisticated formatting power
of some export back-ends. 

There are times, primarily where you need specialised or specific
formatting in a particular format, where you need to drop down to that
low level formatting 'language'. This is the org 'escape hatch', which
provides a way to better leverage off the specific capabilities of a
partricular back-end. For exmaple, you can embed latex in your document
which will be added to your formated output when you generate a PDF/PS
or *.tex output file. However, that formatting often won't work when you
use the same source file to generate an HTML or Markdown or ODT version
because there isn't a clear way to translate the latex to that format.

As pointed out elswhere in this thread, org could support missing
texinfo syntax using texinfo specific blocks. However, that isn't a
great experience from an authoring perspective. It is fine if you only
need to do it occasionally, but if you have to constantly add such
blocks in order to get really well formatted texinfo output, it will
become frustrating. Org also supports a powerful custom link format
which could be used to add missing syntax elements, but I'm unsure on
the usability of such an approach once you have a few such cusotm links.

If we want to avoid this situation, the syntax would need to be added to
the basic org syntax. However, that will also require all back-ends
being able to interpret that syntax in some 'sane' manner, which would
likley be considerable work and in some situations, could be extremely
difficult to do consistently with good results. It would also add to the
overall amount of syntax, potentially making it more complex and harder
to learn. 

Org mode is a great mode, especially for general documentation and
information management, todo and time management and to some extent, for
literate programming and/or 'executable' documents, such as lab books or
even some devops type applicaitons. Being able to have a single source
document which can be used to generate 'reasonable' versions in
different formats. You can even spend a bit of effort to customise
things so that you can get some pretty consistent advanced formatting in
some specific export formats. However, I often find when you need the
advanced formatting power of a specific back-end, your often better just
writing the document in that back-end as it tends to take less effort
and results in cleaner source. 

The other side of th coin is the on-going development of texinfo. I have
not written enough texinfo to really understand what the issues are
which drive the desire to replace it with something else, such as org. I
know there are some criticisms regarding output formats, but I also know
there is on-going work to improve that situation. Is the right strategy
to work on org mode so that it can replace texinfo or work on texinfo to
address limitations (or both?)?

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 383 bytes --]

Note that org's export to texinfo is quite good; as an example, I wrote
the "Emacspeak Turning Twenty" article in org-mode and later exported it
to texinfo among other formats. Org's export to TeX,LaTeX and PDF is
also very good; likely superior to its export to Texinfo.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Tim Cross <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 11:39:07 +1000
> 
> The other side of th coin is the on-going development of texinfo. I have
> not written enough texinfo to really understand what the issues are
> which drive the desire to replace it with something else, such as org. I
> know there are some criticisms regarding output formats, but I also know
> there is on-going work to improve that situation. Is the right strategy
> to work on org mode so that it can replace texinfo or work on texinfo to
> address limitations (or both?)?

The main problem with Texinfo that I'm aware of is its reliance on TeX
for the printed output formats (PDF, PS, etc.).  Many restrictions in
Texinfo come from the fact that something is impossible to implement
in TeX.

Another issue that frequently comes up is the HTML format it produces:
it currently doesn't support many features expected from good on-line
documentation, such as multiple frames etc.

A better place to discuss the shortcomings of Texinfo is on the
Texinfo mailing list, where we can talk to the few people who are
actively developing Texinfo and aware of the main issues.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The main problem with Texinfo that I'm aware of is its reliance on TeX
  > for the printed output formats (PDF, PS, etc.).  Many restrictions in
  > Texinfo come from the fact that something is impossible to implement
  > in TeX.

I am not sure which kind of "something" this refers to.  Formatting
capabilities?  Input syntax?

Since we want to take advantage of TeX to make beautiful printed
manuals, we will be limited to TeX's formatting capabilities.  But I think
they are more powerful than anything else.

The limitations of TeX's parser are things we could avoid, if we didn't
have TeX directly read the texinfo source code.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: theophilusx@gmail.com, monnier@iro.umontreal.ca, acm@muc.de,
> 	emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:46 -0400
> 
>   > The main problem with Texinfo that I'm aware of is its reliance on TeX
>   > for the printed output formats (PDF, PS, etc.).  Many restrictions in
>   > Texinfo come from the fact that something is impossible to implement
>   > in TeX.
> 
> I am not sure which kind of "something" this refers to.  Formatting
> capabilities?  Input syntax?

The fact that Texinfo's TeX implementation is via TeX macros, AFAIU.

Re: Org mode and Emacs

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!

Org mode is not easier to use than Texinfo... at least, I didn't figure
out how to use it.

IIRC it's also much slower to create similarly sized .info files from
Org source than it is with makeinfo, and it also requires an entire
manual to be contained in a single file.

Texinfo also comes with the added benefit of not requiring Emacs to edit
or to translate into other formats.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> IIRC it's also much slower to create similarly sized .info files from
> Org source than it is with makeinfo

Could you please quantify much slower?
Do you have concrete example of a .org file which can demonstrate the
problem?

Also, I think that I have to clarify about texinfo export from Org.
Org does not currently generate .info files directly. .org file is first
transformed into .texi and then uses makeinfo to translate the generated
.texi into .info.

If there is an interest in Org being used instead of texinfo, it should
be based on the extensibility of Org rather than just making Org
replicate what texinfo does. However, I am not sure if all parties in
this discussion have a clear idea about concrete advantages of Org
compared with texinfo. I am not faimiliar with texinfo. Others may not
be as faimilar with Org.

I believe that if we want to be serious about using Org for
documentation more widely, we should first identify which particular
aspects of Org are beneficial for documentation and which particular
aspects are available in other documentation generators and must be then
provided by Org (if not yet available).

> ... , and it also requires an entire
> manual to be contained in a single file.

This is not true. Org mode supports #+include: directives to incorporate
multiple smaller files into one larger file.

> Texinfo also comes with the added benefit of not requiring Emacs to edit
> or to translate into other formats.

You can edit Org files outside Emacs. Say, in vim.
The point about exporting to other formats is valid.

This last point also raises a question. Can Elisp interpreter and
libraries be factored out of Emacs to create a way to execute Elisp
programs without installing all the interactive parts of Emacs?

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>> or to translate into other formats.
>
> You can edit Org files outside Emacs. Say, in vim.
> The point about exporting to other formats is valid.
>
> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Isn't Tim Cross(?) working on something like that -- ie. a parser for
the Org language.  Once we have a solid parser, we can build a standard
(set of?) backend(s) for much of Orgmode.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

Org already have a parser. Written in Elisp. Export is built on top of
the parser.

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This last point also raises a question. Can Elisp interpreter and
>>> libraries be factored out of Emacs to create a way to execute Elisp
>>> programs without installing all the interactive parts of Emacs?
>>
>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>> the Org language.  Once we have a solid parser, we can build a standard
>> (set of?) backend(s) for much of Orgmode.
>
> Org already have a parser. Written in Elisp. Export is built on top of
> the parser.

But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
long run, it would be better to define a "parseable" language as the
standard basis for Org.  From that, people can develop (parts of) Org on
other platforms (Vim, Beorg, Orgzly) and test/prove that they are
compatible with version X of the language.  I think Organice was doing
this, but I haven't looked at it deeply.

Oh, but I see your point about "export". By backend, I was assuming a
true parser would generate a standard "internal" language which could be
fed into simpler backends to actually do the work.  The front-end parser
and back-ends could be translated/rebuilt as needed on new platorms
(iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
the idea.

-- 
David Masterson

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>>> This last point also raises a question. Can Elisp interpreter and
>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>> programs without installing all the interactive parts of Emacs?
>>>
>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>> the Org language.  Once we have a solid parser, we can build a standard
>>> (set of?) backend(s) for much of Orgmode.
>>
>> Org already have a parser. Written in Elisp. Export is built on top of
>> the parser.
>
> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
> long run, it would be better to define a "parseable" language as the
> standard basis for Org.  From that, people can develop (parts of) Org on
> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
> compatible with version X of the language.  I think Organice was doing
> this, but I haven't looked at it deeply.
>

This is all part of the aims and process. However, the first step is to
develop a robust elisp based parser. This helps to ensure the org syntax
is consistent and helps identifies ambiguities which need to be fixed as
well as provides a reference implementation which developers can use to
develop parsers in other languages.

> Oh, but I see your point about "export". By backend, I was assuming a
> true parser would generate a standard "internal" language which could be
> fed into simpler backends to actually do the work.  The front-end parser
> and back-ends could be translated/rebuilt as needed on new platorms
> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
> the idea.

Personally, I think it is unlikely we will ever see an org mode for
other platforms which is equivalent to Emacs. However, the work which is
being done to create a clean parser in elisp and use that as the basis
for an API to manipulate org data, generate font-locking and formatting
and provide a clean API for exporters to use will make it much easier
for people to develop external org support at varying levels. 

Those doing the main work, like Ihor, are very aware of the desire to
facilitate external implementations. It is one reason that a fairly
conservative attitude to change is adopted by the project and often, one
consideration is what imapct a change would have on external org
compatible projects. Sometimes, this is challenging as on one level, we
want ot advance and improve org mode, but on the other, we want it to be
as stable as possible to reduce adverse impact on external projects. So
while the main target is and will always be Emacs, an eye is kept on
efforts of other external projects and an effort is made to support them
when possible and within Emacs and FSF principals/guidelines. 

So far, the key contributors have been doing a very good job. 

Re: Org mode and Emacs

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.

> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to

It is not the goal of Emacs to support tyrant devices such as the iPhone
running proprietary operating systems such as iOS.

If people are going to refactor Org mode, I hope they will not do it
with support for iPhones in mind.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>
> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

I am not sure why you only picked up iPhones from the list suggested by
David. The aim is not supporting iPhones. The aim is supporting
other GNU and Free Software projects that are making use of Org, which
implies conservative approach to breaking changes, especially in syntax.

Best,
Ihor

Re: Org mode and Emacs

[Tim Cross]

Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>>> long run, it would be better to define a "parseable" language as the
>>> standard basis for Org.  From that, people can develop (parts of) Org on
>>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>>> compatible with version X of the language.  I think Organice was doing
>>> this, but I haven't looked at it deeply.
>
>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>

> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

There has never been suggestion that what we are doing is to facilitate
development on non-free platforms. However, if you develop good clean
code and clear algorithms which others are able to use as a reference to
understand the syntax and semantics of a system, you cannot control how
they will use that information. In fact, it can be argued that making
such information readily available and accessible is a large part of the
freedoms being promoted by the FSF.

This is completely different from, for example, providing an interface
which can be used by a non-free platform. 

Besides, if org is as bad and difficult to learn as you have indicated
in other posts, nobody will bother implementing it on those platforms
you seem so threatened by anyway!

Re: Org mode and Emacs

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>>> This last point also raises a question. Can Elisp interpreter and
>>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>>> programs without installing all the interactive parts of Emacs?
>>>>
>>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>>> the Org language.  Once we have a solid parser, we can build a standard
>>>> (set of?) backend(s) for much of Orgmode.
>>>
>>> Org already have a parser. Written in Elisp. Export is built on top of
>>> the parser.
>>
>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.
>>
>
> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to
> develop parsers in other languages.

Semantic/Bovine ??

Agree

>> Oh, but I see your point about "export". By backend, I was assuming a
>> true parser would generate a standard "internal" language which could be
>> fed into simpler backends to actually do the work.  The front-end parser
>> and back-ends could be translated/rebuilt as needed on new platorms
>> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
>> the idea.
>
> Personally, I think it is unlikely we will ever see an org mode for
> other platforms which is equivalent to Emacs. However, the work which is
> being done to create a clean parser in elisp and use that as the basis
> for an API to manipulate org data, generate font-locking and formatting
> and provide a clean API for exporters to use will make it much easier
> for people to develop external org support at varying levels.

Agree

> Those doing the main work, like Ihor, are very aware of the desire to
> facilitate external implementations. It is one reason that a fairly
> conservative attitude to change is adopted by the project and often, one
> consideration is what imapct a change would have on external org
> compatible projects. Sometimes, this is challenging as on one level, we
> want ot advance and improve org mode, but on the other, we want it to be
> as stable as possible to reduce adverse impact on external projects. So
> while the main target is and will always be Emacs, an eye is kept on
> efforts of other external projects and an effort is made to support them
> when possible and within Emacs and FSF principals/guidelines. 
>
> So far, the key contributors have been doing a very good job. 

That's good.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>> develop parsers in other languages.
>
> Semantic/Bovine ??

Org is not context-free.
Also, Org maintaners previously rejected the idea of implementing Org
parser not in Elisp. Mainly because it would limit the ability to
maintain and contribute to Org - one would need to learn another
programming language to alter anything in Org syntax.

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This is all part of the aims and process. However, the first step is to
>>> develop a robust elisp based parser. This helps to ensure the org syntax
>>> is consistent and helps identifies ambiguities which need to be fixed as
>>> well as provides a reference implementation which developers can use to
>>> develop parsers in other languages.
>>
>> Semantic/Bovine ??
>
> Org is not context-free.

But could it be moved in that direction? (ie. Organice)

> Also, Org maintaners previously rejected the idea of implementing Org
> parser not in Elisp. Mainly because it would limit the ability to
> maintain and contribute to Org - one would need to learn another
> programming language to alter anything in Org syntax.

Hmmm. That would make it difficult to keep the language "parseable" by a
different parser.  Elisp would not provide the checks for (say) keeping
the language context-free.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> Semantic/Bovine ??
>>
>> Org is not context-free.
>
> But could it be moved in that direction? (ie. Organice)

I don't think so. It is motivated by the fundamental Org syntax design,
AFAIU. (mostly by first match wins design). We are not going to change
fundamentals of the Org syntax. It will break backward compatibility.

>> Also, Org maintaners previously rejected the idea of implementing Org
>> parser not in Elisp. Mainly because it would limit the ability to
>> maintain and contribute to Org - one would need to learn another
>> programming language to alter anything in Org syntax.
>
> Hmmm. That would make it difficult to keep the language "parseable" by a
> different parser.  Elisp would not provide the checks for (say) keeping
> the language context-free.

At this point, we are trying to "freeze" Org syntax as much as possible.
So, major changes are not expected. Different parsers should not suffer
from future changes (if they do, we should not make those changes to
start with).

As for keeping checks, we do have a set of parser tests using ERT. So,
major breakage will be prevented. On top of this, we plan to make the
parser tests more friendly to third-party tools:
https://orgmode.org/list/87fsqzi4tw.fsf@localhost

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> Semantic/Bovine ??
>>>
>>> Org is not context-free.
>>
>> But could it be moved in that direction? (ie. Organice)
>
> I don't think so. It is motivated by the fundamental Org syntax design,
> AFAIU. (mostly by first match wins design). We are not going to change
> fundamentals of the Org syntax. It will break backward compatibility.

Could Org be moved toward a "well-defined" grammar that could be
separated from the Emacs implementation to allow other systems (iOS,
Android, Windows) to implement (at least part of) a "standard" Org?
Could the backward compatibility be covered by an Emacs library where
necessary? 

>>> Also, Org maintaners previously rejected the idea of implementing Org
>>> parser not in Elisp. Mainly because it would limit the ability to
>>> maintain and contribute to Org - one would need to learn another
>>> programming language to alter anything in Org syntax.
>>
>> Hmmm. That would make it difficult to keep the language "parseable" by a
>> different parser.  Elisp would not provide the checks for (say) keeping
>> the language context-free.
>
> At this point, we are trying to "freeze" Org syntax as much as possible.
> So, major changes are not expected. Different parsers should not suffer
> from future changes (if they do, we should not make those changes to
> start with).
>
> As for keeping checks, we do have a set of parser tests using ERT. So,
> major breakage will be prevented. On top of this, we plan to make the
> parser tests more friendly to third-party tools:
> https://orgmode.org/list/87fsqzi4tw.fsf@localhost

This sounds good.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> But could it be moved in that direction? (ie. Organice)
>>
>> I don't think so. It is motivated by the fundamental Org syntax design,
>> AFAIU. (mostly by first match wins design). We are not going to change
>> fundamentals of the Org syntax. It will break backward compatibility.
>
> Could Org be moved toward a "well-defined" grammar that could be
> separated from the Emacs implementation to allow other systems (iOS,
> Android, Windows) to implement (at least part of) a "standard" Org?
> Could the backward compatibility be covered by an Emacs library where
> necessary? 

Yes. See https://orgmode.org/worg/dev/org-syntax.html
I hope it is well-defined enough for you.
I don't think that we need major changes to allow implementation in
other systems. There is already a number of existing third-party parsers
for Org:

https://github.com/200ok-ch/org-parser
https://github.com/tgbugs/laundry
https://github.com/milisims/tree-sitter-org
https://github.com/tecosaur/Org.jl

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> But could it be moved in that direction? (ie. Organice)
>>>
>>> I don't think so. It is motivated by the fundamental Org syntax design,
>>> AFAIU. (mostly by first match wins design). We are not going to change
>>> fundamentals of the Org syntax. It will break backward compatibility.
>>
>> Could Org be moved toward a "well-defined" grammar that could be
>> separated from the Emacs implementation to allow other systems (iOS,
>> Android, Windows) to implement (at least part of) a "standard" Org?
>> Could the backward compatibility be covered by an Emacs library where
>> necessary? 
>
> Yes. See https://orgmode.org/worg/dev/org-syntax.html
> I hope it is well-defined enough for you.
> I don't think that we need major changes to allow implementation in
> other systems. There is already a number of existing third-party parsers
> for Org:
>
> https://github.com/200ok-ch/org-parser
> https://github.com/tgbugs/laundry
> https://github.com/milisims/tree-sitter-org
> https://github.com/tecosaur/Org.jl

Excellent!

-- 
David Masterson

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>>> or to translate into other formats.
>>
>> You can edit Org files outside Emacs. Say, in vim.
>> The point about exporting to other formats is valid.
>>
>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

No. Ihor is the one working on the parser. 

I think Ihor's question about isolating the Elisp interpreter is about
options to make org mode work outside of Emacs. There are frequent
questions to the org list about making org available to other
editors/environments. However, the big problem is that much of the
really powerful featurs of org mode are intrinsically tied to elisp
functionality - for exmaple all the babel and backend export support.
While it is possible to write a parser in any language which would
enable basic org markup/formatting, that does not solve the problem of
executable blocks, babel/noweb and interaction with back end exporters
etc. 

This would be one example of where something like an elisp LSP module
would be useful. However, the idea of an elisp LSP module was
discouraged by a couple of people on this list over concerns that such a
module would enable non-free platforms to take advantage of elisp - a
fear which I think is overstated and which I think one person referred
to as "Jumping at shadows" 

The overall conclusion was that such a module would be considerable
amount of work. Some suggested you could use emacsclient to create
something which could be used in such a manner. However, I suspect that
if you go to the effort of installing and configuring emacs, you would
just use emacs. 

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@gmail.com> writes:

> Could you please quantify much slower?

Making the Org manual takes about 27 seconds on my system:

  GEN      org.texi
  GEN      ../../info/org.info

real	0m27.210s
user	0m25.567s
sys	0m0.783s

Which is longer than it takes for the entire (much more massive) Emacs
Lisp reference manual:

  GEN      ../../info/elisp.info

real	0m24.561s
user	0m23.660s
sys	0m0.515s

And generating the .info file from the Org manual texinfo source takes
only:

  GEN      ../../info/org.info

real	0m5.381s
user	0m5.109s
sys	0m0.141s

So it's clear that the org to texinfo export takes most of the time.

> I believe that if we want to be serious about using Org for
> documentation more widely, we should first identify which particular
> aspects of Org are beneficial for documentation and which particular
> aspects are available in other documentation generators and must be then
> provided by Org (if not yet available).

I don't see why we should be serious about using Org for our
documentation, when most people already know texinfo and are quite happy
with it.

> This is not true. Org mode supports #+include: directives to incorporate
> multiple smaller files into one larger file.

That's nice, but why isn't the Org manual itself structured that way?

> You can edit Org files outside Emacs. Say, in vim.

But does it work very well?

> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Probably not without a lot of work.

Re: Org mode and Emacs

[chad]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Sat, Jun 11, 2022 at 10:52 PM Po Lu <luangruo@yahoo.com> wrote:

> I don't see why we should be serious about using Org for our
> documentation, when most people already know texinfo and are quite happy
> with it.
>

I think a reasonable examination of the emacs-devel archives as well as the
common practices of most of the people publishing emacs lisp packages today
would lead to a very different conclusion. There are several threads about
maintenance concerns around makeinfo/texinfo, and many discussions about
replacing texinfo with, for example, HTML.There are periodic threads where
people claim that they won't try to add their project to GNU because the
burden of learning and using texinfo is too high, although those have died
down in volume since it became more practical to translate other formats to
texinfo.

None of this is to say that Org is the "right" format for Emacs
documentation. I just don't think it's correct to say "most people already
know texinfo and are quite happy with it".

~Chad

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

Re: Org mode and Emacs

[Po Lu]

chad <yandros@gmail.com> writes:

> I think a reasonable examination of the emacs-devel archives as well
> as the common practices of most of the people publishing emacs lisp
> packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo,
> and many discussions about replacing texinfo with, for example,
> HTML.

Published Emacs Lisp packages in the wild typically come with no
documentation at all, aside from doc strings and a single README file.

Org mode happens to be popular for the latter, but I don't know why.
The learning curve is high and the results seem to be unjustified.  An
easier format such as HTML or Markdown can be used instead.

> There are periodic threads where people claim that they won't try to
> add their project to GNU because the burden of learning and using
> texinfo is too high, although those have died down in volume since it
> became more practical to translate other formats to texinfo.

Why would Org mode be any different?  Unlike HTML, it is completely
specific to Emacs, which actually makes it worse than Texinfo, because
Texinfo is at least widely used throughout the entire GNU project.

And anyway, I make the following offer: if someone doesn't want to
contribute documentation (or code) because he doesn't know Texinfo, he
can write the documentation in plain text, and I will translate it to
Texinfo for him.  IIRC the GDB developers made a similar offer, and for
the many years that it existed nobody ever took advantage of it, which
shows that Texinfo is not a serious barrier to contributors.

Further more, using Org mode for documentation will make Emacs lose many
people who are actually writing documentation, right now.  At least I
have no interest in ever learning Org mode, and there seem to be no
volunteers who will translate Texinfo (or plain text) to Org mode for me
if we begin to use it for documentation in the future.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Published Emacs Lisp packages in the wild typically come with no
> documentation at all, aside from doc strings and a single README file.
>
> Org mode happens to be popular for the latter, but I don't know why.
> The learning curve is high and the results seem to be unjustified.  An
> easier format such as HTML or Markdown can be used instead.

Which indicates that others may not agree with your assertion about the
learning curve.
For simple README
https://gitlab.com/publicvoit/orgdown/-/blob/master/doc/Orgdown1-Syntax-Examples.org
is more than enough.

Best,
Ihor

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Published Emacs Lisp packages in the wild typically come with no
  > documentation at all, aside from doc strings and a single README file.

Depending on the quality of the README file, this could be imperfect
but adequate, or it could be lousy.  Please don't accept it as a standard
to judge other things by.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And anyway, I make the following offer: if someone doesn't want to
  > contribute documentation (or code) because he doesn't know Texinfo, he
  > can write the documentation in plain text, and I will translate it to
  > Texinfo for him.

Thank you for making this offer.  I hope that many contributors will
take you up on it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sat, 11 Jun 2022 23:54:39 -0400
> Cc: Ihor Radchenko <yantar92@gmail.com>, Richard Stallman <rms@gnu.org>,
>  Tim Cross <theophilusx@gmail.com>, 
>  Stefan Monnier <monnier@iro.umontreal.ca>, Alan Mackenzie <acm@muc.de>, 
>  EMACS development team <emacs-devel@gnu.org>
> 
>  I don't see why we should be serious about using Org for our
>  documentation, when most people already know texinfo and are quite happy
>  with it.
> 
> I think a reasonable examination of the emacs-devel archives as well as the common practices of most of
> the people publishing emacs lisp packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo, and many discussions about
> replacing texinfo with, for example, HTML.There are periodic threads where people claim that they won't try
> to add their project to GNU because the burden of learning and using texinfo is too high, although those have
> died down in volume since it became more practical to translate other formats to texinfo.

The real issue behind these claims is that developers seldom like
writing good documentation.  So learning to use a sophisticated markup
system such as Texinfo is overkill for them.  On top of that, if the
package is not an Emacs package, there's a problem that Texinfo
editing is not really well supported outside of Emacs, which is
another reason for complaining by people who don't use Emacs for
developing their package.

Good HTML is no easier to write than good Texinfo, but the difference
is that there are tools out there other than Emacs which can be used
to create HTML, especially if the manual is relatively simple (as many
Free Software manuals are).

Bottom line: the complaints are real, but I'm not sure they help in
this discussion, because there's no known alternative to Texinfo for
creating good documentation of the kind that we are used to in Emacs.
GCC folks are switching to Sphinx, but that has its own problems, and
without good support for it in Emacs (and no, rst.el is not enough, as
it doesn't provide enough markup commands, AFAICT) it isn't (yet) a
viable alternative.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Richard Stallman <rms@gnu.org>,  Tim Cross <theophilusx@gmail.com>,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 10:15:59 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > IIRC it's also much slower to create similarly sized .info files from
> > Org source than it is with makeinfo
> 
> Could you please quantify much slower?
> Do you have concrete example of a .org file which can demonstrate the
> problem?

With an unoptimized build of Emacs 29, generation of org.texi from
org.org takes 2.5 minutes here.  An optimized build of Emacs 28 does
that in 30 sec on one system and 1 minute on another.  Generation of
org.info then takes between 3 and 4 seconds.

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Christopher Dimech]

> Sent: Sunday, June 12, 2022 at 12:42 PM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Tim Cross" <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Making org mode syntax equivalent to texinfo syntax seems like a
>   > mistake to me.
>
> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!
>
>                    The original idea was to have a light weight syntax which
>   > is easyh to learn, not create a clone of texinfo. Besides, I suspect it
>   > would be very difficult to maintain backwards compatibility.
>
> These are real concerns, but they are not real certainties.  If we look
> for solutions, we may find good ones.
>
> I hope someone will give it a try.

An important aspect moving forward is to permit the use of latex
rather than tex.

> One of Texinfo's crucial features is a wide range of semantic markup
> constructs, each of which can generate different output depending on
> the output format.  For instance, Texinfo has @var, @emph and @dfn,
> all of which generate italics in printed output, but they differ in
> what they generate for other output formats.  There are probably 15
> other such constructs.
>
> These constructs make it possible to carry out our documentation style
> constructs.
>
> Does Org format have the ability to make all these distinctions?
> If not, I suspect that the Org mode documentation isn't following
> all our style conventions for documentation.
>
> There is nothing fundamentally hard about supporting these
> distinctions.  Can someone please examine which ones Org supports and
> which ones not, and propose exensions for the ones not yet supported?
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?

Adding to Tim's statement, there is currently no plan to redesign Org
_syntax_. What is going on now is a slow process of consolidating our
parser to be more consistent + clarifying the synatax description.
See https://orgmode.org/worg/dev/org-syntax.html

There are still some places in the parser and in the syntax description
that a vague or contradict each other.

As for Texinfo features, Org syntax is extensible enough to cover all of
them. At least, from quick skimming through the texinfo manual TOC.
Org does not have dedicated syntax for all the features in "6.1
Indicating Definitions, Commands, etc.", but they can be represented as
special link types. Also, Org does not provide index generation for all
the export backends, which has nothing to do with syntax. It's just not
implemented in our generic export library. Only dedicated Texinfo
exporter supprots the index generation.

> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.

Org is able to export to TeX and many other formats. However, I do not
know what features are required in the context of manuals.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?

AFAIK, no, Org syntax is not being changed.  It seems like Org syntax is
already much extensible.

>
> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.

Converting all Texinfo to Org would require significant effort, which I
think should be spend on new packages or existing packages like GCC,
Guix, Emacs (and perhaps Hurd to complete the GNU system, LOL).

Also switching to Org would probably make vi (no, I'm don't use it)
users angry.  (No flamewar please.)

I think we should better make texinfo-mode better.

>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).

I am not sure if I follow the argument. Major modes are allowed to
change defaults. For example, special modes often change truncate-lines
value. Org mode is also tweaking defaults (yes, many of them). I do not
see any problem here in general.

If you have some specific cases when Org mode alters Emacs defaults in a
way that bites the user, please give concrete examples. Otherwise, your
criticism is not very constructive.

> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.

This is reasonable. RMS also asked for this years back
https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
Since we cannot start Org from scratch, factoring out individual modules
is taking a lot of time and having the hostile attitudes expressed in
some of the emails in this thread is not exactly encouraging.
If you want Org to be more modular, please help by reporting
inconsistencies or misbehavior to Org ML.

Best,
Ihor

Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)

[Stefan Kangas]

Ihor Radchenko <yantar92@gmail.com> writes:

> Since we cannot start Org from scratch, factoring out individual modules
> is taking a lot of time and having the hostile attitudes expressed in
> some of the emails in this thread is not exactly encouraging.

Thanks for working on modularizing Org mode, it sounds promising.

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-08 21:22:07] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT the problem seen from Emacs, is that Org is large (even for
>> a basic uses, which occasionally leads to high load times) and that it
>> doesn't follow all the usual Emacs conventions, such as
>> overriding/remapping standard key-bindings (the resulting behaviors may
>> sometimes be similar to the standard one, but even when that's the case
>> the redefinition can easily bite the user).
> I am not sure if I follow the argument. Major modes are allowed to
> change defaults. For example, special modes often change truncate-lines
> value. Org mode is also tweaking defaults (yes, many of them). I do not
> see any problem here in general.
> If you have some specific cases when Org mode alters Emacs defaults in a
> way that bites the user, please give concrete examples. Otherwise, your
> criticism is not very constructive.

[ Alan gave one or two concrete examples of things that bit him.  ]

This is not a criticism, just a description of how Org is perceived from
the side of "old-time Emacs users who aren't Org users".
The key in what you wrote above is the "yes, many of them", which means
that even tho those tweaks are minor they sum up to something that
old-time users will almost inevitably bump into.

It's not a problem for Org itself, and there are good reasons for each
one of those tweaks, I'm sure, but it does create opposition to using
Org "in core", such as in etc/NEWS.

> This is reasonable. RMS also asked for this years back
> https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
> Since we cannot start Org from scratch, factoring out individual modules
> is taking a lot of time and having the hostile attitudes expressed in
> some of the emails in this thread is not exactly encouraging.

I know it's a lot of work and I'm glad you (plural) are taking it on,
and I'm not very happy about some of the things that have been said in
that thread, which is why I tried to start this thread.


        Stefan

Re: Org mode and Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If you have some specific cases when Org mode alters Emacs defaults in a
>> way that bites the user, please give concrete examples. Otherwise, your
>> criticism is not very constructive.
>
> [ Alan gave one or two concrete examples of things that bit him.  ]

Alan gave two concrete examples:

>>> I put org.org into a buffer.  I saw lots of lines ending in ..., which is
>>> fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
>>> in a new window.  Brilliant!  Did I mention I hate context dependent
>>> commands?  So I have to type M-x outline-show-subtree, which works.  I
>>> read the screenful of text, then type C-S-<down> to scroll it up.
>>> Instead of my standard scrolling command, I get an error message about
>>> "Not at a clock log".  Org mode has stolen the key sequences
>>> C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

where he
1. Complains about major mode shadowing key binding reserved for major
   modes, according to D.2 Key Binding Conventions section of the Elisp
   manual:
>      • Sequences consisting of ‘C-c’ followed by a control character or a
>     digit are reserved for major modes.

2. C-S-<down>, which is a valid point. However, arrow commands is one of
   the few core concepts used in Org major mode.
   I am not sure if re-binding C-S-<down> is so much of a sin on Org
   mode side. Emacs manual 49.3 Customizing Key Bindings says:

>         Since most modes define their own key bindings, activating a mode
>      might override your custom key bindings.  A small number of keys are
>      reserved for user-defined bindings, and should not be used by modes, so
>      key bindings using those keys are safer in this regard.  The reserved
>      key sequences are those consisting of ‘C-c’ followed by a letter (either
>      upper or lower case), and function keys <F5> through <F9> without
>      modifiers (*note Modifier Keys::).

   As a compromise, Org may provide some kind of read-only mode just to
   navigate the document. The arrow bindings can be disabled then. They
   are for editing.

>>> I see many of these bindings as context dependent, with the name of the
>>> function named after the key sequence, not the functionality - e.g.
>>> org-shiftcontroldown.  That is not the way the rest of Emacs is
>>> constructed.  I hate context dependence (except when it is properly
>>> implemented, e.g. by major modes).

    Only a handful of bindings are context dependent. In particular,
    control/shift/meta-arrow keys, <TAB>, and C-c C-c. They are most
    frequently used when editing and navigating Org buffers.

    It would help if "properly implemented" were defined more precisely
    in the above statement.

> This is not a criticism, just a description of how Org is perceived from
> the side of "old-time Emacs users who aren't Org users".
> The key in what you wrote above is the "yes, many of them", which means
> that even tho those tweaks are minor they sum up to something that
> old-time users will almost inevitably bump into.
>
> It's not a problem for Org itself, and there are good reasons for each
> one of those tweaks, I'm sure, but it does create opposition to using
> Org "in core", such as in etc/NEWS.

I understand. However, such criticism is not helpful. Only concrete
examples can lead to actual Org improvements in this area.

Best,
Ihor

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since we cannot start Org from scratch, factoring out individual modules
  > > is taking a lot of time and having the hostile attitudes expressed in
  > > some of the emails in this thread is not exactly encouraging.

  > I know it's a lot of work and I'm glad you (plural) are taking it on,

I feel the same.  To move some of the features out of Org, so that
they are available in all Emacs modes, will make enhance their
availabiliy to all Emacs users.

Also, it could eventually make Org itself simpler -- and less daunting
to try out and learn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Alan Mackenzie <acm@muc.de> writes:

> I resent you continally repeating this unfounded assertion.  There are
> 784 entries in org.info's key index.  That is the current state.  That is
> the truth.  How can that be "incorrect"?  Even if 750 of the 784 bindings
> somehow "don't come into play", they still exist, and are still
> problematic.  Are you saying that only every tenth entry actually
> signifies an actual binding, or something like that?
>

I will make a last attempt to clarify. 

When this thread started, the discussion was around whether readme.org
files should be converted to or replaced with just plain 'traditional'
readme files - waht would once probably have been called ASCII files,
though now these are utf8 I'm not sure calling them ascii is accurate.
It was then suggested an alternative would be to have the readme.org
files formatted using org-mode formatting - enabling org mode features
for folding, formatting of tables, links, source code examples etc. It
was also stated this would be a read-only buffer. 

The point which has been missed an probably should have been made
clearer is that nobody was talking about a full blown org mode. The
suggestion was to use primarily org mode rendering/presentation
capabilities in a read-only buffer. 

My point was that most of the key bindings and complexity you were
referring to were not relevant in this scenario. Of the 784 key bindings
you refer to, the majority of them would never come into play because
they are bindings which only exist is specialised sub-modes of org mode,
like the agenda, which would not appear in the rendering of a readme.org
file. When you open an org file, you get 236 org related key bindings,
not 7984. All those additional key bindings only get loaded/defined if
you open specialised org modes like the agenda, which does not have a
key binding by default, so your unlikely to accidentally initiate that
mode. 

So going back to my original post, which was simply saying that
implying you would get 784 additional key bindings just because you open
an org file was misleading because you simply don't get that many
additional key bindings when you open an org file - in fact, you get
only about 1/3 of that many. 

Re: Convert README.org to plain text README while installing package

[chad]

[-- Attachment #1: Type: text/plain, Size: 3239 bytes --]

On Tue, Jun 7, 2022 at 7:04 PM Tim Cross <theophilusx@gmail.com> wrote:

> So going back to my original post, which was simply saying that
> implying you would get 784 additional key bindings just because you open
> an org file was misleading because you simply don't get that many
> additional key bindings when you open an org file - in fact, you get
> only about 1/3 of that many.
>

I'll add to this statement: the vast majority of those keybindings aren't
relevant when viewing a read-only org file, which is what we're talking
about here. In fact, I used Org (very lightly) for a few years with exactly
these two extra keybindings: Control-Return and Meta-Return. I also used
Tab to expand/compact headings, so perhaps three, depending on your usage
(I was coming from outline-mode, so this wasn't new.) This is a read/write
usage, not a read-only usage.

Everything else is basically the same. There are a few places where Org
changes the meaning of keys to accommodate Org features, but most of these
are DWIM-ish adaptations, or they're guarded by an Org warning that Org
uses the keys for something else (for example, shift-selection). A few
years in, I discovered that Org would automatically reflow the plain-text
tables I sometimes used, and I learned another use for the Tab key; as near
as I can tell, this feature had been present for months if not years
without ever bothering me.

These days, most of my work involves creative writing and light number
crunching, not coding; as part of that, I use dozens of text files over
several active topics. I keep these in a simple hierarchy of plain Org
files. These load in emacs instantaneously. I search through them using
standard text-searching tools (mostly find and grep, sometimes trying out
things like fd, ack, ripgrep, and deadgrep). I once started reading the Org
manual (several years ago) and bounced off pretty quickly; I haven't been
back in years (although I hear that the manual has improved quite a lot
since then, and it's on my overly-long todo list for "someday".)

I say all this not to try to convince anyone to try Org; in fact, if you're
not already using outline-mode a bunch, then you're probably only
interested in Org if there's a specific use that you'd like for emacs
(agenda, note-keeping, literate programming, etc). That said, there are
people, especially elisp package authors, who use it and like it and will
continue to use it to write package documentation. In the (few) packages
that I personally load, most have a README.org, a few have a README.md, and
none have a plain README.

Put another way, this isn't about encouraging authors to use Org or not;
the authors have chosen already. For people who don't really want to try to
"learn Org": I hear and understand. I've been there myself; time is usually
rarer than interesting ideas to explore. I'm writing this in the hope that
it encourages people to not be afraid of looking at .org files even if you
don't have the time, interest, or inclination to "learn Org", because while
the ocean is indeed (seemingly?) quite deep, the shallow end is quite
comfortable. In the meantime, we can all hope that the arrival of the
"modular Org" is sooner rather than later.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 3819 bytes --]

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Tim.

On Mon, Jun 06, 2022 at 23:57:55 +1000, Tim Cross wrote:

> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Tim.

> > On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:

[ .... ]

> >> I also find arguments based around org being complex and difficult
> >> to learn to be somewhat overstated.

> > There are 784 key bindings in org mode.  How can you say that this
> > isn't complex and difficult to learn?

> Very simply because the vast majority of  those key bindings only come
> into play when yhou use advanced features of org mode, such as the
> agenda or table editing or noweb mode. If your just using basic org
> mode features, very very few of those key bindings even come into play.
> Just go throgh that list and remove any bindings relating to agenda
> mode, table editing mode, source block modes, clock table mode, list
> editing mode, etc and you end up with very few key bindings (all of
> which are udner the C-c prefix, unlike your previous claim). 

> >> Org is powerful and very configurable, ....

> > That contradicts your previous statement to some extent.  Any program
> > which is very configurable _needs_ to be configured.

> Absolute nonsense. Just because you can configure somethig doesn't
> automatically mean you have to configure it. Emacs is extremely
> configurable, but you can use it just fine without doing any
> configuration at all. 

I find emacs -Q to be all but unusable, except for testing things with.
Only with my configurations does it become (for me) usable.  I expect the
same to hold for other users.

> >> .... which means there can be a lot to learn if you want to leverage
> >> off all it has to offer.

> > I don't want to do this.  I want to avoid org mode being loaded into my
> > Emacs; it fouls up my key bindings to a significant extent.  Particularly
> > if I just want to read a 50 line README.

> Other posts have already explained this is NOT what is being proposed
> and that it would not affect your key bindings in the slightest. All
> that is being proposed here is that a read only buffer will use org mode
> to format/display the content. Very simple and not the monster you
> insist. 

"Very simple" has a habit of becoming increasingly complicated over time.

> >> However, like emacs, the basics are very simple and easy to learn. 

> > I don't agree that the basics of Emacs are simple and easy to learn at
> > all.  That's a strong reason why other editors have become popular.  Why
> > should I have to spend time learning a super-complicated mode just to
> > read a 50 line README?

> Well, basically, you don't. That has already bene explained and does not
> need to be reitterated here. However, to be clear, all that is being
> proposed is that org formatting is applied to the contents of this
> read-only buffer. There will not be huge numbers of
> unwanted/unexpected key bindings .....

For how long?  Politically, org mode is not part of the Emacs core - it
is an add on application for those that want it.  If the maintainers of
org mode wanted to push it into the Emacs core, then they would proceed,
firstly, by getting a foot in the door, then steadily increasing pressure
to add more and more until org mode was in the position, say, that Info
currently is.  This must not happen (for reasons discussed below).

> .... and there won't be a huge number of other changes. You will likely
> have some additional key bindings which may make navigating larger
> README files easier and that is about it. All that is really being
> proposed here is org font locking and outline mode navigation. All very
> simple really. 

Hmm.

> >> While I'm not arguing that org should be forced upon everyone ....

> > If a README file is README.org, then org mode is forced upon anybody
> > wishing to read it in Emacs.  This is why README files should be plain
> > text.

> >> .... and I would agree we need to keep potential load time issues in
> >> mind, there seems to be lots in this thread over stating the issues
> >> and jumping to extremes.

> > I think org mode is an extreme, with its 784 key bindings which seem
> > to violate written and unwritten conventions.

> That 750 key bindings is an extreme over statement and not what is
> being proposed. Yet again, people jumping to extremes based on
> ignorance and paranoia. Spend the time to go through the key bindings
> listed in the org manual and you will soon realise that the majority of
> those bindings only come into effect in specialised modes - none of
> which are relevant to a READM.org ile (for example, all the key
> bindings relating to agenda mode). 

I see many of these bindings as context dependent, with the name of the
function named after the key sequence, not the functionality - e.g.
org-shiftcontroldown.  That is not the way the rest of Emacs is
constructed.  I hate context dependence (except when it is properly
implemented, e.g. by major modes).

> I find it odd that someone can on one hand argue so hard against using a
> mode based on the complexity and vast nubmer of key bindings and at the
> same time admit they have never spent the time to learn the mode.

There are only so many hours in the day.  I do not "admit" not spending
the time learning it, since there is nothing shameful about this.  I
state it as a fact.  I do not want to spend/waste time on it, and it is
my fear I will be forced into it by a stealthy increase in org mode's
prevelance, if I do not protest.

> This seems like an arguument based on fear and uncertainty about
> something you don't know rather thaa one based on fact.

I put org.org into a buffer.  I saw lots of lines ending in ..., which is
fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
in a new window.  Brilliant!  Did I mention I hate context dependent
commands?  So I have to type M-x outline-show-subtree, which works.  I
read the screenful of text, then type C-S-<down> to scroll it up.
Instead of my standard scrolling command, I get an error message about
"Not at a clock log".  Org mode has stolen the key sequences
C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

And so on.  Without some serious configuration, org mode is for me
unusable.  Moving commands around key sequences would be a nightmare,
given the 784 commands (or even the 230 counted by Eli).  And moving
org-shiftcontroldown onto a different key sequence, assuming I could find
one, would make the function name meaningless.

> Perhaps look more closely at what was actually being proposed (which
> was not a full org mode situation) and spend enough time to at least
> understand the basics before condemnation. 

I see what looks like an attempt to force org mode into the Emacs core by
stealth, a bit at a time.  Already you are talking about adding
functionality for folding.

> >> All that seems to really be under consideration is to enable rendering
> >> of *org files in help buffers using org font locking and perhaps
> >> enabling folding, which could be very beneficial for large readme files
> >> and would not matter for small ones. I also suspect this is something
> >> which could be disabled with a simple variable setting for those who
> >> really don't like it. 

And then after adding folding (by some mysterious key sequence), some
bright spark will want to add something else "for large readme files".
The consequence will be large READMEs which cannot be conveniently read
without this new feature.  And so it goes on.

We already have a format in Emacs for big documentation, Info.  Info has
about 40 commands, not 230 or 784.

> > "It" could best be avoided by having plain text README files.

I think we should strongly encourage package writers to include plain
text READMEs in their packages.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Protesilaos Stavrou]

Hello Alan, Tim, everyone,

> From: Alan Mackenzie <acm@muc.de>
> Date: Mon, 06 Jun 2022 19:19:41 +0000
>
>> > "It" could best be avoided by having plain text README files.
>
> I think we should strongly encourage package writers to include plain
> text READMEs in their packages.

As the author of several packages on GNU ELPA which use a README.org:

1. I agree with Alan's main point: we do not need Org in Help buffers.
   Neither directly nor indirectly (if we are having that discussion).

2. I write all my documentation using the .org format because it can be
   exported to HTML and Info, among others.

3. Org is efficient for large files as it can, for example, create
   persistent links/references between headings.  Those are preserved in
   .info and .html exports.  Org also has nice features, such as the
   ability to declare a buffer-local macro which expands into, say,
   "version1.2.3" or "exported on DATE" where DATE is dynamic.  There
   are some cases where such functionality is helpful.

Concerning point 2, the GNU ELPA machinery converts the Org source into
an Info file.  (Those who install the package can access it with C-h i.)
While I handle the export process into HTML.

This arrangement provides two viable options for users who (i) want to
read the documentation yet (ii) do not wish to load org.el and its
accoutrements.  A third option, in the form of a plain README, is a
useful addition, provided it is done automatically by GNU ELPA.

As far as I am concerned, the README.org is a fully fledged resource,
not a generic, "here are the absolute basics" kind of file (not that
there is anything wrong with those).  As such, writing a separate
README, instead of exporting README.org into plain text, would add to
the maintenance burden---I would rather avoid that.

All the best,
Protesilaos (or simply "Prot")

-- 
Protesilaos Stavrou
https://protesilaos.com

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Hi Prot,

Protesilaos Stavrou <info@protesilaos.com> writes:

> As far as I am concerned, the README.org is a fully fledged resource,
> not a generic, "here are the absolute basics" kind of file (not that
> there is anything wrong with those).  As such, writing a separate
> README, instead of exporting README.org into plain text, would add to
> the maintenance burden---I would rather avoid that.

Most packages already come with a commentary section.  While these
sometimes just say "read the readme" and others contain unnecessary
information like how to install the package, many are well written and
give the appropriate, initial pointers for getting started with a new
package (basic configuration recommendations, what the main entry points
are, what options to look out for).

As mentioned before in this thread, I don't think C-h P should give the
same information as the manual -- be in in org-mode or rendered as
ASCII/Unicode.  This is too much information for someone who wants to
just wants to know what does a package "foobar" do (this is especially
critical when package names are not self explanatory/generic, as was
discussed a few weeks back).

So it might make sense to instruct the ELPA build system to not use
README files, and instead just rely on the commentary section of the
main file.  If you check out the elpaa--get-README function
elpa-admin.el, you'll already see that "README.md" files are commented
out, because

        ;; Most README.md files seem to be
        ;; currently worse than the Commentary:
        ;; section :-(

which is indicative of a general problem, of how README files are used
since GitHub (I have commented previously on the issue in this thread
https://news.ycombinator.com/item?id=30336703).

Once again I would recommend publishing guidelines and recommendations
on good packaging practices on elpa.gnu.org, with tips and examples on
what a good commentary section looks like.

Re: Convert README.org to plain text README while installing package

[Protesilaos Stavrou]

Hello Philip,

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Tue, 07 Jun 2022 12:07:21 +0000
>
>> As far as I am concerned, the README.org is a fully fledged resource,
>> not a generic, "here are the absolute basics" kind of file (not that
>> there is anything wrong with those).  As such, writing a separate
>> README, instead of exporting README.org into plain text, would add to
>> the maintenance burden---I would rather avoid that.
>
> Most packages already come with a commentary section.  While these
> sometimes just say "read the readme" and others contain unnecessary
> information like how to install the package, many are well written and
> give the appropriate, initial pointers for getting started with a new
> package (basic configuration recommendations, what the main entry points
> are, what options to look out for).
>
> As mentioned before in this thread, I don't think C-h P should give the
> same information as the manual -- be in in org-mode or rendered as
> ASCII/Unicode.  This is too much information for someone who wants to
> just wants to know what does a package "foobar" do (this is especially
> critical when package names are not self explanatory/generic, as was
> discussed a few weeks back).
>
> [... 14 lines elided]
>
> Once again I would recommend publishing guidelines and recommendations
> on good packaging practices on elpa.gnu.org, with tips and examples on
> what a good commentary section looks like.

Agreed!

I do use the Commentary for this purpose and am happy to conform with
such guidelines.

-- 
Protesilaos Stavrou
https://protesilaos.com

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Concerning point 2, the GNU ELPA machinery converts the Org source into
> an Info file.  (Those who install the package can access it with C-h i.)
> While I handle the export process into HTML.

Side note: elpa.gnu.org also converts the docs to HTML (both for Org
and for Texinfo docs), and puts a link to it under "Manual" as can be
seen on http://elpa.gnu.org/packages/modus-themes.html

[ This HTML isn't ideal, admittedly.  Help welcome.  ]

> This arrangement provides two viable options for users who (i) want to
> read the documentation yet (ii) do not wish to load org.el and its
> accoutrements.  A third option, in the form of a plain README, is a
> useful addition, provided it is done automatically by GNU ELPA.

I have the impression that there's a bit of misunderstanding here.  There
are two pieces of "docs" that elpa.gnu.org handles: the `:readme` and
the `:doc`.  One is meant for an actual manual that can be browsed with Info,
while the other is expected to be a short blurb, to help prospective
users understand what the package is about.
Both accept the Org format, but the `:doc` does not accept "plain text"
(nor Markdown).

Many packages currently use the same file for both (typically called
`README.org`), which tends to be suboptimal (it's often neither a great
`:readme` nor a great `:doc`), but works fine.


        Stefan

Re: Convert README.org to plain text README while installing package

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

> That 750 key bindings is an extreme over statement and not what is being
> proposed. Yet again, people jumping to extremes based on ignorance and
> paranoia. Spend the time to go through the key bindings listed in the
> org manual and you will soon realise that the majority of those bindings
> only come into effect in specialised mdoes - none of which are relevant
> to a READM.org ile (for example, all the key bindings relating to agenda
> mode). 
>
> I find it odd that someone can on one hand argue so hard against using a
> mode based on the complexity and vast nubmer of key bindings and at the
> same time admit they have never spent the time to learn the mode. This
> seems like an arguument based on fear and uncertainty about something
> you don't know rather thaa one based on fact. Perhaps look more closely
> at what was actually being proposed (which was not a full org mode
> situation) and spend enough time to at least udnerstand the basics
> before condemnation. 

What's more odd is expecting someone to go through the entire Org manual
in order to write and edit README files.

Re: Convert README.org to plain text README while installing package

[Kévin Le Gouguec]

Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> That 750 key bindings is an extreme over statement and not what is being
>> proposed. Yet again, people jumping to extremes based on ignorance and
>> paranoia. Spend the time to go through the key bindings listed in the
>> org manual and you will soon realise that the majority of those bindings
>> only come into effect in specialised mdoes - none of which are relevant
>> to a READM.org ile (for example, all the key bindings relating to agenda
>> mode). 
>>
>> I find it odd that someone can on one hand argue so hard against using a
>> mode based on the complexity and vast nubmer of key bindings and at the
>> same time admit they have never spent the time to learn the mode. This
>> seems like an arguument based on fear and uncertainty about something
>> you don't know rather thaa one based on fact. Perhaps look more closely
>> at what was actually being proposed (which was not a full org mode
>> situation) and spend enough time to at least udnerstand the basics
>> before condemnation. 
>
> What's more odd is expecting someone to go through the entire Org manual
> in order to write and edit README files.

This thread has been a bit disheartening to read from the sidelines; as
much as I empathize with wanting to keep Emacs accessible by avoiding
code bloat and keeping cognitive load low, it is hard to know what to
reply to statements such as this one.

No one has ever had to "go through the entire Org manual in order to
write and edit README files".  Just like no one has ever had to read the
Emacs manual cover-to-cover in order to do anything with it (they _can_,
of course, but they don't _have to_).

My first years with Org were spent using (1) TAB folding (2) outline
navigation commands, and that's it.  We already have *Help* commands
which combine those features on the development branch: see C-h b with
the recently added describe-bindings-outline user option.


I don't see how enabling these section-folding and navigation features
could hurt the accessibility of package READMEs written in a structured
format.  Alan rightly points out that Org overreaches with unhelpful key
bindings, and the benchmarks posted earlier show that loading Org
wholesale for displaying READMEs would be irksome; to me that merely
suggests that whatever subset of Org features we find relevant for
_consulting_ READMEs should be better modularized.

So I would expect the discussion to focus on:

(1) what this subset should be (e.g. should it include font-locking),

(2) whether we want to add "escape hatches" for whoever struggles with
    those "rich" READMEs.

    E.g. a package-show-plain-readmes option, which could do any number
    of things depending on the underlying markup format.  For Org
    specifically, it could pass the README through the (ill-named, and
    not limited to ASCII) org-ascii-export-as-ascii, which converts Org
    markup to something more appropriate for font-lock-less viewing
    (sections underlined with equals and hyphens; inline links moved to
    their own paragraphs).

    It might even make sense for (Non)GNU ELPA to automatically run that
    function on Org READMEs in order to ensure the "bare" variant is
    always instantly available to package users.


Instead it feels like a lot of words are spent on

(1) Org is bloated and complex: yes, agreed,

(2) Package maintainers should give users the courtesy of maintaining
    two READMEs: no, I don't see a universe where that will fly.


I hope at the end of all this, we can find some middle ground between
maintainers who happen to find their productivity increased by using
Org, and maintainers who cringe at every cycle their CPU spends on Org
code.  I believe this middle ground exists.

Re: Convert README.org to plain text README while installing package

[Po Lu]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> This thread has been a bit disheartening to read from the sidelines; as
> much as I empathize with wanting to keep Emacs accessible by avoiding
> code bloat and keeping cognitive load low, it is hard to know what to
> reply to statements such as this one.
>
> No one has ever had to "go through the entire Org manual in order to
> write and edit README files".  Just like no one has ever had to read the
> Emacs manual cover-to-cover in order to do anything with it (they _can_,
> of course, but they don't _have to_).

So what was the purpose of the following statement?

>> Spend the time to go through the key bindings listed in the org
>> manual

> I don't see how enabling these section-folding and navigation features
> could hurt the accessibility of package READMEs written in a structured
> format.  Alan rightly points out that Org overreaches with unhelpful key
> bindings, and the benchmarks posted earlier show that loading Org
> wholesale for displaying READMEs would be irksome; to me that merely
> suggests that whatever subset of Org features we find relevant for
> _consulting_ READMEs should be better modularized.

IMO, packages should provide a plain text README, alongside a rich text
version in a relatively standard format like HTML.

By using HTML, it becomes readable not only in Emacs, but also any other
program that understands HTML.  HTML is also understood by many more
people than Org mode.

It's okay for package maintainers to write their documentation using
Org, and then to export it to HTML.  But please don't insist that all
documentation be written using Org mode, or that it's okay for packages
to only provide documentation in that format.

Re: Convert README.org to plain text README while installing package

[Kévin Le Gouguec]

Po Lu <luangruo@yahoo.com> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> No one has ever had to "go through the entire Org manual in order to
>> write and edit README files".  Just like no one has ever had to read the
>> Emacs manual cover-to-cover in order to do anything with it (they _can_,
>> of course, but they don't _have to_).
>
> So what was the purpose of the following statement?
>
>>> Spend the time to go through the key bindings listed in the org
>>> manual

Cannot speak for Alan nor Tim, but my understanding of the exchange that
led to the bit you quote is that Tim was inviting Alan to check how many
of these 784 bindings had any relevance to the topic under discussion
(navigating and, to a lesser extent, authoring Org READMEs) in order to
show that "M-x org TAB" was not a good heuristic to assess how much
complexity _needs_ to be shoved onto an unsuspecting README reader.

In Tim's own words (a bit before the one you quote):

> That 750 key bindings is an extreme over statement and not what is being
> proposed.

So your paraphrasing ("expecting someone to go through the entire Org
manual in order to write and edit README files") struck me as
disconnected from (my own understanding of) Tim's argument: AFAIU,

* he wasn't saying _Org users_ need to go through the manual,

* he was confronting the "784 bindings" argument by saying it was not a
  particularly useful heuristic, inviting _us_ (participants of the
  present discussion) to check for ourselves what proportion of these
  bindings are actually relevant to the discussion.

Again though, cannot speak for Tim; I can only explain my undertanding
of the discussion, and why I was dumbfounded by your reply.

> IMO, packages should provide a plain text README, alongside a rich text
> version in a relatively standard format like HTML.
>
> By using HTML, it becomes readable not only in Emacs, but also any other
> program that understands HTML.  HTML is also understood by many more
> people than Org mode.
>
> It's okay for package maintainers to write their documentation using
> Org, and then to export it to HTML.  But please don't insist that all
> documentation be written using Org mode, or that it's okay for packages
> to only provide documentation in that format.

If there's been advocating for all documentation to be written in Org,
I've (dis)missed that, so sorry for not following closely enough.

As for packages only providing documentation in Org: again, I think the
way forward is automation at the package archive level.  Asking authors
to maintain two sync'ed READMEs stands a small chance of success IMO.

Re: Convert README.org to plain text README while installing package

[Po Lu]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> So your paraphrasing ("expecting someone to go through the entire Org
> manual in order to write and edit README files") struck me as
> disconnected from (my own understanding of) Tim's argument: AFAIU,
>
> * he wasn't saying _Org users_ need to go through the manual,
>
> * he was confronting the "784 bindings" argument by saying it was not a
>   particularly useful heuristic, inviting _us_ (participants of the
>   present discussion) to check for ourselves what proportion of these
>   bindings are actually relevant to the discussion.
>
> Again though, cannot speak for Tim; I can only explain my undertanding
> of the discussion, and why I was dumbfounded by your reply.

Going through the key bindings in the Org manual means to essentially
read the entire manual.

> If there's been advocating for all documentation to be written in Org,
> I've (dis)missed that, so sorry for not following closely enough.

That's how this discussion started, with Stefan K asking for all
documentation to be written in Org mode.

Re: Convert README.org to plain text README while installing package

[Kévin Le Gouguec]

Po Lu <luangruo@yahoo.com> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> So your paraphrasing ("expecting someone to go through the entire Org
>> manual in order to write and edit README files") struck me as
>> disconnected from (my own understanding of) Tim's argument: AFAIU,
>>
>> * he wasn't saying _Org users_ need to go through the manual,
>>
>> * he was confronting the "784 bindings" argument by saying it was not a
>>   particularly useful heuristic, inviting _us_ (participants of the
>>   present discussion) to check for ourselves what proportion of these
>>   bindings are actually relevant to the discussion.
>>
>> Again though, cannot speak for Tim; I can only explain my undertanding
>> of the discussion, and why I was dumbfounded by your reply.
>
> Going through the key bindings in the Org manual means to essentially
> read the entire manual.

Yes, going through keybindings ≈ reading the entire manual.  My point is
that

    asking _participants of a discussion_ to do X in order to make the
    stakes clearer (my understanding of Tim's position)
≪
    asking anyone who wants to write and edit README files to do X (your
    paraphrasing)

>> If there's been advocating for all documentation to be written in Org,
>> I've (dis)missed that, so sorry for not following closely enough.
>
> That's how this discussion started, with Stefan K asking for all
> documentation to be written in Org mode.

Assuming you are referring to
<CADwFkm=6_xP6zRSkDP1kreMKCXGBmp8Ze2WtUMeqq9-gEuGcxw@mail.gmail.com>?

The actual OP, <87leuca7v7.fsf@disroot.org>, started with something much
more modest: converting Org READMEs to plaintext.

Then, AFAIU, came ideas to (in no particular order, and not mutually
exclusive) (1) render the Org syntax rather than converting it "down" to
plaintext, (2) enable Org wholesale, (3) break down Org into smaller
pieces in order to make it less of a pain to fit the "display a rich
README" use-case.

I admit to focusing on those points of the discussion, forgetting about
StefanK's message, and then getting very confused why Org's complexity
_as an authorship tool_ mattered so much.  Apologies for not paying
enough attention.


Anyway, Philip posted a patch to the effect of what the OP suggested in
<877d5ut6z6.fsf@posteo.net>.  Digging into the ELPA admin branch, I also
see that the machinery for the conversion is exactly what I expected: it
lets Org's export library do the heavy lifting.

So AFAICT reason prevailed; package maintainers who enjoy writing
READMEs in Org can keep doing so without impacting their users; and
despite the forecast for today being pretty rainy over here, I'm
starting my day in a resolutely sunny mood.

Re: Convert README.org to plain text README while installing package

[Michael Albinus]

Tim Cross <theophilusx@gmail.com> writes:

Hi Tim,

>> I have no problem if there are structured README.org or README.md files
>> in parallel. But a README file should be plain text.
>
> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me.
>
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.
> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

My point in this discussion is, that package's README files are also
visited by people not using Emacs. I'm confident that Emacs is able to
present an org-mode structure nicely, but this isn't true outside the
Emacs world.

Yes, we're speaking about Emacs packages first-hand. But even they are
distributed outside the Emacs or GitHub/GitLab infrastructure. So we
need a single starting point for users, w/o org-mode dependencies.

Best regards, Michael.

Re: Convert README.org to plain text README while installing package

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode.

We may call it "plain text" and problem is not that we can open Org
files with any editor as plain text, but in formatting. People format
Org files in such ways that they are not readable, they may not make
spacing where it would be otherwise required, in other words, plain
text files do not look nearly as readable as for example RFC text,
like this one: https://www.rfc-editor.org/rfc/rfc1.txt

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode.
>
> We may call it "plain text" and problem is not that we can open Org
> files with any editor as plain text, but in formatting. People format
> Org files in such ways that they are not readable, they may not make
> spacing where it would be otherwise required, in other words, plain
> text files do not look nearly as readable as for example RFC text,
> like this one: https://www.rfc-editor.org/rfc/rfc1.txt

Please give me an example of org mode 'not make space where it would be
otherwise required'. Can you provide a single example of org mode
syntax which is not readable in any text editor. There are quite a few
projects on Github/Gitlab which have readme.org files - can you point to
one which cannot be read with a plain text editor? 

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> Jean Louis <bugs@gnu.support> writes:
>
>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>> points for org mode.
>>
>> We may call it "plain text" and problem is not that we can open Org
>> files with any editor as plain text, but in formatting. People format
>> Org files in such ways that they are not readable, they may not make
>> spacing where it would be otherwise required, in other words, plain
>> text files do not look nearly as readable as for example RFC text,
>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>
> Please give me an example of org mode 'not make space where it would be
> otherwise required'. Can you provide a single example of org mode
> syntax which is not readable in any text editor. There are quite a few
> projects on Github/Gitlab which have readme.org files - can you point to
> one which cannot be read with a plain text editor? 

I'd say that Jean made a fair point.
In Org, authors may not care as much about, for example, indentation.
Especially when the org files are written with org-indent-mode turned
on.

If you look at https://www.rfc-editor.org/rfc/rfc1.txt, the paragraphs a
nicely indented and can be distinguished from the headers. The top-level
headers are numbered and can be easily distinguished from the
lower-level headers. The paragraph text is also filled appropriately,
unlike some Org documents written with truncate-lines set to nil.

So, without Emacs, Org files (some of them!) may be harder to read
compared to properly formatted ASCII.

I am not 100% sure if we need to do anything about this observation.
One practical conclusion that can be made is that we might incorporate
filling, numbering, and indentation into ox-org.el.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Jean Louis <bugs@gnu.support> writes:
>>
>>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>>> points for org mode.
>>>
>>> We may call it "plain text" and problem is not that we can open Org
>>> files with any editor as plain text, but in formatting. People format
>>> Org files in such ways that they are not readable, they may not make
>>> spacing where it would be otherwise required, in other words, plain
>>> text files do not look nearly as readable as for example RFC text,
>>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>>
>> Please give me an example of org mode 'not make space where it would be
>> otherwise required'. Can you provide a single example of org mode
>> syntax which is not readable in any text editor. There are quite a few
>> projects on Github/Gitlab which have readme.org files - can you point to
>> one which cannot be read with a plain text editor? 
>
> I'd say that Jean made a fair point.
> In Org, authors may not care as much about, for example, indentation.
> Especially when the org files are written with org-indent-mode turned
> on.
>
> If you look at https://www.rfc-editor.org/rfc/rfc1.txt, the paragraphs a
> nicely indented and can be distinguished from the headers. The top-level
> headers are numbered and can be easily distinguished from the
> lower-level headers. The paragraph text is also filled appropriately,
> unlike some Org documents written with truncate-lines set to nil.
>
> So, without Emacs, Org files (some of them!) may be harder to read
> compared to properly formatted ASCII.
>
> I am not 100% sure if we need to do anything about this observation.
> One practical conclusion that can be made is that we might incorporate
> filling, numbering, and indentation into ox-org.el.
>

On one level, fair enough. However, this has nothing to do with org mode
in the sense it is not enforced or caused by org mode. My point is that
there is nothing in the org mode syntax which makes it impossible to
read the contents with any plain text editor. You can also write a
poorly formatted ascii file which lacks indentation, spaces etc in any editor. 

There is nothing in org syntax which forces people to set truncate lines
to nil or prevents them from indenting or wrapping or adding underlines
or extra spaces around headings or .... If you want to write rfc
compliant files with org mode, you can. It isn't the org mode syntax
which is the issue. Besides, just because a plain text file doesn't look
as nice or isn't a subjectively readable doesn't mean it isn't a plain
text file anymore. 

Re: Convert README.org to plain text README while installing package

[Jean Louis]

I think that one of points conveyed with the term "plain text" means readable. That is what README should be about. Even the title of the file (README) indicates it is for reading. Org files are source files, they shall for public be exported to the actual text. The term "plain text" is not beneficial in this subject. Those files should be readable without Emacs.

Take as analogy the TeX system, or Texinfo, that is also plain text but not readable. 
 

Jean

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> On one level, fair enough. However, this has nothing to do with org mode
> in the sense it is not enforced or caused by org mode. My point is that
> there is nothing in the org mode syntax which makes it impossible to
> read the contents with any plain text editor. You can also write a
> poorly formatted ascii file which lacks indentation, spaces etc in any editor. 

Yes, but the poor formatting is immediately noticeable in ASCII. Not in
Org.

One may provide an org-lint checker for stylistic formatting issues
maybe?

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Jean Louis]

It was not the point if file is readable by computer program named text editor.

The point is if it is readable for human.

There are many Emacs users who don't use Org. They are not supposed to decipher brackets, properties, links, and lack of authors' sense in writing Org files in human readable manner.

This whole subject is related to accessibility, Org files could be accessible and thus well readable if they would be just simple text without any markup, without  headings, but they are practically not so. Authors write headings without space to its paragraph, it destroys the original meaning of the heading, file does not convey the intended idea unless you are Emscd wizard.

Difficulties in reading reject users.

I will give you examples later.

On June 8, 2022 6:50:24 AM UTC, Tim Cross <theophilusx@gmail.com> wrote:
>
>Jean Louis <bugs@gnu.support> writes:
>
>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>> Org files *are* plain text. This is one of (perhaps the biggest)
>selling
>>> points for org mode.
>>
>> We may call it "plain text" and problem is not that we can open Org
>> files with any editor as plain text, but in formatting. People format
>> Org files in such ways that they are not readable, they may not make
>> spacing where it would be otherwise required, in other words, plain
>> text files do not look nearly as readable as for example RFC text,
>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>
>Please give me an example of org mode 'not make space where it would be
>otherwise required'. Can you provide a single example of org mode
>syntax which is not readable in any text editor. There are quite a few
>projects on Github/Gitlab which have readme.org files - can you point
>to
>one which cannot be read with a plain text editor? 


Jean

Re: Convert README.org to plain text README while installing package

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2022-06-08 09:57]:
> Please give me an example of org mode 'not make space where it would be
> otherwise required'. Can you provide a single example of org mode
> syntax which is not readable in any text editor. There are quite a few
> projects on Github/Gitlab which have readme.org files - can you point to
> one which cannot be read with a plain text editor? 

Hello Tim,

my response time is not so fast.

I am now reviewing package descriptions:

- adoc-mode -- it uses Markdown to display descriptions, this is
  similar case to using Org. However, it is pretty human readable. In
  any case, both Markdown or Org formats should be converted to plain
  text. 

- ample-theme -- uses Org for package description, I consider this
  description not human readable. It is Emacs package theme. It is not
  even related to Org mode, but user has to look into following:

> * ample-theme.el
>   - A Light and Dark pair of themes for Emacs.
>   - Full colored gui and 256 color support only for now
> ** Installation
>     If MELPA is set up: 
>     #+BEGIN_SRC emacs-lisp
>       (package-install 'ample-theme)
      
>       (load-theme 'ample t)
>     #+END_SRC
> ** If you get ugly colors in terminal:
> #+BEGIN_SRC shell-script
>   # throw this in your ~/.bash_profile
>   export TERM="xterm-256color"
> #+END_SRC
> ** Ample-light-theme vs Ample-theme
>     [[http://i.imgur.com/86VLSV9.png]]

In the above Org type of a description, author does not have a sense
even to separate headings with the blank space. 

And how does "#+BEGIN_SRC emacs-lisp" help user to install the theme?
Can we understand that user simply wants to install the theme and that
"#+BEGIN_SRC emacs-lisp" is there not in right place, it is
confusing. Author expects of the user to know what is Org mode in
addition to installation of a theme, that is not proportional. Users
installing themes may not need to know nothing about Org.

Recommended for everybody to read:

Usability 101: Introduction to Usability
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

Usability Testing 101
https://www.nngroup.com/articles/usability-testing-101/

More reviews:

- package corfu, totally not human readable as it begins with:

> #+title: corfu.el - Completion Overlay Region FUnction
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: corfu.texi
> #+texinfo_dir_category: Emacs
> #+texinfo_dir_title: Corfu: (corfu).
> #+texinfo_dir_desc: Completion Overlay Region FUnction

> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
> #+html: <a href="http://elpa.gnu.org/packages/corfu.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/corfu.svg"/></a>
> #+html: <a href="http://elpa.gnu.org/devel/corfu.html"><img
> #alt="GNU-devel ELPA"
> #src="https://elpa.gnu.org/devel/corfu.svg"/></a

- package crdt -- pretty confusing when reading that Org, something
  like this:

> =M-x crdt-list-sessions= lists all sessions.

it expects user to know that "=" is markup. It cannot help to full
understanding. Users of CRDT need not be Org users. Even many Org
users do not know full Org markup.

- package ftable

> #+TITLE: ftable.el

> This package provides some convenient commands for filling a table, i.e., adjusting the layout of the table so it can fit in n columns.

> [[./ftable.gif]]

How is that tag "#+TITLE: ftable.el" helping user? We cannot assume
that users will know about the meaning of "#+" and why shall package
description again repeat the title of the package, that is already in
the header of the package. 

And how am I supposed to know what is "[[./ftable.gif]]" and where it
is? I have no access to it. It communicates zero information to
user. It confuses user.

Usability would be greatly improved if there would be simple link to
the image, as the package description is anyway showing links
correctly. 

- package leaf -- this one completely expects user to know about Org
  markup. It has terrible readibility score.

- package logos -- until user comes to read anything about
  description, must go through this garbage:

> #+title: logos.el: simple focus mode for Emacs with page breaks or outlines
> #+author: Protesilaos Stavrou
> #+email: info@protesilaos.com
> #+language: en
> #+options: ':t toc:nil author:t email:t num:t
> #+startup: content

> #+macro: stable-version 0.3.0
> #+macro: release-date 2022-03-30
> #+macro: development-version 0.4.0-dev
> #+macro: file @@texinfo:@file{@@$1@@texinfo:}@@
> #+macro: space @@texinfo:@: @@
> #+macro: kbd @@texinfo:@kbd{@@$1@@texinfo:}@@

> #+export_file_name: logos.texi

> #+texinfo_filename: logos.info
> #+texinfo_dir_category: Emacs misc features
> #+texinfo_dir_title: Logos: (logos)
> #+texinfo_dir_desc: Simple focus mode with page breaks or outlines
> #+texinfo_header: @set MAINTAINERSITE @uref{https://protesilaos.com,maintainer webpage}
> #+texinfo_header: @set MAINTAINER Protesilaos Stavrou
> #+texinfo_header: @set MAINTAINEREMAIL @email{info@protesilaos.com}
> #+texinfo_header: @set MAINTAINERCONTACT @uref{mailto:info@protesilaos.com,contact the maintainer}

> #+texinfo: @insertcopying

- package org-modern -- author did not check if those links
  work. Example communication by author:

> The screenshots shows [[file:example.org][example.org]] with
> =org-modern-mode= turned on and off

Links appear there, however, the file example.org is not clickable, it
could easily be made properly clickable.

- package osm -- not readable due to Org settings in package
  description. Those settings do not belong to package description.

- package pdf-tools -- same thing, user is supposed to read
  description. Things like below are confusing:

> :PROPERTIES:
> :CREATED:  [2021-12-29 Wed 18:34]
> :ID:       5a884389-6aec-498a-90d5-f37168809b4f
> :EXPORT_FILE_NAME: index
> :END:

Those tags are by no means relevant to package description. User wants
PDF tools, and what is user supposed to mean about the ID hashes and
what else...

- package posframe -- readability is minimized due to usage of heavier
  Org markup

- package svg-lib -- user cannot click on provided screenshots in Org
  markup. Why do that? Provide correct links.

- package system-packages -- Org markup degrades human readability

I have reviewed just some of packages installed on my side. There may
be many others. Readability and usability is greatest issue in
computing and helping user with understanding.

One shall compare Org package descriptions to so many other packages
without Org and find out how Org greatly degrades readability and thus
understanding on what that package is about and how to use it. 

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Convert README.org to plain text README while installing package

[Protesilaos Stavrou]

> From: Jean Louis <bugs@gnu.support>
> Date: Sat, 11 Jun 2022 09:40:44 +0300
>
> [... 109 lines elided]
>
> - package logos -- until user comes to read anything about
>   description, must go through this garbage:
>
> [... 68 lines elided]

Hello Jean,

What you describe as "garbage" package authors consider essential to
providing high quality documentation in a variety of formats.  The
README.org is converted into an Info manual and has a corresponding HTML
page both on GNU ELPA and the author's (me) website:

- <https://elpa.gnu.org/packages/logos.html>
- <https://protesilaos.com/emacs/logos>
- Evaluate this to read the Info manual: (info "(logos) Top")

Nobody forces you to read the Org source and there are good alternatives
on offer.

All the best,
Protesilaos (or simply "Prot")

-- 
Protesilaos Stavrou
https://protesilaos.com

Re: Convert README.org to plain text README while installing package

[Jean Louis]

* Protesilaos Stavrou <info@protesilaos.com> [2022-06-11 14:17]:
> What you describe as "garbage" package authors consider essential to
> providing high quality documentation in a variety of formats.  The
> README.org is converted into an Info manual and has a corresponding HTML
> page both on GNU ELPA and the author's (me) website:

I do not doubt good intentions. I use the word "garbage" from a
viewpoint of user who does not know Org mode and who is interested to
understand package description.

Markup impairs the understanding of package descriptions. Markup does
not belong in package description.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Convert README.org to plain text README while installing package

[Tim Cross]

Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2022-06-08 09:57]:
>> Please give me an example of org mode 'not make space where it would be
>> otherwise required'. Can you provide a single example of org mode
>> syntax which is not readable in any text editor. There are quite a few
>> projects on Github/Gitlab which have readme.org files - can you point to
>> one which cannot be read with a plain text editor? 
>
> Hello Tim,
>
> my response time is not so fast.
>
> I am now reviewing package descriptions:
>
> Recommended for everybody to read:
>
> Usability 101: Introduction to Usability
> https://www.nngroup.com/articles/usability-101-introduction-to-usability/
>
> Usability Testing 101
> https://www.nngroup.com/articles/usability-testing-101/
>

I gues sour differences here are that you have conflated issues of
readabiity and usability into the definition of plain text. Few of the
criticisms you have raised IMO are specific to org mode or due to org
modes syntax. Many of them are due to the individual author's layout
and not the result of org mode syntax. You can also write documents
which are very difficult to read using just plain ASCII. They are both
plain text.

I also think you are over-stating the difficulty of reading the raw org
file contents. While I would agree they are asier to read when correctly
formatted, they are not impossible to understand without it. For example,
to say something like 

> package corfu, totally not human readable as it begins with:

> #+title: corfu.el - Completion Overlay Region FUnction
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: corfu.texi
> #+texinfo_dir_category: Emacs
> #+texinfo_dir_title: Corfu: (corfu).
> #+texinfo_dir_desc: Completion Overlay Region FUnction

is I think totally over stating the situation. I showed the above to
someone who is not a technical person and only uses computers for email,
web and office work and they were able to understand it perfectly well.
They didn't know what the texinfo lines meant, but they understood what
the title was, who the author was, the language and even the
texinfo_dir_desc being a description. I think most people are far more
capable at reading such information than you assert. They might not
understand all of it, but then again, I often read documents where I
don't understand all of it. The point is, there is a big difference
betrween being able to read contents and understand them. 

As a comparison, consider some RFCs which you mentioned previously. I've
read RFCs which perfectly comply with the formatting you referenced that
were (to me) totally unintelligible. Sometimes this is because the
writing is poor or the concepts are too alien or it is simply too
complicated for my limited understanding. However, this is still a plain
text document.  

For me, a file is plain text if you can edit and save it using just a
plain text editor and the file will still be valid. I also consider HTML
files to be plain text, (but far less readable than org files). On the
other hand, *.docx are not plain text and cannot be edited with a plain
text editor, saved and still be valid. Likewise, other formats which
use non-printable control characters are not plain text.

I will try to be clearer when I say something is plain text. Readability
and useability are separate from file formats - they are a higher, but
related, concern. Plain text does not imply anything about readability
or usability other than a plain text file can be opened and physically
read using a plain text editor or a utility like 'cat'. The ability of
the individual to interpret that information does not affect whether it
is plain text. If, for example, you write lots of HTML, an HTML file is
easily understood when displayed with cat. 

Of course, appropriate formatting within a plain text document can
affect how easily the document can be read and I think it is good to try
and format the document to be as easy to read as possible. However, for
org mode, this is really under the control of the author. Org data files
can be formatted in a way which makes them easier to read and there are
various Emacs settings which can be used which affects this (wrap,
visual line mode, adding blank lines at the end of sections etc). The
point is, many of your criticisms are not due to org syntax or org
enforced forrmatting, which is one reason I disagree with assertions
that org files are not plain text.

I'm sure this is apoint we will never agree on, so happy to leave it there.

Re: Convert README.org to plain text README while installing package

[Jean Louis]

* Tim Cross <theophilusx@gmail.com> [2022-06-12 06:15]:
> I gues sour differences here are that you have conflated issues of
> readabiity and usability into the definition of plain text.

I did not. I speak of understanding of package descriptions. When user
clicks on a package within Emacs that user expects package
description, not a markup.

It is totally irrelevant for user if markup and tags, properties are
plain text in the context of editors. 

What matters is if user is helped to understand the package
description. Or maybe user is obstructed to understand it.

> Few of the criticisms you have raised IMO are specific to org mode
> or due to org modes syntax.

They are not specific, it just happens that I had in my package list
several Org type package descriptions. If too complex Markdown markup
is used it will also obstruct user to understand package description.

> Many of them are due to the individual author's layout and not the
> result of org mode syntax. You can also write documents which are
> very difficult to read using just plain ASCII. They are both plain
> text.

I do not think that "plain text" is relevant. HTML markup is also
plain text, it is irrelevant in the context of a user reading package
descriptions. User is not supposed to read HTML tags either, right?

> I also think you are over-stating the difficulty of reading the raw org
> file contents. While I would agree they are asier to read when correctly
> formatted, they are not impossible to understand without it.

You may stay in your viewpoint of knowledgable Emacs user, in that
sense there is also no need for package descriptions, then we can
simply tell users to read sources (irony). One has to mentally become
a user who does not know Org, the one who simply wants to find out
what package is about. Only so one can understand it.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> I did not. I speak of understanding of package descriptions. When user
> clicks on a package within Emacs that user expects package
> description, not a markup.

Dunno (I'm not really following this thread), but
maybe this example is relevant in discussing the
difference in effect between viewing/reading really
_plain_ plain-text and viewing/reading plain-text
that has some mark-up, labels, instructions, or
whatever you might call it mixed in with the text:


Consider the difference between viewing/reading a
Commentary section in a Lisp file and viewing/reading
it in the buffer that `M-x finder-commentary' creates.

This difference is about as minimal as you can get.
And yet `finder-commentary' is useful, just to give
you a friendlier, simpler, more searchable, and more
readable presentation.

All that command does is show you a buffer with just
the Commentary text - no code, comment chars removed,
etc., and with the `...'-quoted names of known
libraries converted to links to those libraries.

It's very rudimentary, but it shows well the benefit
of removing even just a few, slight distractions to
give users a really _plain_ plain-text description,
in the interest of reading it as doc.

Re: Convert README.org to plain text README while installing package

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > I did not. I speak of understanding of package descriptions. When user
  > clicks on a package within Emacs that user expects package
  > description, not a markup.

  > It is totally irrelevant for user if markup and tags, properties are
  > plain text in the context of editors. 

The idea of adding markup to text is that a program will read that
marked-up text and output text which is nicely formatted.  The
marked-up text is harder to read, but the markup effects in the output
make it easier to read.

If we use markup in some text, we shouldn't normally show users that
marked-up source text; we should normally show users the result of
processing the markup.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Tassilo Horn <tsdh@gnu.org> writes:

>> It also takes a relatively long time to load Org mode, so opening a
>> README.org-type file for the first time will always cause an annoying
>> pause.
>
> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.

We, Org devs, are working on modularizing Org.
Currently, extracting fontification is not trivial because it depends on
most of Org being loaded.

If everything goes as planned (famous last words), fontification will
only require Org parser and Org fontification library.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> We, Org devs, are working on modularizing Org.
> Currently, extracting fontification is not trivial because it depends on
> most of Org being loaded.
>
> If everything goes as planned (famous last words), fontification will
> only require Org parser and Org fontification library.

Perhaps it would be a smaller project to just make a super-simple Org
parser/displayer to handle what we typically see in a README.org file?
I.e., something that does nice fontification of #+ blocks and ***
headings and =this= (etc.), and then just removes all the rest of the
markup.

I mean, it sounds like it could be a very small, lightweight library to
get to a 90% solution that would be acceptable to show in the *Help*
buffer.  And then we wouldn't have to convert anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I mean, it sounds like it could be a very small, lightweight library to
> get to a 90% solution that would be acceptable to show in the *Help*
> buffer.  And then we wouldn't have to convert anything.

(And the same for README.md, of course.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> We, Org devs, are working on modularizing Org.
>> Currently, extracting fontification is not trivial because it depends on
>> most of Org being loaded.
>>
>> If everything goes as planned (famous last words), fontification will
>> only require Org parser and Org fontification library.
>
> Perhaps it would be a smaller project to just make a super-simple Org
> parser/displayer to handle what we typically see in a README.org file?
> I.e., something that does nice fontification of #+ blocks and ***
> headings and =this= (etc.), and then just removes all the rest of the
> markup.
>
> I mean, it sounds like it could be a very small, lightweight library to
> get to a 90% solution that would be acceptable to show in the *Help*
> buffer.  And then we wouldn't have to convert anything.

In short, no.

Org is not context free, so we at least need org-element.el parser to
handle various edge cases. In particular, there are quite annoying edge
cases with fontification of links containing */- markup symbols.

One could extract a subset of allowed Org elements from org-element.el,
but it would also involve refactoring org-element--current-element,
making the whole effort a maintenance burden against future changes in
Org.

And even if that job is done, we cannot remove links from the parser.
Yet, according to the previous benchmarks in this thread, loading links
(that is what org-modules contains by default) is giving the largest
overheads.

IMHO, the proposed lightweight library will require similar efforts to
modularizing Org + loading it with some minimal variable settings (for
example, org-modules set to nil).

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> In short, no.

Is that a challenge?  😀

> And even if that job is done, we cannot remove links from the parser.
> Yet, according to the previous benchmarks in this thread, loading links
> (that is what org-modules contains by default) is giving the largest
> overheads.

What do you mean by "links" in this context?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> In short, no.
>
> Is that a challenge?  😀

Sorry, I do not understand what you mean here.

>> And even if that job is done, we cannot remove links from the parser.
>> Yet, according to the previous benchmarks in this thread, loading links
>> (that is what org-modules contains by default) is giving the largest
>> overheads.
>
> What do you mean by "links" in this context?

In this specific paragraph, I was referring to file:, id:, help:, and
https: links. They (except help:) can be easily found in various README
files. Setting up links involves allocation of fairly large strings (see
org-element--set-regexps), which, I suspect, is creating a lot of load
on Emacs GC and slowing down the loading (especially when Emacs session
is already running for some time and GC must sweep across all the
available objects).

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Ihor Radchenko <yantar92@gmail.com> writes:

>>> In short, no.
>>
>> Is that a challenge?  😀
>
> Sorry, I do not understand what you mean here.

Lars is asking if he should hack up a solution which will probably work
well enough for 95% of all existing README.org/md files, so just say
"yes"! ;-)

Bye,
Tassilo

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Tassilo Horn <tsdh@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>>> In short, no.
>>>
>>> Is that a challenge?  😀
>>
>> Sorry, I do not understand what you mean here.
>
> Lars is asking if he should hack up a solution which will probably work
> well enough for 95% of all existing README.org/md files, so just say
> "yes"! ;-)

:) Done.

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> In this specific paragraph, I was referring to file:, id:, help:, and
> https: links.

So you were talking about these?

[[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]

Then I posit that it's trivial to get to a speedy 90% solution for these.

Anyway, for the people that say we should convert README.org files to
"text" and display that -- that's not a very satisfactory solution.  To
take an arbitrary README.org -- the one in eev.  The text version starts
off with:

---
  The best introductions to eev are:

  - "[How to record executable notes with eev - and how to play them
    back]": my talk at the EmacsConf 2019. Executable notes are mostly
    made of [sexp hyperlinks] and [eepitch blocks]. This talk has a
    quick explanation of sexp hyperlinks at [slides 5 and 6], a mention
---

which looks nice, but all those []'s are links, so you get a block down
below that's like:

---
[How to record executable notes with eev - and how to play them back]
<http://angg.twu.net/emacsconf2019.html>

[sexp hyperlinks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#3>

[eepitch blocks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6>
---

Which isn't very nice.  It should be displayed as:






-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

(Sorry, I may have sent this twice.)

Ihor Radchenko <yantar92@gmail.com> writes:

> In this specific paragraph, I was referring to file:, id:, help:, and
> https: links.

So you were talking about these?

[[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]

Then I posit that it's trivial to get to a speedy 90% solution for these.

Anyway, for the people that say we should convert README.org files to
"text" and display that -- that's not a very satisfactory solution.  To
take an arbitrary README.org -- the one in eev.  The text version starts
off with:

---
  The best introductions to eev are:

  - "[How to record executable notes with eev - and how to play them
    back]": my talk at the EmacsConf 2019. Executable notes are mostly
    made of [sexp hyperlinks] and [eepitch blocks]. This talk has a
    quick explanation of sexp hyperlinks at [slides 5 and 6], a mention
---

which looks OK, but all those []'s are links, so you get a block down
below that's like:

---
[How to record executable notes with eev - and how to play them back]
<http://angg.twu.net/emacsconf2019.html>

[sexp hyperlinks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#3>

[eepitch blocks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6>
---

Which isn't very nice.  It should be displayed as:


[-- Attachment #2: Type: image/png, Size: 58071 bytes --]

[-- Attachment #3: Type: text/plain, Size: 247 bytes --]


With clickable links and all.

I suspect that we can get there with not all that much effort if
somebody's interested in just doing the work.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> In this specific paragraph, I was referring to file:, id:, help:, and
>> https: links.
>
> So you were talking about these?
>
> [[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]
>
> Then I posit that it's trivial to get to a speedy 90% solution for these.

If you are interested to do this, feel free to go ahead. If you need any
help to get started, feel free to ask me or post a question in Org ML.

I am a bit biased about fontification because of
https://orgmode.org/list/87ee7c9quk.fsf@localhost
I keep thinking about various edge cases.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Lars Ingebrigtsen]

Tassilo Horn <tsdh@gnu.org> writes:

> So the choice of format is natural for package authors.  And they
> frequently contain markup like headings, bullet lists, bold, italics,
> and code snippets which simply look much better than their text-only
> alternative (even in their editing/non-rendered versions).

I think most of the hostility non-Org users have towards Org files is
the experience of opening one, only seeing a couple of lines with "..."
after them and then going "AAARGH", killing the buffer, and using
`find-file-literally' on it.

But Org files are totally innocuous if you say `M-x org-show-subtree'.
package.el could totally use org files instead of text files for the
readmes if it ran `org-show-subtree' when displaying the Org readmes --
most people wouldn't even notice that they were reading an Org file
(beyond the nice fontification).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

Lars Ingebrigtsen [2022-06-05 13:29:28] wrote:
> Tassilo Horn <tsdh@gnu.org> writes:
>> So the choice of format is natural for package authors.  And they
>> frequently contain markup like headings, bullet lists, bold, italics,
>> and code snippets which simply look much better than their text-only
>> alternative (even in their editing/non-rendered versions).
> I think most of the hostility non-Org users have towards Org files is
> the experience of opening one, only seeing a couple of lines with "..."
> after them and then going "AAARGH", killing the buffer, and using
> `find-file-literally' on it.

There's also the annoying delay, of course (as in "oh, so it takes
a long time to load in order to then mess it up!?")

We should take these as challenges to improve org-mode rather than as
reasons to stay away from it.


        Stefan

Re: Convert README.org to plain text README while installing package

[Alan Mackenzie]

Hello, Stefan.

On Sun, Jun 05, 2022 at 09:39:13 -0400, Stefan Monnier wrote:
> Lars Ingebrigtsen [2022-06-05 13:29:28] wrote:
> > Tassilo Horn <tsdh@gnu.org> writes:
> >> So the choice of format is natural for package authors.  And they
> >> frequently contain markup like headings, bullet lists, bold, italics,
> >> and code snippets which simply look much better than their text-only
> >> alternative (even in their editing/non-rendered versions).
> > I think most of the hostility non-Org users have towards Org files is
> > the experience of opening one, only seeing a couple of lines with "..."
> > after them and then going "AAARGH", killing the buffer, and using
> > `find-file-literally' on it.

> There's also the annoying delay, of course (as in "oh, so it takes
> a long time to load in order to then mess it up!?")

> We should take these as challenges to improve org-mode rather than as
> reasons to stay away from it.

We should not inflict org mode on random Emacs users, e.g. by making it
necessary in order to use *Help* fully.

Org mode is a complicated, difficult to learn package which cannot be to
everybody's taste.  To see just how complicated and difficult to learn
it is, have a look at the Key index in org.info.  There are 784 entries
in that Key index.  That is surely more key bindings than I know and use
in total.  A massive number of these are bindings which don't start with
C-c C-<letter>.  Many of these will most assuredly clash with other
bindings users have set up.

Don't get me wrong, I've no objection to people liking and using a
package like org mode.  Where I draw the line is when I am myself forced
to use it as part of accessing some unrelated feature.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Don't get me wrong, I've no objection to people liking and using a
> package like org mode.  Where I draw the line is when I am myself forced
> to use it as part of accessing some unrelated feature.

Noone's proposing to force you to use it.  The proposition is to use
org-mode internally when displaying a README.org file so as to make the
*Help* text more pretty.  The user wouldn't need to know that Org is
being used and the same key bindings of *Help* buffers would still do
the same as before (maybe additional key-bindings would be offered,
e.g. to fold&unfold parts, of course).


        Stefan

Re: Convert README.org to plain text README while installing package

[Po Lu]

Tassilo Horn <tsdh@gnu.org> writes:

> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on
> some forge (sr.ht, github, gitlab, gitea, whatever) which supports
> rendering those formats in a nice way.  So the choice of format is
> natural for package authors.  And they frequently contain markup like
> headings, bullet lists, bold, italics, and code snippets which simply
> look much better than their text-only alternative (even in their
> editing/non-rendered versions).

But that doesn't mean all documentation should take on such a format.
People read their READMEs in Emacs, not on a forge.

> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.

That wouldn't be Org mode anymore then, just an outline with some
additional fontification rules.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> That's true but wouldn't it be possible to extract the font-lock/faces
>> related parts in a separate file with no dependencies which can be
>> loaded quickly enough?  I mean, the purpose would only be to display org
>> and markdown READMEs with the highlighting intended by the package
>> author but there's no point in enabling the gazillon of other features
>> (from editing to time-tracking to spreadsheet computations) those
>> packages might provide in addition.
>
> That wouldn't be Org mode anymore then, just an outline with some
> additional fontification rules.

Isn't *Help* buffer supposed to be read-only? There is no point loading
Org editing features if editing is not going to happen.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>> That's true but wouldn't it be possible to extract the font-lock/faces
>>> related parts in a separate file with no dependencies which can be
>>> loaded quickly enough?  I mean, the purpose would only be to display org
>>> and markdown READMEs with the highlighting intended by the package
>>> author but there's no point in enabling the gazillon of other features
>>> (from editing to time-tracking to spreadsheet computations) those
>>> packages might provide in addition.
>>
>> That wouldn't be Org mode anymore then, just an outline with some
>> additional fontification rules.
>
> Isn't *Help* buffer supposed to be read-only? There is no point loading
> Org editing features if editing is not going to happen.
>
> Best,
> Ihor

Yes, *Help* is read only.

I can't understand why people are suggesting to Org mode in package
description.  I'm again saying that the README.org should be converted
(exported) to plain text on package installation (I won't recommend to
do this server, since its probably SaaSS and would increase server
load).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Akib Azmain Turja <akib@disroot.org> writes:

Hi Akib,

> I can't understand why people are suggesting to Org mode in package
> description.

I'm not sure if we are miscommunicating.  Of course, nobody should be
mandated to write his package's README as org file.  But as a matter of
fact, many package authors do it that way or in markdown simply because
both are good formats for structured text and are supported by most
forges (markdown even more so), i.e., the rendered version of the README
becomes the project's homepage without any additional work.  IMHO,
that's good: it encourages package authors to write good READMEs because
it's the very first thing a prospective user will see.

So again, I was only suggesting that if (and only if) a package declares
a :readme "README.org" / "README.md", it would be nice if it was
*displayed* as in the "native" modes.  I certainly acknowledge technical
arguments, e.g., loading the org package is too slow (right now).

Let me ask the other way round: why can't you understand that someone
thinks it's a good idea if the package description is displayed as the
author intended?

> I'm again saying that the README.org should be converted (exported) to
> plain text on package installation (I won't recommend to do this
> server, since its probably SaaSS and would increase server load).

I've just tried org-exporting the vertico README.org to a plain-text
UTF-8 version and have to confess, the result is really, really
intriguing.  So, yes, such a conversion is at least much better as the
status quo where the org syntax is displayed literally without
highlighting.  (I almost don't perceive the org/markdown syntax when
there's syntax highlighting but if there isn't, it looks annoying.)

One problem with the "conversion on installation" approach you suggest
is that this would load org, too, so the arguments against using/loading
org for display would also apply here.

Bye,
Tassilo

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> why can't you understand that someone thinks
> it's a good idea if the package description
> is displayed as the author intended?

I'd say as the _user_ intends, not the author.

Displaying a package description is for the
benefit of the user, not the author.  It's
about what's most useful for users.

Best is this, I think:

1. Require a plain-text README, as a minimum.
2. Allow other formats, with appropriate file
   extensions (e.g. .md, .org).

I see no reason for any limit on the kinds of
format, for #2.

There should be, and likely are, simple ways
to generate a plain-text README from this or
that more structured format.  Condition #1
shouldn't be an obstacle to anyone, I'd think.

> I've just tried org-exporting the vertico README.org to a plain-text
> UTF-8 version and have to confess, the result is really, really
> intriguing.  So, yes, such a conversion is at least much better as the
> status quo where the org syntax is displayed literally without
> highlighting.  (I almost don't perceive the org/markdown syntax when
> there's syntax highlighting but if there isn't, it looks annoying.)

For Org, at least, that seems to confirm my
guess that such conversion's already available.

> One problem with the "conversion on installation"
> approach you suggest is that this would load org,
> too, so the arguments against using/loading
> org for display would also apply here.

Why shouldn't a plain-text README be required
at the outset, as part of every package?

(Another possibility, if a plain-text README
that's equivalent to a structured readme isn't
provided, is to allow a plain-text README that
tells you how to convert the structured readme
to a plain-text version.  But I don't see why
plain text shouldn't just be a requirement.)

Re: [External] : Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Drew Adams <drew.adams@oracle.com> writes:

>> why can't you understand that someone thinks
>> it's a good idea if the package description
>> is displayed as the author intended?
>
> I'd say as the _user_ intends, not the author.

You and me are both users and it seems we have different
intentions/preferences.

> Displaying a package description is for the
> benefit of the user, not the author.  It's
> about what's most useful for users.

Why do you think that a free software package author doesn't write his
documentation for this very purpose?  Exactly for this reason she has
structured the README in sections, **emphasized the important parts**,
included usage/config examples as highlighted

#+BEGIN_SRC emacs-lisp
  (code snippets)
#+END_SRC

and added clickable links to [[helpful resources][https://...]].  Why is
it "most useful" for users to strip away parts of that?

> Best is this, I think:
>
> 1. Require a plain-text README, as a minimum.
> 2. Allow other formats, with appropriate file
>    extensions (e.g. .md, .org).
>
> I see no reason for any limit on the kinds of
> format, for #2.
>
> There should be, and likely are, simple ways
> to generate a plain-text README from this or
> that more structured format.  Condition #1
> shouldn't be an obstacle to anyone, I'd think.

If there is some automatic conversion like for org, the obstacle is just
that I as a package author would need to commit a generated file with
the danger that contributors send patches against the generated document
instead of the source.  Not a big deal but also not great.

>> I've just tried org-exporting the vertico README.org to a plain-text
>> UTF-8 version and have to confess, the result is really, really
>> intriguing.  So, yes, such a conversion is at least much better as the
>> status quo where the org syntax is displayed literally without
>> highlighting.  (I almost don't perceive the org/markdown syntax when
>> there's syntax highlighting but if there isn't, it looks annoying.)
>
> For Org, at least, that seems to confirm my
> guess that such conversion's already available.

Yes, indeed.  I haven't seen anything like that for markdown (which is
probably even more popular for READMEs than org), though.

Bye,
Tassilo

Re: [External] : Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Yes, indeed.  I haven't seen anything like that for markdown (which is
> probably even more popular for READMEs than org), though.

I think that's simply because Markdown (as the name suggests) is
designed such that the Markdown source can be very close to the best
possible plain text rendering, significantly reducing the motivation to
design plain text renderer.

[ That's also why I tend to prefer Markdown for small documents, where
  I tend to find Org's markup a bit too loud.  ]


        Stefan

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> > Yes, indeed.  I haven't seen anything like that for markdown (which
> > is probably even more popular for READMEs than org), though.
> 
> I think that's simply because Markdown (as the name suggests) is
> designed such that the Markdown source can be very close to the best
> possible plain text rendering, significantly reducing the motivation to
> design plain text renderer.

Yes.  And that closeness also means that it's
generally not so difficult to go from simple
plain-text without markup to Markdown, and
vice versa.

> [ That's also why I tend to prefer Markdown for small documents, where
>   I tend to find Org's markup a bit too loud. ]

Yes again.

There are lots of nice, rendered or renderable
fancy formats, including Org and Markdown, HTML,
PDF, Info/TexInfo.

Packages should, I think, be able to provide
whatever they like, as long as they also provide
some simple README.

(And again, that simple, plain-text README need
not try to say or do all that another format
might do.  IMO.)

RE: [External] : Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Drew Adams <drew.adams@oracle.com> writes:

> There are lots of nice, rendered or renderable
> fancy formats, including Org and Markdown, HTML,
> PDF, Info/TexInfo.
>
> Packages should, I think, be able to provide
> whatever they like, as long as they also provide
> some simple README.
>
> (And again, that simple, plain-text README need
> not try to say or do all that another format
> might do.  IMO.)

While what you say makes sense, can we reasonably expect the package
authors to provide an extra README file for every single package? If it
was the case, this thread would simply not show up. ASCII README would
be always available.

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> > There are lots of nice, rendered or renderable
> > fancy formats, including Org and Markdown, HTML,
> > PDF, Info/TexInfo.
> >
> > Packages should, I think, be able to provide
> > whatever they like, as long as they also provide
> > some simple README.
> >
> > (And again, that simple, plain-text README need
> > not try to say or do all that another format
> > might do.  IMO.)
> 
> While what you say makes sense, can we reasonably expect the package
> authors to provide an extra README file for every single package?

I think we can.  But only if we ask for it.

> If it was the case, this thread would simply not
> show up. ASCII README would be always available.

No, not if we've never requested that it be included.

If we request it then it'll be provided more often,
I expect.
___

[ But we can't even get vanilla Emacs itself to add
  README's for Emacs releases on MS Windows anymore,
  as it says we can (in the overall Windows README).
  https://lists.gnu.org/archive/html/emacs-devel/2022-05/msg01378.html
]

RE: [External] : Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Drew Adams <drew.adams@oracle.com> writes:

>> If it was the case, this thread would simply not
>> show up. ASCII README would be always available.
>
> No, not if we've never requested that it be included.
>
> If we request it then it'll be provided more often,
> I expect.

Maybe.

As a single data point, if I were asked to provide such a README, I
would not bother writing it separately. Instead, I would simply export
my README.org to ASCII. And it would feel like an unnecessary burden,
honestly.

If I were asked to provide a README as an option, I would not bother at
all and just leave README.org.

Do you know how many ELPA packages provide README.org vs README.md vs
README in ASCII text?

Best,
Ihor

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> Do you know how many ELPA packages provide README.org 
> vs README.md vs README in ASCII text?

Not I.

Re: [External] : Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> As a single data point, if I were asked to provide such a README, I
> would not bother writing it separately. Instead, I would simply export
> my README.org to ASCII. And it would feel like an unnecessary burden,
> honestly.

Don't worry, as de-facto maintainer of GNU ELPA, I'd rather discourage
that kind of duplication anyway (and elpa.gnu.org already builds the
text rendering of .org and.md READMEs anyway, so it would be perverse to
ask the authors to provide another one).


        Stefan

Re: [External] : Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Ihor Radchenko <yantar92@gmail.com> writes:

> Do you know how many ELPA packages provide README.org vs README.md vs
> README in ASCII text?

GNU ELPA: 348 packages, 7 README.org / 3 README.md (one commented out)
NonGNU ELPA: 160 packages, 9 README.org / 22 README.md (one commented out)

But those are only the packages which have explicitly specified a
:readme property.  I'm not sure but think ELPA will pick up a
README.md/org if there's no README automatically so it might not needed
to be declared in which case my stats aren't very meaningful.

I have no stats for MELPA but I guess there the ratio is even more
shifted towards org/md.

Bye,
Tassilo

Re: [External] : Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Tassilo Horn <tsdh@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Do you know how many ELPA packages provide README.org vs README.md vs
>> README in ASCII text?
>
> GNU ELPA: 348 packages, 7 README.org / 3 README.md (one commented out)

There are certainly more. I have found at least 19 README.org on ELPA in
the first 70-ish packages.

Best,
Ihor

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

All I meant there was that it's good if a
package description is written from a user point
of view - what it does for users, not just what
the author intended in writing the package.

The motivation and design underlying the package
are important, and _can_ be useful to document
(for users and others).  But most useful is what
you can do with it as a user.

> > Displaying a package description is for the
> > benefit of the user, not the author.  It's
> > about what's most useful for users.
> 
> Why do you think that a free software package author
> doesn't write his documentation for this very purpose?

I didn't presume anything about how or why every
pkg author writes their pkg doc.  I assume that
most do try to provide useful doc for users.

My point was only that the description should be
useful to users.  Whether the description "is
displayed as the author intended" is, on the
other hand, essentially a nice-to-have.  (And I,
as one user, really like to have it.)

And I was clear that authors should be able to
provide whatever display format/method they like.

But I think _also_ providing a plain-text
description of some sort (a plain README) should
be a minimal requirement - at least something
that's requested with "Please at least provide a
plain-text README."

> Exactly for this reason she has
> structured the README in sections, **emphasized
> the important parts**, included usage/config
> examples as highlighted

Absolutely nothing wrong with that - quite the
contrary.  That's to be hoped for, even expected.
No one said anything to the contrary.

> and added clickable links to [[helpful
> resources][https://...]].

No one said that authors shouldn't be able
to provide helpful links, images, and other
artifacts.  Even flashes and fireworks, if
they like.

> Why is it "most useful" for users to strip
> away parts of that?

No one said that that's the case.  Not I, at
least.  Are you looking for a straw man?

All I suggested was that authors should, at
a minimum, provide a simple, plain-text file
README.

I explicitly even said that (IMO) that file
need not say the same thing as whatever their
other, formatted README.* files say.

Why is a plain-text README desirable - useful
for everyone?  Because it requires minimal
paraphernalia to easily read it or otherwise
process it.  Lowest common denominator.  This
should be a no-brainer, I think.

> > Best is this, I think:
> >
> > 1. Require a plain-text README, as a minimum.
> > 2. Allow other formats, with appropriate file
> >    extensions (e.g. .md, .org).
> >
> > I see no reason for any limit on the kinds of
> > format, for #2.
> >
> > There should be, and likely are, simple ways
> > to generate a plain-text README from this or
> > that more structured format.  Condition #1
> > shouldn't be an obstacle to anyone, I'd think.
> 
> If there is some automatic conversion like for org,
> the obstacle is just that I as a package author
> would need to commit a generated file with the danger
> that contributors send patches against the generated
> document instead of the source.  Not a big deal but
> also not great.

As you say, neither a big deal, nor a great obstacle.
 
> > For Org, at least, that seems to confirm my
> > guess that such conversion's already available.
> 
> Yes, indeed.  I haven't seen anything like that for
> markdown (which is probably even more popular for
> READMEs than org), though.

Even producing a plain-text version by manual editing
is likely not a big deal in most cases.  That's my
guess, at least.

Re: [External] : Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Drew Adams <drew.adams@oracle.com> writes:

>> Why is it "most useful" for users to strip
>> away parts of that?
>
> No one said that that's the case.  Not I, at
> least.  Are you looking for a straw man?

No, but maybe I didn't understand your suggestion which sounded to me as
if you say that there should be a mandatory plain-text README to be
displayed in Emacs plus as many "rich" versions as the package author
wants.  However, those wouldn't be displayed in Emacs.  Or rather, they
would only be displayed when you find-file them inside ~/.emacs.d/elpa/
which is quite unlikely to happen (unless the README says there are also
alternative versions of the package destription available).

>> Yes, indeed.  I haven't seen anything like that for
>> markdown (which is probably even more popular for
>> READMEs than org), though.
>
> Even producing a plain-text version by manual editing
> is likely not a big deal in most cases.  That's my
> guess, at least.

Having to synchronize two identical (wrt. contents) versions of a
document manually, one with markup and one without, is annoying IMO.

Bye,
Tassilo

RE: [External] : Re: Convert README.org to plain text README while installing package

[Drew Adams]

> maybe I didn't understand your suggestion which
> sounded to me as if you say that there should be
> a mandatory plain-text README to be displayed in
> Emacs plus as many "rich" versions as the package
> author wants.

My suggestion is apparently moot now, as Stefan has
already decided what will/won't be done, wrt GNU ELPA.

But yes I suggested that a plain README be required.

I don't think I said anything about it being displayed.
And I'm pretty sure I didn't say anything about any
other formats _not_ being displayed.

I don't have a set opinion about what should be
displayed by default.  (I'd hope that, whatever is the
default for displaying, users could somehow control
what is actually the case.)

I did say "required".  I later changed the wording to
clarify that I meant something more like "requested
politely", e.g., Please consider supplying a plain
README.

> However, those wouldn't be displayed in Emacs.

No; see above.  I didn't mean to say, and I don't
think I said, anything about what gets displayed.

Now that you bring that up, I guess my preference
would be that package authors can control what gets
displayed by default (see above, about users being
perhaps able to override the default).

But others have brought up the question of needing
this or that tool/infrastructure, to be able to
display this or that "rich" format.  I haven't
been focused on the question of display, except to
say that it takes very little to be able to read
a simple, plain-text readme.

My request for including plain README was just to
include it, not to display it.  Make it available.

> Or rather, they would only be displayed when you
> find-file them inside ~/.emacs.d/elpa/ which is
> quite unlikely to happen (unless the README says
> there are also alternative versions of the package
> destription available).

I hadn't thought about any particular mechanisms
for making this or that format available, or
displaying one or the other format.  My suggestion
(concern) was just to try to get authors to (also)
include a plain README.

> >> Yes, indeed.  I haven't seen anything like that for
> >> markdown (which is probably even more popular for
> >> READMEs than org), though.
> >
> > Even producing a plain-text version by manual editing
> > is likely not a big deal in most cases.  That's my
> > guess, at least.
> 
> Having to synchronize two identical (wrt. contents) versions of a
> document manually, one with markup and one without, is annoying IMO.

As I said, the simple, plain README need not have
the same content as any more elaborate README.*.

Its purpose could be just to let you know something
basic about the package - perhaps even including
instructions for how to display a more elaborate
README.*.

To me, a plain README could serve as a minimal
place to start, requiring next to no paraphernalia
or infrastructure to read it.  It need not be long
or in any way complete/definitive.  It should at
least serve as minimal help to get going.

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Akib Azmain Turja <akib@disroot.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>>> That's true but wouldn't it be possible to extract the font-lock/faces
>>>> related parts in a separate file with no dependencies which can be
>>>> loaded quickly enough?  I mean, the purpose would only be to display org
>>>> and markdown READMEs with the highlighting intended by the package
>>>> author but there's no point in enabling the gazillon of other features
>>>> (from editing to time-tracking to spreadsheet computations) those
>>>> packages might provide in addition.
>>>
>>> That wouldn't be Org mode anymore then, just an outline with some
>>> additional fontification rules.
>>
>> Isn't *Help* buffer supposed to be read-only? There is no point loading
>> Org editing features if editing is not going to happen.
>>
>> Best,
>> Ihor
>
> Yes, *Help* is read only.
>
> I can't understand why people are suggesting to Org mode in package
> description.  I'm again saying that the README.org should be converted
> (exported) to plain text on package installation (I won't recommend to
> do this server, since its probably SaaSS and would increase server
> load).

I don't understand how this thread diverged, this proposal seems
absolutely sensible.  Why should a package that is not installed get
converted to plain text, while when installed you see the org source
code?

From what I see this could be fixed if elpa-admin added a "README-elpa"
with the rendered text into the tarball.

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> From what I see this could be fixed if elpa-admin added a "README-elpa"
> with the rendered text into the tarball.

I think you're right.


        Stefan

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From what I see this could be fixed if elpa-admin added a "README-elpa"
>> with the rendered text into the tarball.
>
> I think you're right.

I believe this should do the job:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Render-Org-documentation-in-a-plain-text-README-elpa.patch --]
[-- Type: text/x-diff, Size: 1752 bytes --]

From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Mon, 6 Jun 2022 12:34:40 +0200
Subject: [PATCH] Render Org documentation in a plain-text README-elpa file

* elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
(elpaa--write-plain-readme): Add new function.
---
 elpa-admin.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/elpa-admin.el b/elpa-admin.el
index ecf4bc90d1..60da684f1f 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -621,6 +621,7 @@ auxillary files unless TARBALL-ONLY is non-nil ."
        (elpaa--make pkg-spec dir)
        (elpaa--build-Info pkg-spec dir destdir))
      (elpaa--write-pkg-file dir pkgname metadata revision)
+     (elpaa--write-plain-readme dir pkg-spec)
      ;; FIXME: Allow renaming files or selecting a subset of the files!
      (cl-assert (not (string-match "[][*\\|?]" pkgname)))
      (cl-assert (not (string-match "[][*\\|?]" vers)))
@@ -1162,6 +1163,17 @@ Rename DIR/ to PKG-VERS/, and return the descriptor."
      nil
      pkg-file)))
 
+(defun elpaa--write-plain-readme (pkg-dir pkg-spec)
+  "Render a plain text readme from PKG-SPEC in PKG-DIR.
+This is only done if necessary, that is if the readme contents
+are not already taken to be formatted in plain text."
+  (let* ((readme-content (elpaa--get-README pkg-spec pkg-dir)))
+    (unless (eq (car readme-content ) 'text/plain)
+      (write-region
+       (elpaa--section-to-plain-text readme-content)
+       nil
+       (expand-file-name "README-elpa" pkg-dir)))))
+
 (defun elpaa-batch-generate-description-file (&rest _)
   "(Re)build the <PKG>-pkg.el file for particular packages."
   (while command-line-args-left
-- 
2.36.1

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
> From: Philip Kaludercic <philipk@posteo.net>
> Date: Mon, 6 Jun 2022 12:34:40 +0200
> Subject: [PATCH] Render Org documentation in a plain-text README-elpa file
>
> * elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
> (elpaa--write-plain-readme): Add new function.

That looks OK, feel free to push that to `elpa-admin`.
Comments:
- you'd get more bonus points by arranging for the code
  to (re)use the existing plain text (which is currently produced later
  when (re)generating the HTML page, so it would take a fair bit of
  code reorg to get that.  In the mean time, a FIXME comment seems
  in order).
- Your code may end up "rendering" a Markdown file for no obvious benefit
  since the rendering is a no-op (tho I think this is very hypothetical
  and likely can't happen because of other constraints).
- We might also add a `README-elpa` file when the readme file has a name
  not recognized by `package--get-description`.


        Stefan

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Mon, 6 Jun 2022 12:34:40 +0200
>> Subject: [PATCH] Render Org documentation in a plain-text README-elpa file
>>
>> * elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
>> (elpaa--write-plain-readme): Add new function.
>
> That looks OK, feel free to push that to `elpa-admin`.
> Comments:
> - you'd get more bonus points by arranging for the code
>   to (re)use the existing plain text (which is currently produced later
>   when (re)generating the HTML page, so it would take a fair bit of
>   code reorg to get that.  In the mean time, a FIXME comment seems
>   in order).
> - Your code may end up "rendering" a Markdown file for no obvious benefit
>   since the rendering is a no-op (tho I think this is very hypothetical
>   and likely can't happen because of other constraints).
> - We might also add a `README-elpa` file when the readme file has a name
>   not recognized by `package--get-description`.
>
>
>         Stefan
>

Any update?

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Akib Azmain Turja <akib@disroot.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Date: Mon, 6 Jun 2022 12:34:40 +0200
>>> Subject: [PATCH] Render Org documentation in a plain-text README-elpa file
>>>
>>> * elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
>>> (elpaa--write-plain-readme): Add new function.
>>
>> That looks OK, feel free to push that to `elpa-admin`.
>> Comments:
>> - you'd get more bonus points by arranging for the code
>>   to (re)use the existing plain text (which is currently produced later
>>   when (re)generating the HTML page, so it would take a fair bit of
>>   code reorg to get that.  In the mean time, a FIXME comment seems
>>   in order).
>> - Your code may end up "rendering" a Markdown file for no obvious benefit
>>   since the rendering is a no-op (tho I think this is very hypothetical
>>   and likely can't happen because of other constraints).
>> - We might also add a `README-elpa` file when the readme file has a name
>>   not recognized by `package--get-description`.
>>
>>
>>         Stefan
>>
>
> Any update?

Sorry, this got buried in my inbox, I will try to update the patch today.

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]


I just found a pretty description with `describe-package`.  Thanks.

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Akib Azmain Turja <akib@disroot.org> writes:

> I just found a pretty description with `describe-package`.  Thanks.

What package was it?  Had it been installed locally?

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Akib Azmain Turja <akib@disroot.org> writes:
>
>> I just found a pretty description with `describe-package`.  Thanks.
>
> What package was it?  Had it been installed locally?

A few which I installed/upgraded recently, including:

* cape
* consult
* corfu
* embark
* lin
* marginalia
* osm
* svg-lib

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Mon, 6 Jun 2022 12:34:40 +0200
>> Subject: [PATCH] Render Org documentation in a plain-text README-elpa file
>>
>> * elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
>> (elpaa--write-plain-readme): Add new function.
>
> That looks OK, feel free to push that to `elpa-admin`.
> Comments:

The following should address most of your points, just one question,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Render-Org-documentation-in-a-plain-text-README-elpa.patch --]
[-- Type: text/x-diff, Size: 4063 bytes --]

From c60903f5e916b22678c43c0fba0aa73a9e0bf471 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Mon, 6 Jun 2022 12:34:40 +0200
Subject: [PATCH] Render Org documentation in a plain-text README-elpa file

* elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
(elpaa--write-plain-readme): Add new function.
(elpaa--html-make-pkg): Reuse the rendered plaintext from
`elpaa--write-plain-readme'.
---
 elpa-admin.el | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/elpa-admin.el b/elpa-admin.el
index 58c0ec2e0a..173d247e87 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -610,7 +610,8 @@ auxillary files unless TARBALL-ONLY is non-nil ."
               (lambda (file)
                 (string-match re file)
                 (cons (match-string 1 file) file))
-              (directory-files destdir nil re)))))
+              (directory-files destdir nil re))))
+          rendered)
      (when ldir
        (cl-pushnew (list (file-name-as-directory ldir) "") renames
                    :test #'equal))
@@ -621,6 +622,7 @@ auxillary files unless TARBALL-ONLY is non-nil ."
        (elpaa--make pkg-spec dir)
        (elpaa--build-Info pkg-spec dir destdir))
      (elpaa--write-pkg-file dir pkgname metadata revision)
+     (setq rendered (elpaa--write-plain-readme dir pkg-spec))
      ;; FIXME: Allow renaming files or selecting a subset of the files!
      (cl-assert (not (string-match "[][*\\|?]" pkgname)))
      (cl-assert (not (string-match "[][*\\|?]" vers)))
@@ -677,7 +679,7 @@ auxillary files unless TARBALL-ONLY is non-nil ."
            (elpaa--html-make-pkg pkgdesc pkg-spec
                                  `((,vers . ,(file-name-nondirectory tarball))
                                    . ,oldtarballs)
-                                 dir))))
+                                 dir rendered))))
      'new)))
 
 (defun elpaa--makeenv (version revision)
@@ -1162,6 +1164,22 @@ Rename DIR/ to PKG-VERS/, and return the descriptor."
      nil
      pkg-file)))
 
+(defun elpaa--write-plain-readme (pkg-dir pkg-spec)
+  "Render a plain text readme from PKG-SPEC in PKG-DIR.
+This is only done if necessary, that is if the readme contents
+are not already taken to be formatted in plain text or when the
+readme file has an unconventional name"
+  (let ((readme-file (elpaa--spec-get pkg-spec :readme))
+        (known-readme-names            ;see `package--get-description'
+         '("README-elpa" "README-elpa.md" "README" "README.rst" "README.org"))
+        (readme-content (elpaa--get-README pkg-spec pkg-dir)))
+    (if (or (not (eq (car readme-content) 'text/plain))
+            (not (member readme-file known-readme-names)))
+        (let ((rendered (elpaa--section-to-plain-text readme-content)))
+          (write-region rendered nil (expand-file-name "README-elpa" pkg-dir))
+          rendered)
+      readme-content)))
+
 (defun elpaa-batch-generate-description-file (&rest _)
   "(Re)build the <PKG>-pkg.el file for particular packages."
   (while command-line-args-left
@@ -1489,7 +1507,7 @@ arbitrary code."
 	  ))
       (insert "</dd>\n"))))
 
-(defun elpaa--html-make-pkg (pkg pkg-spec files srcdir)
+(defun elpaa--html-make-pkg (pkg pkg-spec files srcdir plain-readme)
   (let* ((name (symbol-name (car pkg)))
          (latest (package-version-join (aref (cdr pkg) 0)))
          (mainsrcfile (expand-file-name (elpaa--main-file pkg-spec) srcdir))
@@ -1542,7 +1560,7 @@ arbitrary code."
                        <pre>M-x <span class=\"kw\">package-install</span> RET <span class=\"kw\">%s</span> RET</pre>"
                       name))
       (let* ((readme-content (elpaa--get-README pkg-spec srcdir))
-             (readme-text (elpaa--section-to-plain-text readme-content))
+             (readme-text plain-readme)
              (readme-html (elpaa--section-to-html readme-content))
              (readme-output-filename (concat name "-readme.txt")))
         (write-region readme-text nil readme-output-filename)
-- 
2.36.1


[-- Attachment #3: Type: text/plain, Size: 818 bytes --]


> - you'd get more bonus points by arranging for the code
>   to (re)use the existing plain text (which is currently produced later
>   when (re)generating the HTML page, so it would take a fair bit of
>   code reorg to get that.  In the mean time, a FIXME comment seems
>   in order).
> - Your code may end up "rendering" a Markdown file for no obvious benefit
>   since the rendering is a no-op (tho I think this is very hypothetical
>   and likely can't happen because of other constraints).

Why should this be?  Both this and the previous iteration of the patch
check (eq (car readme-content) 'text/plain), and currently there is only
text/plain (which includes markdown) and text/org?

> - We might also add a `README-elpa` file when the readme file has a name
>   not recognized by `package--get-description`.

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> The following should address most of your points, just one question,

Nicely hidden inside the signature :-)

> > - Your code may end up "rendering" a Markdown file for no obvious benefit
> >   since the rendering is a no-op (tho I think this is very hypothetical
> >   and likely can't happen because of other constraints).
> Why should this be?  Both this and the previous iteration of the patch
> check (eq (car readme-content) 'text/plain), and currently there is only
> text/plain (which includes markdown) and text/org?

text/plain does not include markdown.  Markdown files will have type
text/markdown.


        Stefan

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The following should address most of your points, just one question,
>
> Nicely hidden inside the signature :-)
>
>> > - Your code may end up "rendering" a Markdown file for no obvious benefit
>> >   since the rendering is a no-op (tho I think this is very hypothetical
>> >   and likely can't happen because of other constraints).
>> Why should this be?  Both this and the previous iteration of the patch
>> check (eq (car readme-content) 'text/plain), and currently there is only
>> text/plain (which includes markdown) and text/org?
>
> text/plain does not include markdown.  Markdown files will have type
> text/markdown.

But in that case, markdown files could still be "rendered", by replacing
regular double quotes with fancy quotes, removing unnecessary code
blocks, etc.

E.g. using "pandoc -t plain", this file

--8<---------------cut here---------------start------------->8---
# This is a title

So and so *this* and __that__, "they say", yes!

some more stuff
===============

How about some [code](https://code.com)?

```lisp
(defun call-me () (call-me) (call-me) (anytime))
```

or like this

    $ ls -l

### The Remainder

So what is left?

* A list
* A number list
  1. one
  2. two
  3. three
* a horizontal rule

---

that's it
--8<---------------cut here---------------end--------------->8---

is converted into

--8<---------------cut here---------------start------------->8---
This is a title

So and so this and that, “they say”, yes!

some more stuff

How about some code?

    (defun call-me () (call-me) (call-me) (anytime))

or like this

    $ ls -l

The Remainder

So what is left?

-   A list
-   A number list
    1.  one
    2.  two
    3.  three
-   a horizontal rule

------------------------------------------------------------------------

that’s it
--8<---------------cut here---------------end--------------->8---

While not perfect (and I am not arguing to necessarily use pandoc), I do
think that there is enough that can be done with markup while still
dealing with plain text.

>
>         Stefan

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> But in that case, markdown files could still be "rendered", by replacing

In theory, yes.  In practice, currently they aren't.

> is converted into

FWIW I find this particular conversion made things overall worse :-(


        Stefan

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But in that case, markdown files could still be "rendered", by replacing
>
> In theory, yes.  In practice, currently they aren't.

My bad, I had forgotten that `elpaa--extension-to-mime' could also
return text/markdown, which confused me here.  I'll update the patch to
only work on text/x-org and push it.

>> is converted into
>
> FWIW I find this particular conversion made things overall worse :-(

Hmm, what in particular?  As I said it doesn't have to be this specific
formatting.  It could also just be cleaned up markdown, moving links
below paragraphs, wrapping lines and stuff like that.

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

>> FWIW I find this particular conversion made things overall worse :-(
> Hmm, what in particular?

I didn't like the two spaces after the "bullets" and more importantly
the information about what is a title was completely lost.

> As I said it doesn't have to be this specific formatting.

Sure, I just mentioned it because I found it rather odd to return
a worse result than the identity function.

> It could also just be cleaned up markdown, moving links below
> paragraphs, wrapping lines and stuff like that.

Then again, this can (should?) be applied to the source directly.

Don't get me wrong: I'm definitely not opposed to rendering Markdown to
plain text.  I've seen a fair bit of "GFM" files that are not great to
read as plain text, so there is clearly room for improvement.
Maybe throwing out the badges would be a good first step ;-)


        Stefan

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> FWIW I find this particular conversion made things overall worse :-(
>> Hmm, what in particular?
>
> I didn't like the two spaces after the "bullets" and more importantly
> the information about what is a title was completely lost.
>
>> As I said it doesn't have to be this specific formatting.
>
> Sure, I just mentioned it because I found it rather odd to return
> a worse result than the identity function.

My example was a pretty clean markdown file, which is usually readable
enough in its own right, so most post-processing doesn't help.  But I
agree, it isn't as easy as just using pandoc's "plain" output.

>> It could also just be cleaned up markdown, moving links below
>> paragraphs, wrapping lines and stuff like that.
>
> Then again, this can (should?) be applied to the source directly.

As in sending upstream patches?  From my experience a lot of maintainers
(regrettably) prefer something that renders well on sites like GitHub or
GitLab than that is readable as plain text.

Or do you mean writing custom code to process markdown files?

> Don't get me wrong: I'm definitely not opposed to rendering Markdown to
> plain text.  I've seen a fair bit of "GFM" files that are not great to
> read as plain text, so there is clearly room for improvement.
> Maybe throwing out the badges would be a good first step ;-)

Then again, this ties into the README files that look better when
rendered on GitHub/Lab, and where it is worth considering if *not*
using them would be of more use.  If this is about providing a good
overview when using C-h P, using a good commentary section (that does
not include comments on how to install the package, contribution
guidelines, screenshots, etc.) might be better to begin with.

This is also related to the point I had raised earlier in the thread
that when a package generates a manual from a Org README, using the same
file as a README file results in too much (redundant) information.

>         Stefan

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> As in sending upstream patches?

That's what I meant, yes.

> From my experience a lot of maintainers (regrettably) prefer something
> that renders well on sites like GitHub or GitLab than that is readable
> as plain text.

The kinds of changes mentioned shouldn't make any difference in terms
of rendering.

> This is also related to the point I had raised earlier in the thread
> that when a package generates a manual from a Org README, using the same
> file as a README file results in too much (redundant) information.

I don't think it's our role to fix this problem: the (Non)GNU
infrastructure offers a place to put a "README" and another to put
a "manual" and it's up to the package's authors to use this as they
think is best.


        Stefan

Re: Convert README.org to plain text README while installing package

[Yuri Khan]

On Mon, 27 Jun 2022 at 17:05, Philip Kaludercic <philipk@posteo.net> wrote:

> > Maybe throwing out the badges would be a good first step ;-)
>
> Then again, this ties into the README files that look better when
> rendered on GitHub/Lab, and where it is worth considering if *not*
> using them would be of more use.

Badges do not have to be ugly in markdown source. Or, at least, they
can be much less ugly.

Let’s dissect [Eglot’s README.md][1] just as an example (only because
it has badges; otherwise, I think Eglot’s README.md is as non-ugly as
possible):

[1]: https://raw.githubusercontent.com/joaotavora/eglot/master/README.md

    [![Build status](https://github.com/joaotavora/eglot/actions/workflows/test.yml/badge.svg)](https://github.com/joaotavora/eglot/actions/workflows/test.yml)
    [![GNU ELPA](https://elpa.gnu.org/packages/eglot.svg)](https://elpa.gnu.org/packages/eglot.html)
    [![MELPA](https://melpa.org/packages/eglot-badge.svg)](https://melpa.org/#/eglot)

Here we have three source lines, each longer than 80 characters. They
are not separated with blank lines, so they comprise a single
paragraph; the badges are rendered in a single line.

Each badge is a hyperlink [](); brackets contain the link text and
parentheses specify the link target URL. There is alternative syntax
where you put a short reference identifier at the place of use, and
define that reference on a separate line.

    [![Build status](https://github.com/joaotavora/eglot/actions/workflows/test.yml/badge.svg)][build]
    [![GNU ELPA](https://elpa.gnu.org/packages/eglot.svg)][elpa]
    [![MELPA](https://melpa.org/packages/eglot-badge.svg)][melpa]

    [build]: https://github.com/joaotavora/eglot/actions/workflows/test.yml
    [elpa]: https://elpa.gnu.org/packages/eglot.html
    [melpa]: https://melpa.org/#/eglot

Next, each badge’s text is actually an image ![](). Brackets contain
the alt text fallback for user agents that do not display images; and
parens specify the image URL. The same external reference syntax
works:

    [![Build status][build-badge]][build]
    [![GNU ELPA][elpa-badge]][elpa]
    [![MELPA][melpa-badge]][melpa]

    [build]: https://github.com/joaotavora/eglot/actions/workflows/test.yml
    [elpa]: https://elpa.gnu.org/packages/eglot.html
    [melpa]: https://melpa.org/#/eglot

    [build-badge]:
https://github.com/joaotavora/eglot/actions/workflows/test.yml/badge.svg
    [elpa-badge]: https://elpa.gnu.org/packages/eglot.svg
    [melpa-badge]: https://melpa.org/packages/eglot-badge.svg

Now, only one line is longer than 80 characters.

URLs specified this way can appear anywhere in the markdown source.
Eglot’s README actually uses this syntax for all other (non-badge)
hyperlinks and puts URLs at the bottom.

Re: Convert README.org to plain text README while installing package

[Philip Kaludercic]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Mon, 27 Jun 2022 at 17:05, Philip Kaludercic <philipk@posteo.net> wrote:
>
>> > Maybe throwing out the badges would be a good first step ;-)
>>
>> Then again, this ties into the README files that look better when
>> rendered on GitHub/Lab, and where it is worth considering if *not*
>> using them would be of more use.
>
> Badges do not have to be ugly in markdown source. Or, at least, they
> can be much less ugly.
>
> Let’s dissect [Eglot’s README.md][1] just as an example (only because
> it has badges; otherwise, I think Eglot’s README.md is as non-ugly as
> possible):
>

[...]

>
>     [![Build status][build-badge]][build]
>     [![GNU ELPA][elpa-badge]][elpa]
>     [![MELPA][melpa-badge]][melpa]
>
>     [build]: https://github.com/joaotavora/eglot/actions/workflows/test.yml
>     [elpa]: https://elpa.gnu.org/packages/eglot.html
>     [melpa]: https://melpa.org/#/eglot
>
>     [build-badge]:
> https://github.com/joaotavora/eglot/actions/workflows/test.yml/badge.svg
>     [elpa-badge]: https://elpa.gnu.org/packages/eglot.svg
>     [melpa-badge]: https://melpa.org/packages/eglot-badge.svg
>
> Now, only one line is longer than 80 characters.
>
> URLs specified this way can appear anywhere in the markdown source.
> Eglot’s README actually uses this syntax for all other (non-badge)
> hyperlinks and puts URLs at the bottom.

I get the impression that this trades horizontally wasted space for
vertically wasted space (setting aside that by reordering the URLs they
don't have to appear in the middle of the document).  The issue here is
that in the plain-text buffer that C-h P provides, the badges are simply
always redundant, so either they should somehow be handled by the *Help*
buffer, removed from the buffer or (as I argue) we should not display
the README that is usually targeted at a different audience, and instead
recommend to write clean, brief, informative commentary sections in the
main files.

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>>>> I think we should actively encourage using it for most types of
>>>>> documentation, certainly for README type files.
>>>>
>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).
>
>> Hear, hear!
>>
>> It also takes a relatively long time to load Org mode, so opening a
>> README.org-type file for the first time will always cause an annoying
>> pause.
>
> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.
>
> Bye,
> Tassilo

I can't understand why you are suggesting to emulate (or enable) Org
syntax highlighting in package description.  I'm again saying that the
README.org should be converted (exported) to plain text on package
installation (I won't recommend to do this server, since its probably
SaaSS and would increase server load).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Yoni Rabkin]

Tassilo Horn <tsdh@gnu.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>>>> I think we should actively encourage using it for most types of
>>>>> documentation, certainly for README type files.
>>>>
>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).

I personally don't think that org is nice, or natural. Those are
subjective statements. Relying on the fact that lots of people use org
and formatted READMEs on github and such is not a justification either;
lots and lots of people use windows...

Making Emacs look like whatever happens to be popular shouldn't be done
at the expense of existing users. Org is popular, yes, but Emacs
shouldn't be a popularity contest.

The fact that there are people, like me, asking to be able to read a
text file as a text file should be respected. I would be fine with being
able to:

(setq packages-org-to-text t)

Or similar, and never have to to be confronted with ellipses and foreign
(to me) key-bindings when I'm trying to read a few lines of text.

Even better, if I had a simple:

(org-disable-org-mode t)

From then on I won't have to worry about being forced to read plain text
through a reincarnated implementation of "hypertext" from the 80s.

First do no harm.

-- 
   "Cut your own wood and it will warm you twice"

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Hear, hear!
>
> It also takes a relatively long time to load Org mode, so opening a
> README.org-type file for the first time will always cause an annoying
> pause.

I've heard about the problem with loading multiple times and I do agree
that loading can be slow, especially when various kinds of extra
functionality is requested. However, I am wondering about the actual
numbers. How long is long?

Can you execute

(setq org-modules nil)
(benchmark-progn (find-file "/path/to/test.org"))

And share the results?

Using Emacs 28 and built-in Org, the above took
Elapsed time: 0.161496s (0.092033s in 12 GCs)
On my system.

I am not sure if it should be considered slow.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Michael Albinus <michael.albinus@gmx.de>,  Alan Mackenzie <acm@muc.de>,
>  Stefan Kangas <stefan@marxist.se>,  Tassilo Horn <tsdh@gnu.org>,  Akib
>  Azmain Turja <akib@disroot.org>,  Emacs developers <emacs-devel@gnu.org>
> Date: Sun, 05 Jun 2022 23:15:27 +0800
> 
> I've heard about the problem with loading multiple times and I do agree
> that loading can be slow, especially when various kinds of extra
> functionality is requested. However, I am wondering about the actual
> numbers. How long is long?

This:

  (benchmark-run 1 (load "org"))

takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
an unoptimized build of Emacs 29 on MS-Windows.

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> Date: Sun, 05 Jun 2022 18:22:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: luangruo@yahoo.com, michael.albinus@gmx.de, acm@muc.de,
>  stefan@marxist.se, tsdh@gnu.org, akib@disroot.org, emacs-devel@gnu.org
> 
> This:
> 
>   (benchmark-run 1 (load "org"))
> 
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

In an optimized build of Emacs 28 with native-compilation, the above
takes 0.5 sec with about 0.1 sec used up by GC.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> This:
>
>   (benchmark-run 1 (load "org"))
>
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

I tried your example on Emacs 29 and Emacs 28 (both optimized with
native-comp).

On Emacs 29 I got (0.315444431 17 0.154982815)
On Emacs 28 I got (0.17239349 11 0.088822737)

To make sure that different Org versions do not interfere, I also tried
emacs -L /path/to/org/latest/main/branch/lisp -Q

with compiled Org repo (I ran make to compile Org into .elc)

On Emacs 29 I got (0.311368927 15 0.14880696)
On Emacs 28 I got (0.133921351 10 0.076385214)

with uncompiled Org report (make clean; make autoloads):

On Emacs 29 I got (1.040857068 41 0.5445397559999999)
On Emacs 28 I got (0.5897047940000001 28 0.337317824)

Clearly, compiled Org version can be loaded fairly fast on Emacs 28 and
not so fast on Emacs 29.

For comparison, I also tried

(benchmark-run 1 (describe-variable 'emacs-version))

On Emacs 29 I got (0.076433304 4 0.030115377999999998)
On Emacs 28 I got (0.081279133 3 0.035413045000000004)

I consider the last test to represent the compiled built-in help-mode.

org-mode loading is just 2x slower than help-mode on Emacs 28.
Not so great on Emacs 29, but it is something about Emacs 29 rather than
about org-mode. Do you have any ideas?

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:41:06 +0800
> 
> org-mode loading is just 2x slower than help-mode on Emacs 28.
> Not so great on Emacs 29, but it is something about Emacs 29 rather than
> about org-mode. Do you have any ideas?

No, sorry.  Suggest to profile it to see what takes time.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
>> Date: Sun, 05 Jun 2022 23:41:06 +0800
>> 
>> org-mode loading is just 2x slower than help-mode on Emacs 28.
>> Not so great on Emacs 29, but it is something about Emacs 29 rather than
>> about org-mode. Do you have any ideas?
>
> No, sorry.  Suggest to profile it to see what takes time.

Well. I tried, but it is not terribly helpful:

         251  85% - command-execute
         251  85%  - funcall-interactively
         251  85%   - eval-last-sexp
         251  85%    - elisp--eval-last-sexp
         251  85%     - progn
         251  85%      - benchmark-call
         251  85%       - #<lambda 0x14e0668>
         251  85%        - load
         251  85%         - load-with-code-conversion
         130  44%          - require
         120  40%           - load-with-code-conversion
          70  23%            + internal-macroexpand-for-load
          30  10%            - require
          20   6%             - load-with-code-conversion
          20   6%              - require
          10   3%                 load-with-code-conversion
          10   3%             + byte-code
          10   3%             byte-code
         101  34%          + internal-macroexpand-for-load
          10   3%          + custom-declare-variable
          43  14% - ...
          33  11%    Automatic GC
          10   3%  + byte-compile-preprocess

Looks like macroexpand takes time, but I have no idea how to interpret
this.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:53:14 +0800
> 
> > No, sorry.  Suggest to profile it to see what takes time.
> 
> Well. I tried, but it is not terribly helpful:
> 
>          251  85% - command-execute
>          251  85%  - funcall-interactively
>          251  85%   - eval-last-sexp
>          251  85%    - elisp--eval-last-sexp
>          251  85%     - progn
>          251  85%      - benchmark-call
>          251  85%       - #<lambda 0x14e0668>
>          251  85%        - load
>          251  85%         - load-with-code-conversion
>          130  44%          - require
>          120  40%           - load-with-code-conversion
>           70  23%            + internal-macroexpand-for-load
>           30  10%            - require
>           20   6%             - load-with-code-conversion
>           20   6%              - require
>           10   3%                 load-with-code-conversion
>           10   3%             + byte-code
>           10   3%             byte-code
>          101  34%          + internal-macroexpand-for-load
>           10   3%          + custom-declare-variable
>           43  14% - ...
>           33  11%    Automatic GC
>           10   3%  + byte-compile-preprocess
> 
> Looks like macroexpand takes time, but I have no idea how to interpret
> this.

I think also load-with-code-conversion, which probably means we are
loading gobs of Lisp.  That could also explain why macroexpansion
takes a significant percentage.

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> Date: Sun, 05 Jun 2022 18:46:46 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: luangruo@yahoo.com, michael.albinus@gmx.de, acm@muc.de,
>  stefan@marxist.se, tsdh@gnu.org, akib@disroot.org, emacs-devel@gnu.org
> 
> > From: Ihor Radchenko <yantar92@gmail.com>
> > Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
> >   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> > Date: Sun, 05 Jun 2022 23:41:06 +0800
> > 
> > org-mode loading is just 2x slower than help-mode on Emacs 28.
> > Not so great on Emacs 29, but it is something about Emacs 29 rather than
> > about org-mode. Do you have any ideas?
> 
> No, sorry.  Suggest to profile it to see what takes time.

For some more data point, I ran

   (benchmark-run 1 (find-file "~/org/Plan.org"))

Plan.org is a 1500-line file, so not very large, and it uses only very
basic Org features: links to URLs.

The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
and 1.5 sec (0.4 in GC) in optimized Emacs 28.

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> For some more data point, I ran
>
>    (benchmark-run 1 (find-file "~/org/Plan.org"))
>
> Plan.org is a 1500-line file, so not very large, and it uses only very
> basic Org features: links to URLs.
>
> The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
> and 1.5 sec (0.4 in GC) in optimized Emacs 28.

Does setting org-modules to nil prior opening the file change much for
you?

(setq org-modules nil)
(benchmark-run 1 (find-file "~/org/Plan.org"))

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:58:40 +0800
> 
> > The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
> > and 1.5 sec (0.4 in GC) in optimized Emacs 28.
> 
> Does setting org-modules to nil prior opening the file change much for
> you?
> 
> (setq org-modules nil)
> (benchmark-run 1 (find-file "~/org/Plan.org"))

Yes, it cuts the time in more than half: only 2.0 sec out of 5.1, in
the unoptimized build of Emacs 29.

Re: Convert README.org to plain text README while installing package

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> I've heard about the problem with loading multiple times and I do
>> agree that loading can be slow, especially when various kinds of
>> extra functionality is requested. However, I am wondering about the
>> actual numbers. How long is long?
>
> This:
>
>   (benchmark-run 1 (load "org"))
>
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

Here it is "Elapsed time: 0.233381s (0.142206s in 12 GCs)" after emacs
-Q with an unoptimized but default native-compiled emacs built from the
current master on GNU/Linux (on a 8 years old notebook).

Bye,
Tassilo

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> I've heard about the problem with loading multiple times and I do agree
> that loading can be slow, especially when various kinds of extra
> functionality is requested. However, I am wondering about the actual
> numbers.  How long is long?

For me it's just the act of loading Org even for very plain org-mode
files (plain enough that they fall within the common subset covered by
`outline-mode`; the only reason I use org-mode for those files is
because it highlight and "activates" URLs).

I think it's fairly hard to quantify accurately: last time I tried to
benchmark it, the numbers (around 1s) were much lower than what I was
experiencing subjectively (I'd estimate 5s or more).  So, AFAICT, the
time to load Org mode is strongly affected by the size of the running
session or maybe by the presence of various third party packages, which
means benchmarking

    emacs --batch  --eval '(require `org)'

doesn't give a representative picture of the problem.


        Stefan

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've heard about the problem with loading multiple times and I do agree
>> that loading can be slow, especially when various kinds of extra
>> functionality is requested. However, I am wondering about the actual
>> numbers.  How long is long?
>
> I think it's fairly hard to quantify accurately: last time I tried to
> benchmark it, the numbers (around 1s) were much lower than what I was
> experiencing subjectively (I'd estimate 5s or more).  So, AFAICT, the
> time to load Org mode is strongly affected by the size of the running
> session or maybe by the presence of various third party packages, which
> means benchmarking
>
>     emacs --batch  --eval '(require `org)'
>
> doesn't give a representative picture of the problem.

I suspect that garbage collection is strongly affecting your loading
times. It is taking half of the time even with clean Emacs. I imagine
that GC sweeps are even more costly when you have a sizable allocated
memory.

Loading ol-* staff appears to generate a lot of GC runs because of
mapconcats over all the links types with O(N_links^2) strings being
allocated. I took a note of this. I am sure that memory usage can be
improved on loading.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Po Lu <luangruo@yahoo.com>,  Michael Albinus <michael.albinus@gmx.de>,
>  Alan Mackenzie <acm@muc.de>,  Stefan Kangas <stefan@marxist.se>,  Tassilo
>  Horn <tsdh@gnu.org>,  Akib Azmain Turja <akib@disroot.org>,  Emacs
>  developers <emacs-devel@gnu.org>
> Date: Wed, 08 Jun 2022 20:51:35 +0800
> 
> >     emacs --batch  --eval '(require `org)'
> >
> > doesn't give a representative picture of the problem.
> 
> I suspect that garbage collection is strongly affecting your loading
> times. It is taking half of the time even with clean Emacs. I imagine
> that GC sweeps are even more costly when you have a sizable allocated
> memory.

Which, btw, yet another problem: why does loading Org produce so much
garbage?  Can this be kept in check somehow?

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I suspect that garbage collection is strongly affecting your loading
>> times. It is taking half of the time even with clean Emacs. I imagine
>> that GC sweeps are even more costly when you have a sizable allocated
>> memory.
>
> Which, btw, yet another problem: why does loading Org produce so much
> garbage?  Can this be kept in check somehow?

AFAIU, so much garbage is produced simply because nobody looked into
memory usage. It is not the easiest thing to track in Emacs.

Can it be kept in check? Maybe. We need to identify first which parts
are taking excess memory. org-link-set-parameters appears to be one of
the bottlenecks. I do not know any others.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

>> Which, btw, yet another problem: why does loading Org produce so much
>> garbage?  Can this be kept in check somehow?
>
> AFAIU, so much garbage is produced simply because nobody looked into
> memory usage. It is not the easiest thing to track in Emacs.
>
> Can it be kept in check? Maybe. We need to identify first which parts
> are taking excess memory. org-link-set-parameters appears to be one of
> the bottlenecks. I do not know any others.

w.r.t spending time in the GC, since the GC is run based on the amount
of memory that has been allocated (as opposed to the amount of garbage
generated, for example. which we sadly can't know in advance), the "mem"
profiler (aka `M-x profilter-start RET mem RET`) can be very useful
since it does a sampling-based profiling where the "time" is measured in
number of byte allocated (or some approximation thereof).


        Stefan

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Which, btw, yet another problem: why does loading Org produce so much
>>> garbage?  Can this be kept in check somehow?
>>
>> AFAIU, so much garbage is produced simply because nobody looked into
>> memory usage. It is not the easiest thing to track in Emacs.
>>
>> Can it be kept in check? Maybe. We need to identify first which parts
>> are taking excess memory. org-link-set-parameters appears to be one of
>> the bottlenecks. I do not know any others.
>
> w.r.t spending time in the GC, since the GC is run based on the amount
> of memory that has been allocated (as opposed to the amount of garbage
> generated, for example. which we sadly can't know in advance), the "mem"
> profiler (aka `M-x profilter-start RET mem RET`) can be very useful
> since it does a sampling-based profiling where the "time" is measured in
> number of byte allocated (or some approximation thereof).

I know the 'mem profiler. It can help in some cases. That's how I
noticed org-link-set-parameter.

However, here is one recent memory profiler report:

  1,307,745,624  23%                            - org-persist-load
  1,307,726,070  23%                             - org-persist-read
  1,281,385,671  22%                              - seq-find
  1,281,385,671  22%                               - seq-do
  1,281,385,671  22%                                - mapc
  1,281,385,671  22%                                 - #<compiled 0x43febaeb8a58606>
  1,281,385,671  22%                                  - #<compiled -0x148951e07c95fd1a>
  1,281,385,671  22%                                   - run-hook-with-args-until-success
        333,712   0%                                    + org-element--cache-persist-before-read

Unless I misinterpret things, run-hook-with-args-until-success is
allocating memory. However, that hook contains a single function which
is not allocating much according to the profiler.

Moreover, I know myself that the actual memory allocation is done down
in org-perist-read when running (read (current-buffer)). I got memory
profiler wrong significant amount of time to learn not to trust it too
much.

cpu profiler is a bit more robust in this regard. At least not until I
C-g in the middle of the profiling.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Moreover, I know myself that the actual memory allocation is done down
> in org-perist-read when running (read (current-buffer)). I got memory
> profiler wrong significant amount of time to learn not to trust it too
> much.

Indeed, the memory profiler is not very reliable.  Some of that is
a conscious implementation choice (e.g. we take samples when a whole new
16KB block is allocated to be divided into several smaller objects
rather than for every object allocation).  Other problems are things for
which I don't have an explanation (and haven't spent the time to try
and track them down).

I've still found it useful to help figure out in which part of the code
most of the allocation takes place, but you need to supplement it
without your own knowledge of what the code does  :-(


        Stefan

Re: Convert README.org to plain text README while installing package

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I've still found it useful to help figure out in which part of the code
> most of the allocation takes place, but you need to supplement it
> without your own knowledge of what the code does  :-(

Unfortunately, the profiler (both cpu and mem) is particularly useless
when I try to measure loading time.

For example, my last profiler run on Emacs-29 + native-comp yielded the
following:

Elapsed time: 0.505677s (0.295615s in 23 GCs)

Memory:

     32,661,616  98% - command-execute
     32,378,659  97%  - funcall-interactively
     28,537,695  86%   - eval-last-sexp
     28,537,695  86%    - elisp--eval-last-sexp
     28,470,377  86%     - progn
     28,470,377  86%      - let
     28,469,646  86%       - progn
     26,927,800  81%        - org-mode
     18,457,233  55%         - org-load-modules-maybe
     18,113,241  54%          + require
      7,294,107  22%         - byte-code
      4,198,028  12%          + require

CPU:

         251  79% - command-execute
         230  72%  - funcall-interactively
         225  70%   - eval-last-sexp
         225  70%    - elisp--eval-last-sexp
         225  70%     - progn
         225  70%      - let
         225  70%       - progn
         218  68%        - org-mode
         158  49%         - org-load-modules-maybe
         148  46%          - require
         148  46%           - byte-code
         129  40%            - require
         129  40%             - byte-code
         119  37%              - require
         119  37%               - byte-code

Sure, loading appears to take a lot of time, but loading of which file?
The profiler gives no answer whatsoever.

I went as far as stripping all the requires from org.el and removing all
but defun/defvar/defcustom statements. Still, the loading time was
almost 0.3 sec with no easy way to tell which part is creating the
bottleneck.

In summary, I see no other way of speeding up Org loading than splitting
org.el and other big files into modules and hoping for the best.

A practical interim solution could be doing (require 'org) in init.el
Given that Org mode is compiled and loaded on Emacs startup, the loading
generally takes a 0.3-0.5 seconds, which is much better compared to 5.1
seconds reported by Eli.

Best,
Ihor

Re: Convert README.org to plain text README while installing package

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  luangruo@yahoo.com,
>   michael.albinus@gmx.de,  acm@muc.de,  stefan@marxist.se,  tsdh@gnu.org,
>   akib@disroot.org,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 10:43:44 -0400
> 
> w.r.t spending time in the GC, since the GC is run based on the amount
> of memory that has been allocated (as opposed to the amount of garbage
> generated, for example. which we sadly can't know in advance), the "mem"
> profiler (aka `M-x profilter-start RET mem RET`) can be very useful
> since it does a sampling-based profiling where the "time" is measured in
> number of byte allocated (or some approximation thereof).

Did you actually try this, and could indeed deduce where most of the
garbage is produced?

The problem with your idea, IME, is that Emacs calls memory-allocation
functions from code that has nothing to do with consing Lisp objects,
and it does that _a_lot_.  So given a "memory" profile it is not very
easy to see which of the parts are related to GC, and many times the
hot spots aren't.

But it doesn't hurt to try what you suggest, maybe we'd get a lucky
break in this case.

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tassilo.
>
> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>> Akib Azmain Turja <akib@disroot.org> writes:
>
>> Hi Akib,
>
>> > Packages on ELPA or NonGNU ELPA with README.org show pretty
>> > description both with describe-package and web browser.  But after
>> > installing a package, describe-package shows the Org source code,
>> > which contains many useless things (I mean useless for showing in the
>> > *Help* buffer).
>
>> > Isn't it possible to convert README.org to plain text README while
>> > compiling the package, possibly optionally?  Changing package--compile
>> > function should do the job.
>
>> An alternative would be to enable (parts of) org-mode in the *Help*
>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>> That looked really nice and didn't seem to cause bad effects except that
>> the [back] / [forward] buttons stopped working.
>
> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.  What you're proposing is
> a slippery slope, where ever greater portions of org mode would get
> pushed into the core, causing ever greater problems for those who do not
> use org mode.
>
> You pointed out one such problem yourself, org mode key bindings will
> take up key binding space currently used by other modes and by users.

Yeah, I don't also use Org mode very often, so enabling Org mode will be
an even bigger problem to me.

>
> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.

Writing docs in Texinfo is obviously good, but I think its too much work
for small packages.

>
>> Bye,
>> Tassilo
>
> -- 
> Alan Mackenzie (Nuremberg, Germany).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Akib Azmain Turja <akib@disroot.org> writes:
>
> Hi Akib,
>
>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>> description both with describe-package and web browser.  But after
>> installing a package, describe-package shows the Org source code,
>> which contains many useless things (I mean useless for showing in the
>> *Help* buffer).
>>
>> Isn't it possible to convert README.org to plain text README while
>> compiling the package, possibly optionally?  Changing package--compile
>> function should do the job.
>
> An alternative would be to enable (parts of) org-mode in the *Help*
> buffer.  I've just tried `M-x describe-package RET corfu RET` which
> comes with a README.org and then enabling org-mode in the *Help* buffer.
> That looked really nice and didn't seem to cause bad effects except that
> the [back] / [forward] buttons stopped working.
>
> Bye,
> Tassilo

I don't see why enabling some features of Org mode in *Help* buffer is
useful.  IMHO writing a command to "find-file" the README is better than
this.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

Akib Azmain Turja [2022-06-04 19:14:20] wrote:
> Packages on ELPA or NonGNU ELPA with README.org show pretty description
> both with describe-package and web browser.  But after installing a
> package, describe-package shows the Org source code, which contains
> many useless things (I mean useless for showing in the *Help* buffer).
>
> Isn't it possible to convert README.org to plain text README while
> compiling the package, possibly optionally?

Actually, if the package is not yet installed, you should get this
"README.org converted to plain text" (which we get by fetching it from
`elpa.(non)gnu.org`, because it's distributed separately from the
package itself).

This is related to the problem that package tarballs do not come with
a description of where we can find their README (even tho the site that
packages it does know (and uses it in the webpage it generates)) :-(


        Stefan

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Akib Azmain Turja [2022-06-04 19:14:20] wrote:
>> Packages on ELPA or NonGNU ELPA with README.org show pretty description
>> both with describe-package and web browser.  But after installing a
>> package, describe-package shows the Org source code, which contains
>> many useless things (I mean useless for showing in the *Help* buffer).
>>
>> Isn't it possible to convert README.org to plain text README while
>> compiling the package, possibly optionally?
>
> Actually, if the package is not yet installed, you should get this
> "README.org converted to plain text" (which we get by fetching it from
> `elpa.(non)gnu.org`, because it's distributed separately from the
> package itself).
>
> This is related to the problem that package tarballs do not come with
> a description of where we can find their README (even tho the site that
> packages it does know (and uses it in the webpage it generates)) :-(
>
>
>         Stefan
>

Yeah, you correctly understood my problem.  But I think the convertion
(or export) should be done while installation, not while packaging the
package (which would probably be SaaSS), for the sake of freedom.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Convert README.org to plain text README while installing package

[Stefan Monnier]

> Yeah, you correctly understood my problem.  But I think the convertion
> (or export) should be done while installation, not while packaging the
> package (which would probably be SaaSS), for the sake of freedom.

The server (elpa.gnu.org) already has the READMEorg in its "converted to
plain text form".  It could simply add it into the tarball.
It's not SaaSS.


        Stefan

Re: Convert README.org to plain text README while installing package

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yeah, you correctly understood my problem.  But I think the convertion
>> (or export) should be done while installation, not while packaging the
>> package (which would probably be SaaSS), for the sake of freedom.
>
> The server (elpa.gnu.org) already has the READMEorg in its "converted to
> plain text form".  It could simply add it into the tarball.
> It's not SaaSS.
>
>
>         Stefan
>

If its not SaaSS then its the best solution, as it won't require to
change client side (package.el) code.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]