unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Convert README.org to plain text README while installing package
@ 2022-06-04 13:14 Akib Azmain Turja
  2022-06-04 13:32 ` Tassilo Horn
  2022-06-04 16:36 ` Stefan Monnier
  0 siblings, 2 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-04 13:14 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]


Packages on ELPA or NonGNU ELPA with README.org show pretty description
both with describe-package and web browser.  But after installing a
package, describe-package shows the Org source code, which contains
many useless things (I mean useless for showing in the *Help* buffer).

Isn't it possible to convert README.org to plain text README while
compiling the package, possibly optionally?  Changing package--compile
function should do the job.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 13:14 Convert README.org to plain text README while installing package Akib Azmain Turja
@ 2022-06-04 13:32 ` Tassilo Horn
  2022-06-04 14:17   ` Alan Mackenzie
  2022-06-04 17:22   ` Akib Azmain Turja
  2022-06-04 16:36 ` Stefan Monnier
  1 sibling, 2 replies; 376+ messages in thread
From: Tassilo Horn @ 2022-06-04 13:32 UTC (permalink / raw)
  To: Akib Azmain Turja; +Cc: emacs-devel

Akib Azmain Turja <akib@disroot.org> writes:

Hi Akib,

> Packages on ELPA or NonGNU ELPA with README.org show pretty
> description both with describe-package and web browser.  But after
> installing a package, describe-package shows the Org source code,
> which contains many useless things (I mean useless for showing in the
> *Help* buffer).
>
> Isn't it possible to convert README.org to plain text README while
> compiling the package, possibly optionally?  Changing package--compile
> function should do the job.

An alternative would be to enable (parts of) org-mode in the *Help*
buffer.  I've just tried `M-x describe-package RET corfu RET` which
comes with a README.org and then enabling org-mode in the *Help* buffer.
That looked really nice and didn't seem to cause bad effects except that
the [back] / [forward] buttons stopped working.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 13:32 ` Tassilo Horn
@ 2022-06-04 14:17   ` Alan Mackenzie
  2022-06-04 14:31     ` Tassilo Horn
                       ` (2 more replies)
  2022-06-04 17:22   ` Akib Azmain Turja
  1 sibling, 3 replies; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-04 14:17 UTC (permalink / raw)
  To: Tassilo Horn; +Cc: Akib Azmain Turja, emacs-devel

Hello, Tassilo.

On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
> Akib Azmain Turja <akib@disroot.org> writes:

> Hi Akib,

> > Packages on ELPA or NonGNU ELPA with README.org show pretty
> > description both with describe-package and web browser.  But after
> > installing a package, describe-package shows the Org source code,
> > which contains many useless things (I mean useless for showing in the
> > *Help* buffer).

> > Isn't it possible to convert README.org to plain text README while
> > compiling the package, possibly optionally?  Changing package--compile
> > function should do the job.

> An alternative would be to enable (parts of) org-mode in the *Help*
> buffer.  I've just tried `M-x describe-package RET corfu RET` which
> comes with a README.org and then enabling org-mode in the *Help* buffer.
> That looked really nice and didn't seem to cause bad effects except that
> the [back] / [forward] buttons stopped working.

No, no, no, no!  Org mode is a highly complicated, obscure mode which is
NOT part of core Emacs, and mustn't become so.  What you're proposing is
a slippery slope, where ever greater portions of org mode would get
pushed into the core, causing ever greater problems for those who do not
use org mode.

You pointed out one such problem yourself, org mode key bindings will
take up key binding space currently used by other modes and by users.

Surely the solution has got to be to encourage package authors to write
plain text (or .texi) documentation, by pointing out the difficulties
the non-standard .org format creates.

> Bye,
> Tassilo

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 14:17   ` Alan Mackenzie
@ 2022-06-04 14:31     ` Tassilo Horn
  2022-06-04 17:39       ` Akib Azmain Turja
  2022-06-04 16:09     ` Stefan Kangas
  2022-06-04 17:32     ` Akib Azmain Turja
  2 siblings, 1 reply; 376+ messages in thread
From: Tassilo Horn @ 2022-06-04 14:31 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Akib Azmain Turja, emacs-devel

Don't panic, Alan! 😁

The idea would be to only activate the font-lock rules of org (or markdown-mode, if available) while somehow protecting the package headers and help-mode footer section (i.e., the buttons).

Bye,
Tassilo

04.06.2022 16:18:21 Alan Mackenzie <acm@muc.de>:

> Hello, Tassilo.
> 
> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>> Akib Azmain Turja <akib@disroot.org> writes:
> 
>> Hi Akib,
> 
>>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>>> description both with describe-package and web browser.  But after
>>> installing a package, describe-package shows the Org source code,
>>> which contains many useless things (I mean useless for showing in the
>>> *Help* buffer).
> 
>>> Isn't it possible to convert README.org to plain text README while
>>> compiling the package, possibly optionally?  Changing package--compile
>>> function should do the job.
> 
>> An alternative would be to enable (parts of) org-mode in the *Help*
>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>> That looked really nice and didn't seem to cause bad effects except that
>> the [back] / [forward] buttons stopped working.
> 
> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.  What you're proposing is
> a slippery slope, where ever greater portions of org mode would get
> pushed into the core, causing ever greater problems for those who do not
> use org mode.
> 
> You pointed out one such problem yourself, org mode key bindings will
> take up key binding space currently used by other modes and by users.
> 
> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.
> 
>> Bye,
>> Tassilo
> 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 14:17   ` Alan Mackenzie
  2022-06-04 14:31     ` Tassilo Horn
@ 2022-06-04 16:09     ` Stefan Kangas
  2022-06-04 17:27       ` Alan Mackenzie
  2022-06-04 17:32     ` Akib Azmain Turja
  2 siblings, 1 reply; 376+ messages in thread
From: Stefan Kangas @ 2022-06-04 16:09 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Tassilo Horn, Akib Azmain Turja, Emacs developers

Alan Mackenzie <acm@muc.de> writes:

> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.

AFAICT, Org-mode has been distributed with Emacs for more than five
major releases by now (approx. 15 years).

> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.

In the Emacs community, Org-mode is ubiquitous.  I think we should
actively encourage using it for most types of documentation, certainly
for README type files.

Texinfo has its strengths of course, but it is unfortunately perceived
as arcane and hard to use by many.  And Org-mode can be converted to
be read in 'M-x info'.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 13:14 Convert README.org to plain text README while installing package Akib Azmain Turja
  2022-06-04 13:32 ` Tassilo Horn
@ 2022-06-04 16:36 ` Stefan Monnier
  2022-06-05 12:29   ` Akib Azmain Turja
  1 sibling, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-04 16:36 UTC (permalink / raw)
  To: Akib Azmain Turja; +Cc: emacs-devel

Akib Azmain Turja [2022-06-04 19:14:20] wrote:
> Packages on ELPA or NonGNU ELPA with README.org show pretty description
> both with describe-package and web browser.  But after installing a
> package, describe-package shows the Org source code, which contains
> many useless things (I mean useless for showing in the *Help* buffer).
>
> Isn't it possible to convert README.org to plain text README while
> compiling the package, possibly optionally?

Actually, if the package is not yet installed, you should get this
"README.org converted to plain text" (which we get by fetching it from
`elpa.(non)gnu.org`, because it's distributed separately from the
package itself).

This is related to the problem that package tarballs do not come with
a description of where we can find their README (even tho the site that
packages it does know (and uses it in the webpage it generates)) :-(


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 13:32 ` Tassilo Horn
  2022-06-04 14:17   ` Alan Mackenzie
@ 2022-06-04 17:22   ` Akib Azmain Turja
  1 sibling, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-04 17:22 UTC (permalink / raw)
  To: Tassilo Horn; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Akib Azmain Turja <akib@disroot.org> writes:
>
> Hi Akib,
>
>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>> description both with describe-package and web browser.  But after
>> installing a package, describe-package shows the Org source code,
>> which contains many useless things (I mean useless for showing in the
>> *Help* buffer).
>>
>> Isn't it possible to convert README.org to plain text README while
>> compiling the package, possibly optionally?  Changing package--compile
>> function should do the job.
>
> An alternative would be to enable (parts of) org-mode in the *Help*
> buffer.  I've just tried `M-x describe-package RET corfu RET` which
> comes with a README.org and then enabling org-mode in the *Help* buffer.
> That looked really nice and didn't seem to cause bad effects except that
> the [back] / [forward] buttons stopped working.
>
> Bye,
> Tassilo

I don't see why enabling some features of Org mode in *Help* buffer is
useful.  IMHO writing a command to "find-file" the README is better than
this.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 16:09     ` Stefan Kangas
@ 2022-06-04 17:27       ` Alan Mackenzie
  2022-06-04 21:10         ` Stefan Kangas
  2022-06-05  8:38         ` Michael Albinus
  0 siblings, 2 replies; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-04 17:27 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: Tassilo Horn, Akib Azmain Turja, Emacs developers

Hello, Stefan.

On Sat, Jun 04, 2022 at 18:09:45 +0200, Stefan Kangas wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > No, no, no, no!  Org mode is a highly complicated, obscure mode which
> > is NOT part of core Emacs, and mustn't become so.

> AFAICT, Org-mode has been distributed with Emacs for more than five
> major releases by now (approx. 15 years).

That's a strawman.  Org mode is not part of the Emacs core.  It's not
something in the part of Emacs inevitably used by any normal proficient
user.  Neither is Gnus.  Neither is CC Mode, for that matter.  By
contrast *Help*, Info, font-lock, the package manager, a lot of other
things, and a vast selection of commands bound to keys in the global key
map, are the core.

> > Surely the solution has got to be to encourage package authors to
> > write plain text (or .texi) documentation, by pointing out the
> > difficulties the non-standard .org format creates.

> In the Emacs community, Org-mode is ubiquitous.

Only in the sense that it's distributed with each Emacs release.

> I think we should actively encourage using it for most types of
> documentation, certainly for README type files.

I disagree entirely with you.  Org mode is highly complicated, obscure
(I've never managed to get a feel for what it does), and difficult to
learn (I've never managed it).  A text file is far easier to read for
those not familiar with org.  We're talking about READMEs here.  They're
typically 20 to 100 lines long.  A text file is ideal for these.

> Texinfo has its strengths of course, but it is unfortunately perceived
> as arcane and hard to use by many.

It's moderately easy to use, somewhat difficult to learn.

> And Org-mode can be converted to be read in 'M-x info'.

You mean it _has_ to be converted to be read?

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 14:17   ` Alan Mackenzie
  2022-06-04 14:31     ` Tassilo Horn
  2022-06-04 16:09     ` Stefan Kangas
@ 2022-06-04 17:32     ` Akib Azmain Turja
  2 siblings, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-04 17:32 UTC (permalink / raw)
  To: Alan Mackenzie, Tassilo Horn; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tassilo.
>
> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>> Akib Azmain Turja <akib@disroot.org> writes:
>
>> Hi Akib,
>
>> > Packages on ELPA or NonGNU ELPA with README.org show pretty
>> > description both with describe-package and web browser.  But after
>> > installing a package, describe-package shows the Org source code,
>> > which contains many useless things (I mean useless for showing in the
>> > *Help* buffer).
>
>> > Isn't it possible to convert README.org to plain text README while
>> > compiling the package, possibly optionally?  Changing package--compile
>> > function should do the job.
>
>> An alternative would be to enable (parts of) org-mode in the *Help*
>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>> That looked really nice and didn't seem to cause bad effects except that
>> the [back] / [forward] buttons stopped working.
>
> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
> NOT part of core Emacs, and mustn't become so.  What you're proposing is
> a slippery slope, where ever greater portions of org mode would get
> pushed into the core, causing ever greater problems for those who do not
> use org mode.
>
> You pointed out one such problem yourself, org mode key bindings will
> take up key binding space currently used by other modes and by users.

Yeah, I don't also use Org mode very often, so enabling Org mode will be
an even bigger problem to me.

>
> Surely the solution has got to be to encourage package authors to write
> plain text (or .texi) documentation, by pointing out the difficulties
> the non-standard .org format creates.

Writing docs in Texinfo is obviously good, but I think its too much work
for small packages.

>
>> Bye,
>> Tassilo
>
> -- 
> Alan Mackenzie (Nuremberg, Germany).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 14:31     ` Tassilo Horn
@ 2022-06-04 17:39       ` Akib Azmain Turja
  0 siblings, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-04 17:39 UTC (permalink / raw)
  To: Tassilo Horn, Alan Mackenzie; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Don't panic, Alan! 😁
>
> The idea would be to only activate the font-lock rules of org (or markdown-mode, if available) while somehow protecting the package headers and help-mode footer section (i.e., the buttons).

But I don't like that idea.  I don't think that's a really good idea,
since the *Help* buffer is not for only describe-package, its for all
purposes.  I think converting Org (or Markdown or whatever) to plain
text should solve most (if not all) of the problems.  Then showing that
file shouldn't need any special treatment at all.

>
> Bye,
> Tassilo
>
> 04.06.2022 16:18:21 Alan Mackenzie <acm@muc.de>:
>
>> Hello, Tassilo.
>> 
>> On Sat, Jun 04, 2022 at 15:32:10 +0200, Tassilo Horn wrote:
>>> Akib Azmain Turja <akib@disroot.org> writes:
>> 
>>> Hi Akib,
>> 
>>>> Packages on ELPA or NonGNU ELPA with README.org show pretty
>>>> description both with describe-package and web browser.  But after
>>>> installing a package, describe-package shows the Org source code,
>>>> which contains many useless things (I mean useless for showing in the
>>>> *Help* buffer).
>> 
>>>> Isn't it possible to convert README.org to plain text README while
>>>> compiling the package, possibly optionally?  Changing package--compile
>>>> function should do the job.
>> 
>>> An alternative would be to enable (parts of) org-mode in the *Help*
>>> buffer.  I've just tried `M-x describe-package RET corfu RET` which
>>> comes with a README.org and then enabling org-mode in the *Help* buffer.
>>> That looked really nice and didn't seem to cause bad effects except that
>>> the [back] / [forward] buttons stopped working.
>> 
>> No, no, no, no!  Org mode is a highly complicated, obscure mode which is
>> NOT part of core Emacs, and mustn't become so.  What you're proposing is
>> a slippery slope, where ever greater portions of org mode would get
>> pushed into the core, causing ever greater problems for those who do not
>> use org mode.
>> 
>> You pointed out one such problem yourself, org mode key bindings will
>> take up key binding space currently used by other modes and by users.
>> 
>> Surely the solution has got to be to encourage package authors to write
>> plain text (or .texi) documentation, by pointing out the difficulties
>> the non-standard .org format creates.
>> 
>>> Bye,
>>> Tassilo
>> 

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 17:27       ` Alan Mackenzie
@ 2022-06-04 21:10         ` Stefan Kangas
  2022-06-05  8:38         ` Michael Albinus
  1 sibling, 0 replies; 376+ messages in thread
From: Stefan Kangas @ 2022-06-04 21:10 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Tassilo Horn, Akib Azmain Turja, Emacs developers

Alan Mackenzie <acm@muc.de> writes:

> We're talking about READMEs here.  They're
> typically 20 to 100 lines long.  A text file is ideal for these.

A text file is fine of course, but they lack useful features such as
including images (e.g. screenshots).  You would need some markup for
that, and Org-mode happens to have solid built-in support.

> > Texinfo has its strengths of course, but it is unfortunately perceived
> > as arcane and hard to use by many.
>
> It's moderately easy to use, somewhat difficult to learn.

Agreed.

> > And Org-mode can be converted to be read in 'M-x info'.
>
> You mean it _has_ to be converted to be read?

I don't think so, no.  I'm just saying that you can do what we do with
the org manual, so you get to write Org-mode, but still get to read in
'M-x info'.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 17:27       ` Alan Mackenzie
  2022-06-04 21:10         ` Stefan Kangas
@ 2022-06-05  8:38         ` Michael Albinus
  2022-06-05  8:51           ` Po Lu
  1 sibling, 1 reply; 376+ messages in thread
From: Michael Albinus @ 2022-06-05  8:38 UTC (permalink / raw)
  To: Alan Mackenzie
  Cc: Stefan Kangas, Tassilo Horn, Akib Azmain Turja, Emacs developers

Alan Mackenzie <acm@muc.de> writes:

> Hello, Stefan.

Hi,

>> I think we should actively encourage using it for most types of
>> documentation, certainly for README type files.
>
> I disagree entirely with you.  Org mode is highly complicated, obscure
> (I've never managed to get a feel for what it does), and difficult to
> learn (I've never managed it).  A text file is far easier to read for
> those not familiar with org.  We're talking about READMEs here.  They're
> typically 20 to 100 lines long.  A text file is ideal for these.

1+. README is a text file which is consulted in order to get information
about the related software. There's no rule to visit it by Emacs only.

Best regards, Michael.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05  8:38         ` Michael Albinus
@ 2022-06-05  8:51           ` Po Lu
  2022-06-05 10:26             ` Tassilo Horn
  2022-06-05 15:15             ` Ihor Radchenko
  0 siblings, 2 replies; 376+ messages in thread
From: Po Lu @ 2022-06-05  8:51 UTC (permalink / raw)
  To: Michael Albinus
  Cc: Alan Mackenzie, Stefan Kangas, Tassilo Horn, Akib Azmain Turja,
	Emacs developers

Michael Albinus <michael.albinus@gmx.de> writes:

> Alan Mackenzie <acm@muc.de> writes:
>
>> Hello, Stefan.
>
> Hi,
>
>>> I think we should actively encourage using it for most types of
>>> documentation, certainly for README type files.
>>
>> I disagree entirely with you.  Org mode is highly complicated, obscure
>> (I've never managed to get a feel for what it does), and difficult to
>> learn (I've never managed it).  A text file is far easier to read for
>> those not familiar with org.  We're talking about READMEs here.  They're
>> typically 20 to 100 lines long.  A text file is ideal for these.
>
> 1+. README is a text file which is consulted in order to get information
> about the related software. There's no rule to visit it by Emacs only.
>
> Best regards, Michael.

Hear, hear!

It also takes a relatively long time to load Org mode, so opening a
README.org-type file for the first time will always cause an annoying
pause.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05  8:51           ` Po Lu
@ 2022-06-05 10:26             ` Tassilo Horn
  2022-06-05 11:15               ` Michael Albinus
                                 ` (5 more replies)
  2022-06-05 15:15             ` Ihor Radchenko
  1 sibling, 6 replies; 376+ messages in thread
From: Tassilo Horn @ 2022-06-05 10:26 UTC (permalink / raw)
  To: Po Lu
  Cc: Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

>>>> I think we should actively encourage using it for most types of
>>>> documentation, certainly for README type files.
>>>
>>> I disagree entirely with you.  Org mode is highly complicated,
>>> obscure (I've never managed to get a feel for what it does), and
>>> difficult to learn (I've never managed it).  A text file is far
>>> easier to read for those not familiar with org.  We're talking about
>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>> is ideal for these.
>>
>> 1+. README is a text file which is consulted in order to get
>> information about the related software. There's no rule to visit it
>> by Emacs only.

People nowadays write README.org and README.md files instead of
plain-text READMEs because the packages' repositories are hosted on some
forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
those formats in a nice way.  So the choice of format is natural for
package authors.  And they frequently contain markup like headings,
bullet lists, bold, italics, and code snippets which simply look much
better than their text-only alternative (even in their
editing/non-rendered versions).

> Hear, hear!
>
> It also takes a relatively long time to load Org mode, so opening a
> README.org-type file for the first time will always cause an annoying
> pause.

That's true but wouldn't it be possible to extract the font-lock/faces
related parts in a separate file with no dependencies which can be
loaded quickly enough?  I mean, the purpose would only be to display org
and markdown READMEs with the highlighting intended by the package
author but there's no point in enabling the gazillon of other features
(from editing to time-tracking to spreadsheet computations) those
packages might provide in addition.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
@ 2022-06-05 11:15               ` Michael Albinus
  2022-06-05 16:52                 ` [External] : " Drew Adams
  2022-06-06  0:19                 ` Tim Cross
  2022-06-05 11:23               ` Ihor Radchenko
                                 ` (4 subsequent siblings)
  5 siblings, 2 replies; 376+ messages in thread
From: Michael Albinus @ 2022-06-05 11:15 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Po Lu, Alan Mackenzie, Stefan Kangas, Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

Hi Tassilo,

>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).

Not everybody uses Github/Gitlab/<you name it> infrastructure
exclusively. There are dinosaurs like me, who simply inspect tarballs
and alike.

I have no problem if there are structured README.org or README.md files
in parallel. But a README file should be plain text.

> Bye,
> Tassilo

Best regards, Michael.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
  2022-06-05 11:15               ` Michael Albinus
@ 2022-06-05 11:23               ` Ihor Radchenko
  2022-06-06  0:33                 ` Lars Ingebrigtsen
  2022-06-05 11:29               ` Lars Ingebrigtsen
                                 ` (3 subsequent siblings)
  5 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 11:23 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Po Lu, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

>> It also takes a relatively long time to load Org mode, so opening a
>> README.org-type file for the first time will always cause an annoying
>> pause.
>
> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.

We, Org devs, are working on modularizing Org.
Currently, extracting fontification is not trivial because it depends on
most of Org being loaded.

If everything goes as planned (famous last words), fontification will
only require Org parser and Org fontification library.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
  2022-06-05 11:15               ` Michael Albinus
  2022-06-05 11:23               ` Ihor Radchenko
@ 2022-06-05 11:29               ` Lars Ingebrigtsen
  2022-06-05 13:39                 ` Stefan Monnier
  2022-06-05 11:45               ` Po Lu
                                 ` (2 subsequent siblings)
  5 siblings, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-05 11:29 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Po Lu, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

> So the choice of format is natural for package authors.  And they
> frequently contain markup like headings, bullet lists, bold, italics,
> and code snippets which simply look much better than their text-only
> alternative (even in their editing/non-rendered versions).

I think most of the hostility non-Org users have towards Org files is
the experience of opening one, only seeing a couple of lines with "..."
after them and then going "AAARGH", killing the buffer, and using
`find-file-literally' on it.

But Org files are totally innocuous if you say `M-x org-show-subtree'.
package.el could totally use org files instead of text files for the
readmes if it ran `org-show-subtree' when displaying the Org readmes --
most people wouldn't even notice that they were reading an Org file
(beyond the nice fontification).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
                                 ` (2 preceding siblings ...)
  2022-06-05 11:29               ` Lars Ingebrigtsen
@ 2022-06-05 11:45               ` Po Lu
  2022-06-05 11:59                 ` Ihor Radchenko
  2022-06-05 11:59               ` Akib Azmain Turja
  2022-06-08 17:21               ` Yoni Rabkin
  5 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-05 11:45 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on
> some forge (sr.ht, github, gitlab, gitea, whatever) which supports
> rendering those formats in a nice way.  So the choice of format is
> natural for package authors.  And they frequently contain markup like
> headings, bullet lists, bold, italics, and code snippets which simply
> look much better than their text-only alternative (even in their
> editing/non-rendered versions).

But that doesn't mean all documentation should take on such a format.
People read their READMEs in Emacs, not on a forge.

> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.

That wouldn't be Org mode anymore then, just an outline with some
additional fontification rules.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
                                 ` (3 preceding siblings ...)
  2022-06-05 11:45               ` Po Lu
@ 2022-06-05 11:59               ` Akib Azmain Turja
  2022-06-08 17:21               ` Yoni Rabkin
  5 siblings, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-05 11:59 UTC (permalink / raw)
  To: Tassilo Horn, Po Lu
  Cc: Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

Tassilo Horn <tsdh@gnu.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>>>> I think we should actively encourage using it for most types of
>>>>> documentation, certainly for README type files.
>>>>
>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).
>
>> Hear, hear!
>>
>> It also takes a relatively long time to load Org mode, so opening a
>> README.org-type file for the first time will always cause an annoying
>> pause.
>
> That's true but wouldn't it be possible to extract the font-lock/faces
> related parts in a separate file with no dependencies which can be
> loaded quickly enough?  I mean, the purpose would only be to display org
> and markdown READMEs with the highlighting intended by the package
> author but there's no point in enabling the gazillon of other features
> (from editing to time-tracking to spreadsheet computations) those
> packages might provide in addition.
>
> Bye,
> Tassilo

I can't understand why you are suggesting to emulate (or enable) Org
syntax highlighting in package description.  I'm again saying that the
README.org should be converted (exported) to plain text on package
installation (I won't recommend to do this server, since its probably
SaaSS and would increase server load).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 11:45               ` Po Lu
@ 2022-06-05 11:59                 ` Ihor Radchenko
  2022-06-05 13:11                   ` Akib Azmain Turja
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 11:59 UTC (permalink / raw)
  To: Po Lu
  Cc: Tassilo Horn, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

>> That's true but wouldn't it be possible to extract the font-lock/faces
>> related parts in a separate file with no dependencies which can be
>> loaded quickly enough?  I mean, the purpose would only be to display org
>> and markdown READMEs with the highlighting intended by the package
>> author but there's no point in enabling the gazillon of other features
>> (from editing to time-tracking to spreadsheet computations) those
>> packages might provide in addition.
>
> That wouldn't be Org mode anymore then, just an outline with some
> additional fontification rules.

Isn't *Help* buffer supposed to be read-only? There is no point loading
Org editing features if editing is not going to happen.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-04 16:36 ` Stefan Monnier
@ 2022-06-05 12:29   ` Akib Azmain Turja
  2022-06-05 13:34     ` Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-05 12:29 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Akib Azmain Turja [2022-06-04 19:14:20] wrote:
>> Packages on ELPA or NonGNU ELPA with README.org show pretty description
>> both with describe-package and web browser.  But after installing a
>> package, describe-package shows the Org source code, which contains
>> many useless things (I mean useless for showing in the *Help* buffer).
>>
>> Isn't it possible to convert README.org to plain text README while
>> compiling the package, possibly optionally?
>
> Actually, if the package is not yet installed, you should get this
> "README.org converted to plain text" (which we get by fetching it from
> `elpa.(non)gnu.org`, because it's distributed separately from the
> package itself).
>
> This is related to the problem that package tarballs do not come with
> a description of where we can find their README (even tho the site that
> packages it does know (and uses it in the webpage it generates)) :-(
>
>
>         Stefan
>

Yeah, you correctly understood my problem.  But I think the convertion
(or export) should be done while installation, not while packaging the
package (which would probably be SaaSS), for the sake of freedom.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 11:59                 ` Ihor Radchenko
@ 2022-06-05 13:11                   ` Akib Azmain Turja
  2022-06-05 15:32                     ` Tassilo Horn
  2022-06-05 15:52                     ` Philip Kaludercic
  0 siblings, 2 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-05 13:11 UTC (permalink / raw)
  To: Ihor Radchenko, Po Lu
  Cc: Tassilo Horn, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>> That's true but wouldn't it be possible to extract the font-lock/faces
>>> related parts in a separate file with no dependencies which can be
>>> loaded quickly enough?  I mean, the purpose would only be to display org
>>> and markdown READMEs with the highlighting intended by the package
>>> author but there's no point in enabling the gazillon of other features
>>> (from editing to time-tracking to spreadsheet computations) those
>>> packages might provide in addition.
>>
>> That wouldn't be Org mode anymore then, just an outline with some
>> additional fontification rules.
>
> Isn't *Help* buffer supposed to be read-only? There is no point loading
> Org editing features if editing is not going to happen.
>
> Best,
> Ihor

Yes, *Help* is read only.

I can't understand why people are suggesting to Org mode in package
description.  I'm again saying that the README.org should be converted
(exported) to plain text on package installation (I won't recommend to
do this server, since its probably SaaSS and would increase server
load).

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 12:29   ` Akib Azmain Turja
@ 2022-06-05 13:34     ` Stefan Monnier
  2022-06-06  3:34       ` Akib Azmain Turja
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 13:34 UTC (permalink / raw)
  To: Akib Azmain Turja; +Cc: emacs-devel

> Yeah, you correctly understood my problem.  But I think the convertion
> (or export) should be done while installation, not while packaging the
> package (which would probably be SaaSS), for the sake of freedom.

The server (elpa.gnu.org) already has the READMEorg in its "converted to
plain text form".  It could simply add it into the tarball.
It's not SaaSS.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 11:29               ` Lars Ingebrigtsen
@ 2022-06-05 13:39                 ` Stefan Monnier
  2022-06-05 14:24                   ` Alan Mackenzie
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 13:39 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Lars Ingebrigtsen [2022-06-05 13:29:28] wrote:
> Tassilo Horn <tsdh@gnu.org> writes:
>> So the choice of format is natural for package authors.  And they
>> frequently contain markup like headings, bullet lists, bold, italics,
>> and code snippets which simply look much better than their text-only
>> alternative (even in their editing/non-rendered versions).
> I think most of the hostility non-Org users have towards Org files is
> the experience of opening one, only seeing a couple of lines with "..."
> after them and then going "AAARGH", killing the buffer, and using
> `find-file-literally' on it.

There's also the annoying delay, of course (as in "oh, so it takes
a long time to load in order to then mess it up!?")

We should take these as challenges to improve org-mode rather than as
reasons to stay away from it.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 13:39                 ` Stefan Monnier
@ 2022-06-05 14:24                   ` Alan Mackenzie
  2022-06-05 14:38                     ` Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-05 14:24 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Lars Ingebrigtsen, Tassilo Horn, Po Lu, Michael Albinus,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Hello, Stefan.

On Sun, Jun 05, 2022 at 09:39:13 -0400, Stefan Monnier wrote:
> Lars Ingebrigtsen [2022-06-05 13:29:28] wrote:
> > Tassilo Horn <tsdh@gnu.org> writes:
> >> So the choice of format is natural for package authors.  And they
> >> frequently contain markup like headings, bullet lists, bold, italics,
> >> and code snippets which simply look much better than their text-only
> >> alternative (even in their editing/non-rendered versions).
> > I think most of the hostility non-Org users have towards Org files is
> > the experience of opening one, only seeing a couple of lines with "..."
> > after them and then going "AAARGH", killing the buffer, and using
> > `find-file-literally' on it.

> There's also the annoying delay, of course (as in "oh, so it takes
> a long time to load in order to then mess it up!?")

> We should take these as challenges to improve org-mode rather than as
> reasons to stay away from it.

We should not inflict org mode on random Emacs users, e.g. by making it
necessary in order to use *Help* fully.

Org mode is a complicated, difficult to learn package which cannot be to
everybody's taste.  To see just how complicated and difficult to learn
it is, have a look at the Key index in org.info.  There are 784 entries
in that Key index.  That is surely more key bindings than I know and use
in total.  A massive number of these are bindings which don't start with
C-c C-<letter>.  Many of these will most assuredly clash with other
bindings users have set up.

Don't get me wrong, I've no objection to people liking and using a
package like org mode.  Where I draw the line is when I am myself forced
to use it as part of accessing some unrelated feature.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 14:24                   ` Alan Mackenzie
@ 2022-06-05 14:38                     ` Stefan Monnier
  0 siblings, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 14:38 UTC (permalink / raw)
  To: Alan Mackenzie
  Cc: Lars Ingebrigtsen, Tassilo Horn, Po Lu, Michael Albinus,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

> Don't get me wrong, I've no objection to people liking and using a
> package like org mode.  Where I draw the line is when I am myself forced
> to use it as part of accessing some unrelated feature.

Noone's proposing to force you to use it.  The proposition is to use
org-mode internally when displaying a README.org file so as to make the
*Help* text more pretty.  The user wouldn't need to know that Org is
being used and the same key bindings of *Help* buffers would still do
the same as before (maybe additional key-bindings would be offered,
e.g. to fold&unfold parts, of course).


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05  8:51           ` Po Lu
  2022-06-05 10:26             ` Tassilo Horn
@ 2022-06-05 15:15             ` Ihor Radchenko
  2022-06-05 15:22               ` Eli Zaretskii
  2022-06-05 17:40               ` Stefan Monnier
  1 sibling, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 15:15 UTC (permalink / raw)
  To: Po Lu
  Cc: Michael Albinus, Alan Mackenzie, Stefan Kangas, Tassilo Horn,
	Akib Azmain Turja, Emacs developers

Po Lu <luangruo@yahoo.com> writes:

> Hear, hear!
>
> It also takes a relatively long time to load Org mode, so opening a
> README.org-type file for the first time will always cause an annoying
> pause.

I've heard about the problem with loading multiple times and I do agree
that loading can be slow, especially when various kinds of extra
functionality is requested. However, I am wondering about the actual
numbers. How long is long?

Can you execute

(setq org-modules nil)
(benchmark-progn (find-file "/path/to/test.org"))

And share the results?

Using Emacs 28 and built-in Org, the above took
Elapsed time: 0.161496s (0.092033s in 12 GCs)
On my system.

I am not sure if it should be considered slow.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:15             ` Ihor Radchenko
@ 2022-06-05 15:22               ` Eli Zaretskii
  2022-06-05 15:27                 ` Eli Zaretskii
                                   ` (2 more replies)
  2022-06-05 17:40               ` Stefan Monnier
  1 sibling, 3 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 15:22 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Michael Albinus <michael.albinus@gmx.de>,  Alan Mackenzie <acm@muc.de>,
>  Stefan Kangas <stefan@marxist.se>,  Tassilo Horn <tsdh@gnu.org>,  Akib
>  Azmain Turja <akib@disroot.org>,  Emacs developers <emacs-devel@gnu.org>
> Date: Sun, 05 Jun 2022 23:15:27 +0800
> 
> I've heard about the problem with loading multiple times and I do agree
> that loading can be slow, especially when various kinds of extra
> functionality is requested. However, I am wondering about the actual
> numbers. How long is long?

This:

  (benchmark-run 1 (load "org"))

takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
an unoptimized build of Emacs 29 on MS-Windows.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:22               ` Eli Zaretskii
@ 2022-06-05 15:27                 ` Eli Zaretskii
  2022-06-05 15:41                 ` Ihor Radchenko
  2022-06-05 16:16                 ` Tassilo Horn
  2 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 15:27 UTC (permalink / raw)
  To: yantar92; +Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> Date: Sun, 05 Jun 2022 18:22:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: luangruo@yahoo.com, michael.albinus@gmx.de, acm@muc.de,
>  stefan@marxist.se, tsdh@gnu.org, akib@disroot.org, emacs-devel@gnu.org
> 
> This:
> 
>   (benchmark-run 1 (load "org"))
> 
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

In an optimized build of Emacs 28 with native-compilation, the above
takes 0.5 sec with about 0.1 sec used up by GC.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 13:11                   ` Akib Azmain Turja
@ 2022-06-05 15:32                     ` Tassilo Horn
  2022-06-05 17:46                       ` [External] : " Drew Adams
  2022-06-05 15:52                     ` Philip Kaludercic
  1 sibling, 1 reply; 376+ messages in thread
From: Tassilo Horn @ 2022-06-05 15:32 UTC (permalink / raw)
  To: Akib Azmain Turja
  Cc: Ihor Radchenko, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, emacs-devel

Akib Azmain Turja <akib@disroot.org> writes:

Hi Akib,

> I can't understand why people are suggesting to Org mode in package
> description.

I'm not sure if we are miscommunicating.  Of course, nobody should be
mandated to write his package's README as org file.  But as a matter of
fact, many package authors do it that way or in markdown simply because
both are good formats for structured text and are supported by most
forges (markdown even more so), i.e., the rendered version of the README
becomes the project's homepage without any additional work.  IMHO,
that's good: it encourages package authors to write good READMEs because
it's the very first thing a prospective user will see.

So again, I was only suggesting that if (and only if) a package declares
a :readme "README.org" / "README.md", it would be nice if it was
*displayed* as in the "native" modes.  I certainly acknowledge technical
arguments, e.g., loading the org package is too slow (right now).

Let me ask the other way round: why can't you understand that someone
thinks it's a good idea if the package description is displayed as the
author intended?

> I'm again saying that the README.org should be converted (exported) to
> plain text on package installation (I won't recommend to do this
> server, since its probably SaaSS and would increase server load).

I've just tried org-exporting the vertico README.org to a plain-text
UTF-8 version and have to confess, the result is really, really
intriguing.  So, yes, such a conversion is at least much better as the
status quo where the org syntax is displayed literally without
highlighting.  (I almost don't perceive the org/markdown syntax when
there's syntax highlighting but if there isn't, it looks annoying.)

One problem with the "conversion on installation" approach you suggest
is that this would load org, too, so the arguments against using/loading
org for display would also apply here.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:22               ` Eli Zaretskii
  2022-06-05 15:27                 ` Eli Zaretskii
@ 2022-06-05 15:41                 ` Ihor Radchenko
  2022-06-05 15:46                   ` Eli Zaretskii
  2022-06-05 16:16                 ` Tassilo Horn
  2 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 15:41 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> This:
>
>   (benchmark-run 1 (load "org"))
>
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

I tried your example on Emacs 29 and Emacs 28 (both optimized with
native-comp).

On Emacs 29 I got (0.315444431 17 0.154982815)
On Emacs 28 I got (0.17239349 11 0.088822737)

To make sure that different Org versions do not interfere, I also tried
emacs -L /path/to/org/latest/main/branch/lisp -Q

with compiled Org repo (I ran make to compile Org into .elc)

On Emacs 29 I got (0.311368927 15 0.14880696)
On Emacs 28 I got (0.133921351 10 0.076385214)

with uncompiled Org report (make clean; make autoloads):

On Emacs 29 I got (1.040857068 41 0.5445397559999999)
On Emacs 28 I got (0.5897047940000001 28 0.337317824)

Clearly, compiled Org version can be loaded fairly fast on Emacs 28 and
not so fast on Emacs 29.

For comparison, I also tried

(benchmark-run 1 (describe-variable 'emacs-version))

On Emacs 29 I got (0.076433304 4 0.030115377999999998)
On Emacs 28 I got (0.081279133 3 0.035413045000000004)

I consider the last test to represent the compiled built-in help-mode.

org-mode loading is just 2x slower than help-mode on Emacs 28.
Not so great on Emacs 29, but it is something about Emacs 29 rather than
about org-mode. Do you have any ideas?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:41                 ` Ihor Radchenko
@ 2022-06-05 15:46                   ` Eli Zaretskii
  2022-06-05 15:53                     ` Ihor Radchenko
  2022-06-05 15:53                     ` Eli Zaretskii
  0 siblings, 2 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 15:46 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:41:06 +0800
> 
> org-mode loading is just 2x slower than help-mode on Emacs 28.
> Not so great on Emacs 29, but it is something about Emacs 29 rather than
> about org-mode. Do you have any ideas?

No, sorry.  Suggest to profile it to see what takes time.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 13:11                   ` Akib Azmain Turja
  2022-06-05 15:32                     ` Tassilo Horn
@ 2022-06-05 15:52                     ` Philip Kaludercic
  2022-06-05 17:44                       ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Philip Kaludercic @ 2022-06-05 15:52 UTC (permalink / raw)
  To: Akib Azmain Turja
  Cc: Ihor Radchenko, Po Lu, Tassilo Horn, Michael Albinus,
	Alan Mackenzie, Stefan Kangas, emacs-devel

Akib Azmain Turja <akib@disroot.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>>> That's true but wouldn't it be possible to extract the font-lock/faces
>>>> related parts in a separate file with no dependencies which can be
>>>> loaded quickly enough?  I mean, the purpose would only be to display org
>>>> and markdown READMEs with the highlighting intended by the package
>>>> author but there's no point in enabling the gazillon of other features
>>>> (from editing to time-tracking to spreadsheet computations) those
>>>> packages might provide in addition.
>>>
>>> That wouldn't be Org mode anymore then, just an outline with some
>>> additional fontification rules.
>>
>> Isn't *Help* buffer supposed to be read-only? There is no point loading
>> Org editing features if editing is not going to happen.
>>
>> Best,
>> Ihor
>
> Yes, *Help* is read only.
>
> I can't understand why people are suggesting to Org mode in package
> description.  I'm again saying that the README.org should be converted
> (exported) to plain text on package installation (I won't recommend to
> do this server, since its probably SaaSS and would increase server
> load).

I don't understand how this thread diverged, this proposal seems
absolutely sensible.  Why should a package that is not installed get
converted to plain text, while when installed you see the org source
code?

From what I see this could be fixed if elpa-admin added a "README-elpa"
with the rendered text into the tarball.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:46                   ` Eli Zaretskii
@ 2022-06-05 15:53                     ` Ihor Radchenko
  2022-06-05 16:25                       ` Eli Zaretskii
  2022-06-05 15:53                     ` Eli Zaretskii
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 15:53 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
>> Date: Sun, 05 Jun 2022 23:41:06 +0800
>> 
>> org-mode loading is just 2x slower than help-mode on Emacs 28.
>> Not so great on Emacs 29, but it is something about Emacs 29 rather than
>> about org-mode. Do you have any ideas?
>
> No, sorry.  Suggest to profile it to see what takes time.

Well. I tried, but it is not terribly helpful:

         251  85% - command-execute
         251  85%  - funcall-interactively
         251  85%   - eval-last-sexp
         251  85%    - elisp--eval-last-sexp
         251  85%     - progn
         251  85%      - benchmark-call
         251  85%       - #<lambda 0x14e0668>
         251  85%        - load
         251  85%         - load-with-code-conversion
         130  44%          - require
         120  40%           - load-with-code-conversion
          70  23%            + internal-macroexpand-for-load
          30  10%            - require
          20   6%             - load-with-code-conversion
          20   6%              - require
          10   3%                 load-with-code-conversion
          10   3%             + byte-code
          10   3%             byte-code
         101  34%          + internal-macroexpand-for-load
          10   3%          + custom-declare-variable
          43  14% - ...
          33  11%    Automatic GC
          10   3%  + byte-compile-preprocess

Looks like macroexpand takes time, but I have no idea how to interpret
this.

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:46                   ` Eli Zaretskii
  2022-06-05 15:53                     ` Ihor Radchenko
@ 2022-06-05 15:53                     ` Eli Zaretskii
  2022-06-05 15:58                       ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 15:53 UTC (permalink / raw)
  To: yantar92; +Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> Date: Sun, 05 Jun 2022 18:46:46 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: luangruo@yahoo.com, michael.albinus@gmx.de, acm@muc.de,
>  stefan@marxist.se, tsdh@gnu.org, akib@disroot.org, emacs-devel@gnu.org
> 
> > From: Ihor Radchenko <yantar92@gmail.com>
> > Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
> >   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> > Date: Sun, 05 Jun 2022 23:41:06 +0800
> > 
> > org-mode loading is just 2x slower than help-mode on Emacs 28.
> > Not so great on Emacs 29, but it is something about Emacs 29 rather than
> > about org-mode. Do you have any ideas?
> 
> No, sorry.  Suggest to profile it to see what takes time.

For some more data point, I ran

   (benchmark-run 1 (find-file "~/org/Plan.org"))

Plan.org is a 1500-line file, so not very large, and it uses only very
basic Org features: links to URLs.

The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
and 1.5 sec (0.4 in GC) in optimized Emacs 28.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:53                     ` Eli Zaretskii
@ 2022-06-05 15:58                       ` Ihor Radchenko
  2022-06-05 16:27                         ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-05 15:58 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> For some more data point, I ran
>
>    (benchmark-run 1 (find-file "~/org/Plan.org"))
>
> Plan.org is a 1500-line file, so not very large, and it uses only very
> basic Org features: links to URLs.
>
> The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
> and 1.5 sec (0.4 in GC) in optimized Emacs 28.

Does setting org-modules to nil prior opening the file change much for
you?

(setq org-modules nil)
(benchmark-run 1 (find-file "~/org/Plan.org"))

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:22               ` Eli Zaretskii
  2022-06-05 15:27                 ` Eli Zaretskii
  2022-06-05 15:41                 ` Ihor Radchenko
@ 2022-06-05 16:16                 ` Tassilo Horn
  2 siblings, 0 replies; 376+ messages in thread
From: Tassilo Horn @ 2022-06-05 16:16 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: Ihor Radchenko, luangruo, michael.albinus, acm, stefan, akib,
	emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I've heard about the problem with loading multiple times and I do
>> agree that loading can be slow, especially when various kinds of
>> extra functionality is requested. However, I am wondering about the
>> actual numbers. How long is long?
>
> This:
>
>   (benchmark-run 1 (load "org"))
>
> takes around 1.7 sec here, including 1.1 sec used by GC.  This is in
> an unoptimized build of Emacs 29 on MS-Windows.

Here it is "Elapsed time: 0.233381s (0.142206s in 12 GCs)" after emacs
-Q with an unoptimized but default native-compiled emacs built from the
current master on GNU/Linux (on a 8 years old notebook).

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:53                     ` Ihor Radchenko
@ 2022-06-05 16:25                       ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 16:25 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:53:14 +0800
> 
> > No, sorry.  Suggest to profile it to see what takes time.
> 
> Well. I tried, but it is not terribly helpful:
> 
>          251  85% - command-execute
>          251  85%  - funcall-interactively
>          251  85%   - eval-last-sexp
>          251  85%    - elisp--eval-last-sexp
>          251  85%     - progn
>          251  85%      - benchmark-call
>          251  85%       - #<lambda 0x14e0668>
>          251  85%        - load
>          251  85%         - load-with-code-conversion
>          130  44%          - require
>          120  40%           - load-with-code-conversion
>           70  23%            + internal-macroexpand-for-load
>           30  10%            - require
>           20   6%             - load-with-code-conversion
>           20   6%              - require
>           10   3%                 load-with-code-conversion
>           10   3%             + byte-code
>           10   3%             byte-code
>          101  34%          + internal-macroexpand-for-load
>           10   3%          + custom-declare-variable
>           43  14% - ...
>           33  11%    Automatic GC
>           10   3%  + byte-compile-preprocess
> 
> Looks like macroexpand takes time, but I have no idea how to interpret
> this.

I think also load-with-code-conversion, which probably means we are
loading gobs of Lisp.  That could also explain why macroexpansion
takes a significant percentage.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:58                       ` Ihor Radchenko
@ 2022-06-05 16:27                         ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-05 16:27 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: luangruo@yahoo.com,  michael.albinus@gmx.de,  acm@muc.de,
>   stefan@marxist.se,  tsdh@gnu.org,  akib@disroot.org,  emacs-devel@gnu.org
> Date: Sun, 05 Jun 2022 23:58:40 +0800
> 
> > The above tool 5.1 sec (3.1 of which in GC) in non-optimized Emacs 29
> > and 1.5 sec (0.4 in GC) in optimized Emacs 28.
> 
> Does setting org-modules to nil prior opening the file change much for
> you?
> 
> (setq org-modules nil)
> (benchmark-run 1 (find-file "~/org/Plan.org"))

Yes, it cuts the time in more than half: only 2.0 sec out of 5.1, in
the unoptimized build of Emacs 29.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 11:15               ` Michael Albinus
@ 2022-06-05 16:52                 ` Drew Adams
  2022-06-06  0:19                 ` Tim Cross
  1 sibling, 0 replies; 376+ messages in thread
From: Drew Adams @ 2022-06-05 16:52 UTC (permalink / raw)
  To: Michael Albinus, Tassilo Horn
  Cc: Po Lu, Alan Mackenzie, Stefan Kangas, Akib Azmain Turja, emacs-devel

> I have no problem if there are structured README.org or README.md files
> in parallel. But a README file should be plain text.

This.  Not just either/or.

Plain-text README should be a minimal requirement.

Other formats are nice-to-have, for some contexts
& some users.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:15             ` Ihor Radchenko
  2022-06-05 15:22               ` Eli Zaretskii
@ 2022-06-05 17:40               ` Stefan Monnier
  2022-06-08 12:51                 ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 17:40 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Po Lu, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Tassilo Horn, Akib Azmain Turja, Emacs developers

> I've heard about the problem with loading multiple times and I do agree
> that loading can be slow, especially when various kinds of extra
> functionality is requested. However, I am wondering about the actual
> numbers.  How long is long?

For me it's just the act of loading Org even for very plain org-mode
files (plain enough that they fall within the common subset covered by
`outline-mode`; the only reason I use org-mode for those files is
because it highlight and "activates" URLs).

I think it's fairly hard to quantify accurately: last time I tried to
benchmark it, the numbers (around 1s) were much lower than what I was
experiencing subjectively (I'd estimate 5s or more).  So, AFAICT, the
time to load Org mode is strongly affected by the size of the running
session or maybe by the presence of various third party packages, which
means benchmarking

    emacs --batch  --eval '(require `org)'

doesn't give a representative picture of the problem.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 15:52                     ` Philip Kaludercic
@ 2022-06-05 17:44                       ` Stefan Monnier
  2022-06-06 10:35                         ` Philip Kaludercic
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 17:44 UTC (permalink / raw)
  To: Philip Kaludercic
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Tassilo Horn,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

> From what I see this could be fixed if elpa-admin added a "README-elpa"
> with the rendered text into the tarball.

I think you're right.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 15:32                     ` Tassilo Horn
@ 2022-06-05 17:46                       ` Drew Adams
  2022-06-05 17:57                         ` Tassilo Horn
  0 siblings, 1 reply; 376+ messages in thread
From: Drew Adams @ 2022-06-05 17:46 UTC (permalink / raw)
  To: Tassilo Horn, Akib Azmain Turja
  Cc: Ihor Radchenko, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, emacs-devel

> why can't you understand that someone thinks
> it's a good idea if the package description
> is displayed as the author intended?

I'd say as the _user_ intends, not the author.

Displaying a package description is for the
benefit of the user, not the author.  It's
about what's most useful for users.

Best is this, I think:

1. Require a plain-text README, as a minimum.
2. Allow other formats, with appropriate file
   extensions (e.g. .md, .org).

I see no reason for any limit on the kinds of
format, for #2.

There should be, and likely are, simple ways
to generate a plain-text README from this or
that more structured format.  Condition #1
shouldn't be an obstacle to anyone, I'd think.

> I've just tried org-exporting the vertico README.org to a plain-text
> UTF-8 version and have to confess, the result is really, really
> intriguing.  So, yes, such a conversion is at least much better as the
> status quo where the org syntax is displayed literally without
> highlighting.  (I almost don't perceive the org/markdown syntax when
> there's syntax highlighting but if there isn't, it looks annoying.)

For Org, at least, that seems to confirm my
guess that such conversion's already available.

> One problem with the "conversion on installation"
> approach you suggest is that this would load org,
> too, so the arguments against using/loading
> org for display would also apply here.

Why shouldn't a plain-text README be required
at the outset, as part of every package?

(Another possibility, if a plain-text README
that's equivalent to a structured readme isn't
provided, is to allow a plain-text README that
tells you how to convert the structured readme
to a plain-text version.  But I don't see why
plain text shouldn't just be a requirement.)



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 17:46                       ` [External] : " Drew Adams
@ 2022-06-05 17:57                         ` Tassilo Horn
  2022-06-05 20:12                           ` Stefan Monnier
  2022-06-06  0:23                           ` Drew Adams
  0 siblings, 2 replies; 376+ messages in thread
From: Tassilo Horn @ 2022-06-05 17:57 UTC (permalink / raw)
  To: Drew Adams
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Michael Albinus,
	Alan Mackenzie, Stefan Kangas, emacs-devel

Drew Adams <drew.adams@oracle.com> writes:

>> why can't you understand that someone thinks
>> it's a good idea if the package description
>> is displayed as the author intended?
>
> I'd say as the _user_ intends, not the author.

You and me are both users and it seems we have different
intentions/preferences.

> Displaying a package description is for the
> benefit of the user, not the author.  It's
> about what's most useful for users.

Why do you think that a free software package author doesn't write his
documentation for this very purpose?  Exactly for this reason she has
structured the README in sections, **emphasized the important parts**,
included usage/config examples as highlighted

#+BEGIN_SRC emacs-lisp
  (code snippets)
#+END_SRC

and added clickable links to [[helpful resources][https://...]].  Why is
it "most useful" for users to strip away parts of that?

> Best is this, I think:
>
> 1. Require a plain-text README, as a minimum.
> 2. Allow other formats, with appropriate file
>    extensions (e.g. .md, .org).
>
> I see no reason for any limit on the kinds of
> format, for #2.
>
> There should be, and likely are, simple ways
> to generate a plain-text README from this or
> that more structured format.  Condition #1
> shouldn't be an obstacle to anyone, I'd think.

If there is some automatic conversion like for org, the obstacle is just
that I as a package author would need to commit a generated file with
the danger that contributors send patches against the generated document
instead of the source.  Not a big deal but also not great.

>> I've just tried org-exporting the vertico README.org to a plain-text
>> UTF-8 version and have to confess, the result is really, really
>> intriguing.  So, yes, such a conversion is at least much better as the
>> status quo where the org syntax is displayed literally without
>> highlighting.  (I almost don't perceive the org/markdown syntax when
>> there's syntax highlighting but if there isn't, it looks annoying.)
>
> For Org, at least, that seems to confirm my
> guess that such conversion's already available.

Yes, indeed.  I haven't seen anything like that for markdown (which is
probably even more popular for READMEs than org), though.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 17:57                         ` Tassilo Horn
@ 2022-06-05 20:12                           ` Stefan Monnier
  2022-06-06  0:24                             ` Drew Adams
  2022-06-06  0:23                           ` Drew Adams
  1 sibling, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-05 20:12 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Drew Adams, Akib Azmain Turja, Ihor Radchenko, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

> Yes, indeed.  I haven't seen anything like that for markdown (which is
> probably even more popular for READMEs than org), though.

I think that's simply because Markdown (as the name suggests) is
designed such that the Markdown source can be very close to the best
possible plain text rendering, significantly reducing the motivation to
design plain text renderer.

[ That's also why I tend to prefer Markdown for small documents, where
  I tend to find Org's markup a bit too loud.  ]


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 11:15               ` Michael Albinus
  2022-06-05 16:52                 ` [External] : " Drew Adams
@ 2022-06-06  0:19                 ` Tim Cross
  2022-06-06  9:59                   ` Philip Kaludercic
                                     ` (3 more replies)
  1 sibling, 4 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-06  0:19 UTC (permalink / raw)
  To: emacs-devel


Michael Albinus <michael.albinus@gmx.de> writes:

>
> I have no problem if there are structured README.org or README.md files
> in parallel. But a README file should be plain text.
>

I've seen this mentioned multiple times in this thread and it doesn't
make sense to me. 

Org files *are* plain text. This is one of (perhaps the biggest) selling
points for org mode. They don't use any form of 'binary' data and can be
read just fine in any text editor or just using cat/less/more whatever.
They may look a little *ugly*, especially with respect to URLs, but are
still quite readable - a lot more readable than other plain text formats
such as xml or html or json etc.

I also find arguments based around org being complex and difficult to
learn to be somewhat overstated. Org is powerful and very configurable,
which means there can be a lot to learn if you want to leverage off all
it has to offer. However, like emacs, the basics are very simple and
easy to learn. 

While I'm not arguing that org should be forced upon everyone and I
would agree we need to keep potential load time issues in mind, there
seems to be lots in this thread over stating the issues and jumping to
extremes. All that seems to really be under consideration is to enable
rendering of *org files in help buffers using org font locking and
perhaps enabling folding, which could be very beneficial for large
readme files and would not matter for small ones. I also suspect this is
something which could be disabled with a simple variable setting for
those who really don't like it. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 17:57                         ` Tassilo Horn
  2022-06-05 20:12                           ` Stefan Monnier
@ 2022-06-06  0:23                           ` Drew Adams
  2022-06-06  7:20                             ` Tassilo Horn
  1 sibling, 1 reply; 376+ messages in thread
From: Drew Adams @ 2022-06-06  0:23 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Michael Albinus,
	Alan Mackenzie, Stefan Kangas, emacs-devel

All I meant there was that it's good if a
package description is written from a user point
of view - what it does for users, not just what
the author intended in writing the package.

The motivation and design underlying the package
are important, and _can_ be useful to document
(for users and others).  But most useful is what
you can do with it as a user.

> > Displaying a package description is for the
> > benefit of the user, not the author.  It's
> > about what's most useful for users.
> 
> Why do you think that a free software package author
> doesn't write his documentation for this very purpose?

I didn't presume anything about how or why every
pkg author writes their pkg doc.  I assume that
most do try to provide useful doc for users.

My point was only that the description should be
useful to users.  Whether the description "is
displayed as the author intended" is, on the
other hand, essentially a nice-to-have.  (And I,
as one user, really like to have it.)

And I was clear that authors should be able to
provide whatever display format/method they like.

But I think _also_ providing a plain-text
description of some sort (a plain README) should
be a minimal requirement - at least something
that's requested with "Please at least provide a
plain-text README."

> Exactly for this reason she has
> structured the README in sections, **emphasized
> the important parts**, included usage/config
> examples as highlighted

Absolutely nothing wrong with that - quite the
contrary.  That's to be hoped for, even expected.
No one said anything to the contrary.

> and added clickable links to [[helpful
> resources][https://...]].

No one said that authors shouldn't be able
to provide helpful links, images, and other
artifacts.  Even flashes and fireworks, if
they like.

> Why is it "most useful" for users to strip
> away parts of that?

No one said that that's the case.  Not I, at
least.  Are you looking for a straw man?

All I suggested was that authors should, at
a minimum, provide a simple, plain-text file
README.

I explicitly even said that (IMO) that file
need not say the same thing as whatever their
other, formatted README.* files say.

Why is a plain-text README desirable - useful
for everyone?  Because it requires minimal
paraphernalia to easily read it or otherwise
process it.  Lowest common denominator.  This
should be a no-brainer, I think.

> > Best is this, I think:
> >
> > 1. Require a plain-text README, as a minimum.
> > 2. Allow other formats, with appropriate file
> >    extensions (e.g. .md, .org).
> >
> > I see no reason for any limit on the kinds of
> > format, for #2.
> >
> > There should be, and likely are, simple ways
> > to generate a plain-text README from this or
> > that more structured format.  Condition #1
> > shouldn't be an obstacle to anyone, I'd think.
> 
> If there is some automatic conversion like for org,
> the obstacle is just that I as a package author
> would need to commit a generated file with the danger
> that contributors send patches against the generated
> document instead of the source.  Not a big deal but
> also not great.

As you say, neither a big deal, nor a great obstacle.
 
> > For Org, at least, that seems to confirm my
> > guess that such conversion's already available.
> 
> Yes, indeed.  I haven't seen anything like that for
> markdown (which is probably even more popular for
> READMEs than org), though.

Even producing a plain-text version by manual editing
is likely not a big deal in most cases.  That's my
guess, at least.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-05 20:12                           ` Stefan Monnier
@ 2022-06-06  0:24                             ` Drew Adams
  2022-06-06  0:48                               ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Drew Adams @ 2022-06-06  0:24 UTC (permalink / raw)
  To: Stefan Monnier, Tassilo Horn
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Michael Albinus,
	Alan Mackenzie, Stefan Kangas, emacs-devel

> > Yes, indeed.  I haven't seen anything like that for markdown (which
> > is probably even more popular for READMEs than org), though.
> 
> I think that's simply because Markdown (as the name suggests) is
> designed such that the Markdown source can be very close to the best
> possible plain text rendering, significantly reducing the motivation to
> design plain text renderer.

Yes.  And that closeness also means that it's
generally not so difficult to go from simple
plain-text without markup to Markdown, and
vice versa.

> [ That's also why I tend to prefer Markdown for small documents, where
>   I tend to find Org's markup a bit too loud. ]

Yes again.

There are lots of nice, rendered or renderable
fancy formats, including Org and Markdown, HTML,
PDF, Info/TexInfo.

Packages should, I think, be able to provide
whatever they like, as long as they also provide
some simple README.

(And again, that simple, plain-text README need
not try to say or do all that another format
might do.  IMO.)




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 11:23               ` Ihor Radchenko
@ 2022-06-06  0:33                 ` Lars Ingebrigtsen
  2022-06-06  0:41                   ` Lars Ingebrigtsen
  2022-06-06  0:59                   ` Ihor Radchenko
  0 siblings, 2 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-06  0:33 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> We, Org devs, are working on modularizing Org.
> Currently, extracting fontification is not trivial because it depends on
> most of Org being loaded.
>
> If everything goes as planned (famous last words), fontification will
> only require Org parser and Org fontification library.

Perhaps it would be a smaller project to just make a super-simple Org
parser/displayer to handle what we typically see in a README.org file?
I.e., something that does nice fontification of #+ blocks and ***
headings and =this= (etc.), and then just removes all the rest of the
markup.

I mean, it sounds like it could be a very small, lightweight library to
get to a 90% solution that would be acceptable to show in the *Help*
buffer.  And then we wouldn't have to convert anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:33                 ` Lars Ingebrigtsen
@ 2022-06-06  0:41                   ` Lars Ingebrigtsen
  2022-06-06  0:59                   ` Ihor Radchenko
  1 sibling, 0 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-06  0:41 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I mean, it sounds like it could be a very small, lightweight library to
> get to a 90% solution that would be acceptable to show in the *Help*
> buffer.  And then we wouldn't have to convert anything.

(And the same for README.md, of course.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  0:24                             ` Drew Adams
@ 2022-06-06  0:48                               ` Ihor Radchenko
  2022-06-06  1:18                                 ` Drew Adams
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-06  0:48 UTC (permalink / raw)
  To: Drew Adams
  Cc: Stefan Monnier, Tassilo Horn, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

Drew Adams <drew.adams@oracle.com> writes:

> There are lots of nice, rendered or renderable
> fancy formats, including Org and Markdown, HTML,
> PDF, Info/TexInfo.
>
> Packages should, I think, be able to provide
> whatever they like, as long as they also provide
> some simple README.
>
> (And again, that simple, plain-text README need
> not try to say or do all that another format
> might do.  IMO.)

While what you say makes sense, can we reasonably expect the package
authors to provide an extra README file for every single package? If it
was the case, this thread would simply not show up. ASCII README would
be always available.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:33                 ` Lars Ingebrigtsen
  2022-06-06  0:41                   ` Lars Ingebrigtsen
@ 2022-06-06  0:59                   ` Ihor Radchenko
  2022-06-06  1:07                     ` Lars Ingebrigtsen
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-06  0:59 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> We, Org devs, are working on modularizing Org.
>> Currently, extracting fontification is not trivial because it depends on
>> most of Org being loaded.
>>
>> If everything goes as planned (famous last words), fontification will
>> only require Org parser and Org fontification library.
>
> Perhaps it would be a smaller project to just make a super-simple Org
> parser/displayer to handle what we typically see in a README.org file?
> I.e., something that does nice fontification of #+ blocks and ***
> headings and =this= (etc.), and then just removes all the rest of the
> markup.
>
> I mean, it sounds like it could be a very small, lightweight library to
> get to a 90% solution that would be acceptable to show in the *Help*
> buffer.  And then we wouldn't have to convert anything.

In short, no.

Org is not context free, so we at least need org-element.el parser to
handle various edge cases. In particular, there are quite annoying edge
cases with fontification of links containing */- markup symbols.

One could extract a subset of allowed Org elements from org-element.el,
but it would also involve refactoring org-element--current-element,
making the whole effort a maintenance burden against future changes in
Org.

And even if that job is done, we cannot remove links from the parser.
Yet, according to the previous benchmarks in this thread, loading links
(that is what org-modules contains by default) is giving the largest
overheads.

IMHO, the proposed lightweight library will require similar efforts to
modularizing Org + loading it with some minimal variable settings (for
example, org-modules set to nil).

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:59                   ` Ihor Radchenko
@ 2022-06-06  1:07                     ` Lars Ingebrigtsen
  2022-06-06  1:33                       ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-06  1:07 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> In short, no.

Is that a challenge?  😀

> And even if that job is done, we cannot remove links from the parser.
> Yet, according to the previous benchmarks in this thread, loading links
> (that is what org-modules contains by default) is giving the largest
> overheads.

What do you mean by "links" in this context?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  0:48                               ` Ihor Radchenko
@ 2022-06-06  1:18                                 ` Drew Adams
  2022-06-06  1:38                                   ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Drew Adams @ 2022-06-06  1:18 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Stefan Monnier, Tassilo Horn, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

> > There are lots of nice, rendered or renderable
> > fancy formats, including Org and Markdown, HTML,
> > PDF, Info/TexInfo.
> >
> > Packages should, I think, be able to provide
> > whatever they like, as long as they also provide
> > some simple README.
> >
> > (And again, that simple, plain-text README need
> > not try to say or do all that another format
> > might do.  IMO.)
> 
> While what you say makes sense, can we reasonably expect the package
> authors to provide an extra README file for every single package?

I think we can.  But only if we ask for it.

> If it was the case, this thread would simply not
> show up. ASCII README would be always available.

No, not if we've never requested that it be included.

If we request it then it'll be provided more often,
I expect.
___

[ But we can't even get vanilla Emacs itself to add
  README's for Emacs releases on MS Windows anymore,
  as it says we can (in the overall Windows README).
  https://lists.gnu.org/archive/html/emacs-devel/2022-05/msg01378.html
]



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  1:07                     ` Lars Ingebrigtsen
@ 2022-06-06  1:33                       ` Ihor Radchenko
  2022-06-06  7:39                         ` Tassilo Horn
                                           ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-06  1:33 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> In short, no.
>
> Is that a challenge?  😀

Sorry, I do not understand what you mean here.

>> And even if that job is done, we cannot remove links from the parser.
>> Yet, according to the previous benchmarks in this thread, loading links
>> (that is what org-modules contains by default) is giving the largest
>> overheads.
>
> What do you mean by "links" in this context?

In this specific paragraph, I was referring to file:, id:, help:, and
https: links. They (except help:) can be easily found in various README
files. Setting up links involves allocation of fairly large strings (see
org-element--set-regexps), which, I suspect, is creating a lot of load
on Emacs GC and slowing down the loading (especially when Emacs session
is already running for some time and GC must sweep across all the
available objects).

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  1:18                                 ` Drew Adams
@ 2022-06-06  1:38                                   ` Ihor Radchenko
  2022-06-06  1:41                                     ` Drew Adams
                                                       ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-06  1:38 UTC (permalink / raw)
  To: Drew Adams
  Cc: Stefan Monnier, Tassilo Horn, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

Drew Adams <drew.adams@oracle.com> writes:

>> If it was the case, this thread would simply not
>> show up. ASCII README would be always available.
>
> No, not if we've never requested that it be included.
>
> If we request it then it'll be provided more often,
> I expect.

Maybe.

As a single data point, if I were asked to provide such a README, I
would not bother writing it separately. Instead, I would simply export
my README.org to ASCII. And it would feel like an unnecessary burden,
honestly.

If I were asked to provide a README as an option, I would not bother at
all and just leave README.org.

Do you know how many ELPA packages provide README.org vs README.md vs
README in ASCII text?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  1:38                                   ` Ihor Radchenko
@ 2022-06-06  1:41                                     ` Drew Adams
  2022-06-06  1:47                                     ` Stefan Monnier
  2022-06-06  7:44                                     ` Tassilo Horn
  2 siblings, 0 replies; 376+ messages in thread
From: Drew Adams @ 2022-06-06  1:41 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Stefan Monnier, Tassilo Horn, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

> Do you know how many ELPA packages provide README.org 
> vs README.md vs README in ASCII text?

Not I.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  1:38                                   ` Ihor Radchenko
  2022-06-06  1:41                                     ` Drew Adams
@ 2022-06-06  1:47                                     ` Stefan Monnier
  2022-06-06  7:44                                     ` Tassilo Horn
  2 siblings, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-06  1:47 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-devel

> As a single data point, if I were asked to provide such a README, I
> would not bother writing it separately. Instead, I would simply export
> my README.org to ASCII. And it would feel like an unnecessary burden,
> honestly.

Don't worry, as de-facto maintainer of GNU ELPA, I'd rather discourage
that kind of duplication anyway (and elpa.gnu.org already builds the
text rendering of .org and.md READMEs anyway, so it would be perverse to
ask the authors to provide another one).


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 13:34     ` Stefan Monnier
@ 2022-06-06  3:34       ` Akib Azmain Turja
  0 siblings, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-06  3:34 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yeah, you correctly understood my problem.  But I think the convertion
>> (or export) should be done while installation, not while packaging the
>> package (which would probably be SaaSS), for the sake of freedom.
>
> The server (elpa.gnu.org) already has the READMEorg in its "converted to
> plain text form".  It could simply add it into the tarball.
> It's not SaaSS.
>
>
>         Stefan
>

If its not SaaSS then its the best solution, as it won't require to
change client side (package.el) code.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  0:23                           ` Drew Adams
@ 2022-06-06  7:20                             ` Tassilo Horn
  2022-06-06 15:10                               ` Drew Adams
  0 siblings, 1 reply; 376+ messages in thread
From: Tassilo Horn @ 2022-06-06  7:20 UTC (permalink / raw)
  To: Drew Adams
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Michael Albinus,
	Alan Mackenzie, Stefan Kangas, emacs-devel

Drew Adams <drew.adams@oracle.com> writes:

>> Why is it "most useful" for users to strip
>> away parts of that?
>
> No one said that that's the case.  Not I, at
> least.  Are you looking for a straw man?

No, but maybe I didn't understand your suggestion which sounded to me as
if you say that there should be a mandatory plain-text README to be
displayed in Emacs plus as many "rich" versions as the package author
wants.  However, those wouldn't be displayed in Emacs.  Or rather, they
would only be displayed when you find-file them inside ~/.emacs.d/elpa/
which is quite unlikely to happen (unless the README says there are also
alternative versions of the package destription available).

>> Yes, indeed.  I haven't seen anything like that for
>> markdown (which is probably even more popular for
>> READMEs than org), though.
>
> Even producing a plain-text version by manual editing
> is likely not a big deal in most cases.  That's my
> guess, at least.

Having to synchronize two identical (wrt. contents) versions of a
document manually, one with markup and one without, is annoying IMO.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  1:33                       ` Ihor Radchenko
@ 2022-06-06  7:39                         ` Tassilo Horn
  2022-06-08 12:56                           ` Ihor Radchenko
  2022-06-06 12:54                         ` Lars Ingebrigtsen
  2022-06-06 12:56                         ` Lars Ingebrigtsen
  2 siblings, 1 reply; 376+ messages in thread
From: Tassilo Horn @ 2022-06-06  7:39 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Lars Ingebrigtsen, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

>>> In short, no.
>>
>> Is that a challenge?  😀
>
> Sorry, I do not understand what you mean here.

Lars is asking if he should hack up a solution which will probably work
well enough for 95% of all existing README.org/md files, so just say
"yes"! ;-)

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  1:38                                   ` Ihor Radchenko
  2022-06-06  1:41                                     ` Drew Adams
  2022-06-06  1:47                                     ` Stefan Monnier
@ 2022-06-06  7:44                                     ` Tassilo Horn
  2022-06-06  9:34                                       ` Ihor Radchenko
  2 siblings, 1 reply; 376+ messages in thread
From: Tassilo Horn @ 2022-06-06  7:44 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Drew Adams, Stefan Monnier, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> Do you know how many ELPA packages provide README.org vs README.md vs
> README in ASCII text?

GNU ELPA: 348 packages, 7 README.org / 3 README.md (one commented out)
NonGNU ELPA: 160 packages, 9 README.org / 22 README.md (one commented out)

But those are only the packages which have explicitly specified a
:readme property.  I'm not sure but think ELPA will pick up a
README.md/org if there's no README automatically so it might not needed
to be declared in which case my stats aren't very meaningful.

I have no stats for MELPA but I guess there the ratio is even more
shifted towards org/md.

Bye,
Tassilo



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  7:44                                     ` Tassilo Horn
@ 2022-06-06  9:34                                       ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-06  9:34 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Drew Adams, Stefan Monnier, Akib Azmain Turja, Po Lu,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Do you know how many ELPA packages provide README.org vs README.md vs
>> README in ASCII text?
>
> GNU ELPA: 348 packages, 7 README.org / 3 README.md (one commented out)

There are certainly more. I have found at least 19 README.org on ELPA in
the first 70-ish packages.

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:19                 ` Tim Cross
@ 2022-06-06  9:59                   ` Philip Kaludercic
  2022-06-06 13:47                     ` Tim Cross
  2022-06-06 11:33                   ` Alan Mackenzie
                                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 376+ messages in thread
From: Philip Kaludercic @ 2022-06-06  9:59 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>>
>> I have no problem if there are structured README.org or README.md files
>> in parallel. But a README file should be plain text.
>>
>
> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me. 
>
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.
> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

I have consult installed, and if I want to read the package summary
generated by C-h P, I half of the buffer is just Org configuration:

--8<---------------cut here---------------start------------->8---
#+title: consult.el - Consulting completing-read
#+author: Daniel Mendler
#+language: en
#+export_file_name: consult.texi
#+texinfo_dir_category: Emacs misc features
#+texinfo_dir_title: Consult: (consult).
#+texinfo_dir_desc: Useful commands built on completing-read.

#+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
#+html: <a href="http://elpa.gnu.org/packages/consult.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
#+html: <a href="http://elpa.gnu.org/devel/consult.html"><img alt="GNU-devel ELPA" src="https://elpa.gnu.org/devel/consult.svg"/></a>
#+html: <a href="https://melpa.org/#/consult"><img alt="MELPA" src="https://melpa.org/packages/consult-badge.svg"/></a>
#+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA Stable" src="https://stable.melpa.org/packages/consult-badge.svg"/></a>

* Introduction
:properties:
:description: Why Consult?
:end:
#+cindex: introduction
--8<---------------cut here---------------end--------------->8---

While plain text, it is not what I am interested in at this point.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 17:44                       ` Stefan Monnier
@ 2022-06-06 10:35                         ` Philip Kaludercic
  2022-06-07 17:55                           ` Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Philip Kaludercic @ 2022-06-06 10:35 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Tassilo Horn,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From what I see this could be fixed if elpa-admin added a "README-elpa"
>> with the rendered text into the tarball.
>
> I think you're right.

I believe this should do the job:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Render-Org-documentation-in-a-plain-text-README-elpa.patch --]
[-- Type: text/x-diff, Size: 1752 bytes --]

From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Mon, 6 Jun 2022 12:34:40 +0200
Subject: [PATCH] Render Org documentation in a plain-text README-elpa file

* elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
(elpaa--write-plain-readme): Add new function.
---
 elpa-admin.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/elpa-admin.el b/elpa-admin.el
index ecf4bc90d1..60da684f1f 100644
--- a/elpa-admin.el
+++ b/elpa-admin.el
@@ -621,6 +621,7 @@ auxillary files unless TARBALL-ONLY is non-nil ."
        (elpaa--make pkg-spec dir)
        (elpaa--build-Info pkg-spec dir destdir))
      (elpaa--write-pkg-file dir pkgname metadata revision)
+     (elpaa--write-plain-readme dir pkg-spec)
      ;; FIXME: Allow renaming files or selecting a subset of the files!
      (cl-assert (not (string-match "[][*\\|?]" pkgname)))
      (cl-assert (not (string-match "[][*\\|?]" vers)))
@@ -1162,6 +1163,17 @@ Rename DIR/ to PKG-VERS/, and return the descriptor."
      nil
      pkg-file)))
 
+(defun elpaa--write-plain-readme (pkg-dir pkg-spec)
+  "Render a plain text readme from PKG-SPEC in PKG-DIR.
+This is only done if necessary, that is if the readme contents
+are not already taken to be formatted in plain text."
+  (let* ((readme-content (elpaa--get-README pkg-spec pkg-dir)))
+    (unless (eq (car readme-content ) 'text/plain)
+      (write-region
+       (elpaa--section-to-plain-text readme-content)
+       nil
+       (expand-file-name "README-elpa" pkg-dir)))))
+
 (defun elpaa-batch-generate-description-file (&rest _)
   "(Re)build the <PKG>-pkg.el file for particular packages."
   (while command-line-args-left
-- 
2.36.1


^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:19                 ` Tim Cross
  2022-06-06  9:59                   ` Philip Kaludercic
@ 2022-06-06 11:33                   ` Alan Mackenzie
  2022-06-06 12:26                     ` Akib Azmain Turja
  2022-06-06 13:57                     ` Tim Cross
  2022-06-06 16:56                   ` Michael Albinus
  2022-06-07 20:57                   ` Jean Louis
  3 siblings, 2 replies; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-06 11:33 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Hello, Tim.

On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:

> Michael Albinus <michael.albinus@gmx.de> writes:


> > I have no problem if there are structured README.org or README.md
> > files in parallel. But a README file should be plain text.

I agree with this.

> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me. 

> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.

We're reduced to arguing about the meaning of "plain text".  The way I
see it, plain text means to be read as is by the user.  In that sense,
the formats you mention below, xml, html, etc. are _not_ plain text -
they're designed for machine processing.  A typical spam email in HTML
has the human readable pieces swamped by the machine readable pieces.  I
think you're arguing that this isn't the case with org mode files, though
Philip Kaludercic pointed out an org file where this wasn't the case.

> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

And their performance in standard programs like grep, wc, etc. is that
much worse than plain text.

> I also find arguments based around org being complex and difficult to
> learn to be somewhat overstated.

There are 784 key bindings in org mode.  How can you say that this isn't
complex and difficult to learn?

> Org is powerful and very configurable, ....

That contradicts your previous statement to some extent.  Any program
which is very configurable _needs_ to be configured.

> .... which means there can be a lot to learn if you want to leverage
> off all it has to offer.

I don't want to do this.  I want to avoid org mode being loaded into my
Emacs; it fouls up my key bindings to a significant extent.  Particularly
if I just want to read a 50 line README.

> However, like emacs, the basics are very simple and easy to learn. 

I don't agree that the basics of Emacs are simple and easy to learn at
all.  That's a strong reason why other editors have become popular.  Why
should I have to spend time learning a super-complicated mode just to
read a 50 line README?

> While I'm not arguing that org should be forced upon everyone ....

If a README file is README.org, then org mode is forced upon anybody
wishing to read it in Emacs.  This is why README files should be plain
text.

> .... and I would agree we need to keep potential load time issues in
> mind, there seems to be lots in this thread over stating the issues and
> jumping to extremes.

I think org mode is an extreme, with its 784 key bindings which seem to
violate written and unwritten conventions.

> All that seems to really be under consideration is to enable rendering
> of *org files in help buffers using org font locking and perhaps
> enabling folding, which could be very beneficial for large readme files
> and would not matter for small ones. I also suspect this is something
> which could be disabled with a simple variable setting for those who
> really don't like it. 

"It" could best be avoided by having plain text README files.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 11:33                   ` Alan Mackenzie
@ 2022-06-06 12:26                     ` Akib Azmain Turja
  2022-06-06 13:57                     ` Tim Cross
  1 sibling, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-06 12:26 UTC (permalink / raw)
  To: Alan Mackenzie, Tim Cross; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 3871 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Tim.
>
> On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>
>> > I have no problem if there are structured README.org or README.md
>> > files in parallel. But a README file should be plain text.
>
> I agree with this.
>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>
> We're reduced to arguing about the meaning of "plain text".  The way I
> see it, plain text means to be read as is by the user.  In that sense,
> the formats you mention below, xml, html, etc. are _not_ plain text -
> they're designed for machine processing.  A typical spam email in HTML
> has the human readable pieces swamped by the machine readable pieces.  I
> think you're arguing that this isn't the case with org mode files, though
> Philip Kaludercic pointed out an org file where this wasn't the case.
>
>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> And their performance in standard programs like grep, wc, etc. is that
> much worse than plain text.
>
>> I also find arguments based around org being complex and difficult to
>> learn to be somewhat overstated.
>
> There are 784 key bindings in org mode.  How can you say that this isn't
> complex and difficult to learn?
>
>> Org is powerful and very configurable, ....
>
> That contradicts your previous statement to some extent.  Any program
> which is very configurable _needs_ to be configured.
>
>> .... which means there can be a lot to learn if you want to leverage
>> off all it has to offer.
>
> I don't want to do this.  I want to avoid org mode being loaded into my
> Emacs; it fouls up my key bindings to a significant extent.  Particularly
> if I just want to read a 50 line README.
>
>> However, like emacs, the basics are very simple and easy to learn. 
>
> I don't agree that the basics of Emacs are simple and easy to learn at
> all.  That's a strong reason why other editors have become popular.  Why
> should I have to spend time learning a super-complicated mode just to
> read a 50 line README?
>
>> While I'm not arguing that org should be forced upon everyone ....
>
> If a README file is README.org, then org mode is forced upon anybody
> wishing to read it in Emacs.  This is why README files should be plain
> text.
>
>> .... and I would agree we need to keep potential load time issues in
>> mind, there seems to be lots in this thread over stating the issues and
>> jumping to extremes.
>
> I think org mode is an extreme, with its 784 key bindings which seem to
> violate written and unwritten conventions.
>
>> All that seems to really be under consideration is to enable rendering
>> of *org files in help buffers using org font locking and perhaps
>> enabling folding, which could be very beneficial for large readme files
>> and would not matter for small ones. I also suspect this is something
>> which could be disabled with a simple variable setting for those who
>> really don't like it. 
>
> "It" could best be avoided by having plain text README files.
>
> -- 
> Alan Mackenzie (Nuremberg, Germany).
>

Just to add, many of those 784 keybindings of Org are DWIM or
context-dependent, which makes it even harder to learn.

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  1:33                       ` Ihor Radchenko
  2022-06-06  7:39                         ` Tassilo Horn
@ 2022-06-06 12:54                         ` Lars Ingebrigtsen
  2022-06-06 12:56                         ` Lars Ingebrigtsen
  2 siblings, 0 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-06 12:54 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> In this specific paragraph, I was referring to file:, id:, help:, and
> https: links.

So you were talking about these?

[[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]

Then I posit that it's trivial to get to a speedy 90% solution for these.

Anyway, for the people that say we should convert README.org files to
"text" and display that -- that's not a very satisfactory solution.  To
take an arbitrary README.org -- the one in eev.  The text version starts
off with:

---
  The best introductions to eev are:

  - "[How to record executable notes with eev - and how to play them
    back]": my talk at the EmacsConf 2019. Executable notes are mostly
    made of [sexp hyperlinks] and [eepitch blocks]. This talk has a
    quick explanation of sexp hyperlinks at [slides 5 and 6], a mention
---

which looks nice, but all those []'s are links, so you get a block down
below that's like:

---
[How to record executable notes with eev - and how to play them back]
<http://angg.twu.net/emacsconf2019.html>

[sexp hyperlinks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#3>

[eepitch blocks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6>
---

Which isn't very nice.  It should be displayed as:






-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  1:33                       ` Ihor Radchenko
  2022-06-06  7:39                         ` Tassilo Horn
  2022-06-06 12:54                         ` Lars Ingebrigtsen
@ 2022-06-06 12:56                         ` Lars Ingebrigtsen
  2022-06-08 12:55                           ` Ihor Radchenko
  2 siblings, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-06 12:56 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

(Sorry, I may have sent this twice.)

Ihor Radchenko <yantar92@gmail.com> writes:

> In this specific paragraph, I was referring to file:, id:, help:, and
> https: links.

So you were talking about these?

[[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]

Then I posit that it's trivial to get to a speedy 90% solution for these.

Anyway, for the people that say we should convert README.org files to
"text" and display that -- that's not a very satisfactory solution.  To
take an arbitrary README.org -- the one in eev.  The text version starts
off with:

---
  The best introductions to eev are:

  - "[How to record executable notes with eev - and how to play them
    back]": my talk at the EmacsConf 2019. Executable notes are mostly
    made of [sexp hyperlinks] and [eepitch blocks]. This talk has a
    quick explanation of sexp hyperlinks at [slides 5 and 6], a mention
---

which looks OK, but all those []'s are links, so you get a block down
below that's like:

---
[How to record executable notes with eev - and how to play them back]
<http://angg.twu.net/emacsconf2019.html>

[sexp hyperlinks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#3>

[eepitch blocks]
<http://angg.twu.net/eev-intros/find-eev-quick-intro.html#6>
---

Which isn't very nice.  It should be displayed as:


[-- Attachment #2: Type: image/png, Size: 58071 bytes --]

[-- Attachment #3: Type: text/plain, Size: 247 bytes --]


With clickable links and all.

I suspect that we can get there with not all that much effort if
somebody's interested in just doing the work.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  9:59                   ` Philip Kaludercic
@ 2022-06-06 13:47                     ` Tim Cross
  2022-06-06 14:15                       ` Philip Kaludercic
  0 siblings, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-06 13:47 UTC (permalink / raw)
  To: Philip Kaludercic; +Cc: emacs-devel


Philip Kaludercic <philipk@posteo.net> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>>
>>>
>>> I have no problem if there are structured README.org or README.md files
>>> in parallel. But a README file should be plain text.
>>>
>>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> I have consult installed, and if I want to read the package summary
> generated by C-h P, I half of the buffer is just Org configuration:
>
> #+title: consult.el - Consulting completing-read
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: consult.texi
> #+texinfo_dir_category: Emacs misc features
> #+texinfo_dir_title: Consult: (consult).
> #+texinfo_dir_desc: Useful commands built on completing-read.
>
> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
> #+html: <a href="http://elpa.gnu.org/packages/consult.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
> #+html: <a href="http://elpa.gnu.org/devel/consult.html"><img alt="GNU-devel ELPA" src="https://elpa.gnu.org/devel/consult.svg"/></a>
> #+html: <a href="https://melpa.org/#/consult"><img alt="MELPA" src="https://melpa.org/packages/consult-badge.svg"/></a>
> #+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA Stable" src="https://stable.melpa.org/packages/consult-badge.svg"/></a>
>
> * Introduction
> :properties:
> :description: Why Consult?
> :end:
> #+cindex: introduction
>
> While plain text, it is not what I am interested in at this point.

and in what seems to be typical in this thread, you have selected an
extreme example to demonstrate your point. This is definitely not a
typical README.org file - at least not from my experience. 

This is also a completely pointless debate as Stefan has already
indicated auto translation to ascii will likely be what is done anyway. 




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 11:33                   ` Alan Mackenzie
  2022-06-06 12:26                     ` Akib Azmain Turja
@ 2022-06-06 13:57                     ` Tim Cross
  2022-06-06 16:02                       ` Eli Zaretskii
                                         ` (2 more replies)
  1 sibling, 3 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-06 13:57 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: emacs-devel


Alan Mackenzie <acm@muc.de> writes:

> Hello, Tim.
>
> On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:
>
>> Michael Albinus <michael.albinus@gmx.de> writes:
>
>
>> > I have no problem if there are structured README.org or README.md
>> > files in parallel. But a README file should be plain text.
>
> I agree with this.
>
>> I've seen this mentioned multiple times in this thread and it doesn't
>> make sense to me. 
>
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode. They don't use any form of 'binary' data and can be
>> read just fine in any text editor or just using cat/less/more whatever.
>
> We're reduced to arguing about the meaning of "plain text".  The way I
> see it, plain text means to be read as is by the user.  In that sense,
> the formats you mention below, xml, html, etc. are _not_ plain text -
> they're designed for machine processing.  A typical spam email in HTML
> has the human readable pieces swamped by the machine readable pieces.  I
> think you're arguing that this isn't the case with org mode files, though
> Philip Kaludercic pointed out an org file where this wasn't the case.
>

Philip's example is an extreme case and not representative of normal
README.org files. 

>> They may look a little *ugly*, especially with respect to URLs, but are
>> still quite readable - a lot more readable than other plain text formats
>> such as xml or html or json etc.
>
> And their performance in standard programs like grep, wc, etc. is that
> much worse than plain text.


No, I disagree with this completely. Org mode markup is very simple and
rarely has any impact with regards to grep, ag or any other 'shell' tool
It is precisely why I consider it to be a plain text format  - all of
these standard plain trext processing tools work in the main just as
well as they do on any ASCII file. Again, org mode is not like XML or
HTML or other formats which do make such processing difficult. 
>
>> I also find arguments based around org being complex and difficult to
>> learn to be somewhat overstated.
>
> There are 784 key bindings in org mode.  How can you say that this isn't
> complex and difficult to learn?
>
Very simply because the vast majority of  those key bindings only come
into play when yhou use advanced features of org mode, such as the
agenda or table editing or noweb mode. If your just using basic org mode
features, very very few of those key bindings even come into play. Just
go throgh that list and remove any bindings relating to agenda mode,
table editing mode, source block modes, clock table mode, list editing
mode, etc and you end up with very few key bindings (all of which are
udner the C-c prefix, unlike your previous claim). 

>> Org is powerful and very configurable, ....
>
> That contradicts your previous statement to some extent.  Any program
> which is very configurable _needs_ to be configured.
>

ABsolute nonsense. Just because you can configure somethig doesn't
automatically mean you have to configure it. Emacs is extremely
configurable, but you can use it jsut fine without doing any
configuration at all. 

>> .... which means there can be a lot to learn if you want to leverage
>> off all it has to offer.
>
> I don't want to do this.  I want to avoid org mode being loaded into my
> Emacs; it fouls up my key bindings to a significant extent.  Particularly
> if I just want to read a 50 line README.
>

Other posts have already explained this is NOT what is being proposed
and that it would not affect your key bindings in the slightest. All
that is being propsoed here is that a read only buffer will use org mode
to format/display the content. Very simple and not the monster you
insist. 


>> However, like emacs, the basics are very simple and easy to learn. 
>
> I don't agree that the basics of Emacs are simple and easy to learn at
> all.  That's a strong reason why other editors have become popular.  Why
> should I have to spend time learning a super-complicated mode just to
> read a 50 line README?
>

Well, basically, you don't. That has already bene explained and does not
need to be reitterated here. However, to be clear, all that is being
proposed is that org formatting is applied to the contgents of this
read-only buffer. There will not be huge numbers of
unwanted/unexptrected key bindings and there won't be a huge number of
other changes. You will likely have some additional key bindings which
may make navigating larger README files easier and that is about it. All
that is really being proposed here is org font locking and outline mode
navigation. All very simple really. 

>> While I'm not arguing that org should be forced upon everyone ....
>
> If a README file is README.org, then org mode is forced upon anybody
> wishing to read it in Emacs.  This is why README files should be plain
> text.
>
>> .... and I would agree we need to keep potential load time issues in
>> mind, there seems to be lots in this thread over stating the issues and
>> jumping to extremes.
>
> I think org mode is an extreme, with its 784 key bindings which seem to
> violate written and unwritten conventions.


That 750 key bindings is an extreme over statement and not what is being
proposed. Yet again, people jumping to extremes based on ignorance and
paranoia. Spend the time to go through the key bindings listed in the
org manual and you will soon realise that the majority of those bindings
only come into effect in specialised mdoes - none of which are relevant
to a READM.org ile (for example, all the key bindings relating to agenda
mode). 

I find it odd that someone can on one hand argue so hard against using a
mode based on the complexity and vast nubmer of key bindings and at the
same time admit they have never spent the time to learn the mode. This
seems like an arguument based on fear and uncertainty about something
you don't know rather thaa one based on fact. Perhaps look more closely
at what was actually being proposed (which was not a full org mode
situation) and spend enough time to at least udnerstand the basics
before condemnation. 
>
>> All that seems to really be under consideration is to enable rendering
>> of *org files in help buffers using org font locking and perhaps
>> enabling folding, which could be very beneficial for large readme files
>> and would not matter for small ones. I also suspect this is something
>> which could be disabled with a simple variable setting for those who
>> really don't like it. 
>
> "It" could best be avoided by having plain text README files.




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 13:47                     ` Tim Cross
@ 2022-06-06 14:15                       ` Philip Kaludercic
  0 siblings, 0 replies; 376+ messages in thread
From: Philip Kaludercic @ 2022-06-06 14:15 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> Michael Albinus <michael.albinus@gmx.de> writes:
>>>
>>>>
>>>> I have no problem if there are structured README.org or README.md files
>>>> in parallel. But a README file should be plain text.
>>>>
>>>
>>> I've seen this mentioned multiple times in this thread and it doesn't
>>> make sense to me. 
>>>
>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>> points for org mode. They don't use any form of 'binary' data and can be
>>> read just fine in any text editor or just using cat/less/more whatever.
>>> They may look a little *ugly*, especially with respect to URLs, but are
>>> still quite readable - a lot more readable than other plain text formats
>>> such as xml or html or json etc.
>>
>> I have consult installed, and if I want to read the package summary
>> generated by C-h P, I half of the buffer is just Org configuration:
>>
>> #+title: consult.el - Consulting completing-read
>> #+author: Daniel Mendler
>> #+language: en
>> #+export_file_name: consult.texi
>> #+texinfo_dir_category: Emacs misc features
>> #+texinfo_dir_title: Consult: (consult).
>> #+texinfo_dir_desc: Useful commands built on completing-read.
>>
>> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU
>> Emacs"
>> src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
>> #+html: <a href="http://elpa.gnu.org/packages/consult.html"><img
>> alt="GNU ELPA" src="https://elpa.gnu.org/packages/consult.svg"/></a>
>> #+html: <a href="http://elpa.gnu.org/devel/consult.html"><img
>> alt="GNU-devel ELPA"
>> src="https://elpa.gnu.org/devel/consult.svg"/></a>
>> #+html: <a href="https://melpa.org/#/consult"><img alt="MELPA"
>> src="https://melpa.org/packages/consult-badge.svg"/></a>
>> #+html: <a href="https://stable.melpa.org/#/consult"><img alt="MELPA
>> Stable"
>> src="https://stable.melpa.org/packages/consult-badge.svg"/></a>
>>
>> * Introduction
>> :properties:
>> :description: Why Consult?
>> :end:
>> #+cindex: introduction
>>
>> While plain text, it is not what I am interested in at this point.
>
> and in what seems to be typical in this thread, you have selected an
> extreme example to demonstrate your point. This is definitely not a
> typical README.org file - at least not from my experience. 

While true, I don't get why this is a problem.  I don't think making the
package developers simplify their READMEs would a better solution, since
they are also frequently used to generate manuals.  

As to whether or not the package description should be the same as the
manual, is a different question (I am inclined to think the package
description should be a brief summary of a few paragraphs at most).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-06  7:20                             ` Tassilo Horn
@ 2022-06-06 15:10                               ` Drew Adams
  0 siblings, 0 replies; 376+ messages in thread
From: Drew Adams @ 2022-06-06 15:10 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, emacs-devel,
	Michael Albinus, Alan Mackenzie, Stefan Kangas

> maybe I didn't understand your suggestion which
> sounded to me as if you say that there should be
> a mandatory plain-text README to be displayed in
> Emacs plus as many "rich" versions as the package
> author wants.

My suggestion is apparently moot now, as Stefan has
already decided what will/won't be done, wrt GNU ELPA.

But yes I suggested that a plain README be required.

I don't think I said anything about it being displayed.
And I'm pretty sure I didn't say anything about any
other formats _not_ being displayed.

I don't have a set opinion about what should be
displayed by default.  (I'd hope that, whatever is the
default for displaying, users could somehow control
what is actually the case.)

I did say "required".  I later changed the wording to
clarify that I meant something more like "requested
politely", e.g., Please consider supplying a plain
README.

> However, those wouldn't be displayed in Emacs.

No; see above.  I didn't mean to say, and I don't
think I said, anything about what gets displayed.

Now that you bring that up, I guess my preference
would be that package authors can control what gets
displayed by default (see above, about users being
perhaps able to override the default).

But others have brought up the question of needing
this or that tool/infrastructure, to be able to
display this or that "rich" format.  I haven't
been focused on the question of display, except to
say that it takes very little to be able to read
a simple, plain-text readme.

My request for including plain README was just to
include it, not to display it.  Make it available.

> Or rather, they would only be displayed when you
> find-file them inside ~/.emacs.d/elpa/ which is
> quite unlikely to happen (unless the README says
> there are also alternative versions of the package
> destription available).

I hadn't thought about any particular mechanisms
for making this or that format available, or
displaying one or the other format.  My suggestion
(concern) was just to try to get authors to (also)
include a plain README.

> >> Yes, indeed.  I haven't seen anything like that for
> >> markdown (which is probably even more popular for
> >> READMEs than org), though.
> >
> > Even producing a plain-text version by manual editing
> > is likely not a big deal in most cases.  That's my
> > guess, at least.
> 
> Having to synchronize two identical (wrt. contents) versions of a
> document manually, one with markup and one without, is annoying IMO.

As I said, the simple, plain README need not have
the same content as any more elaborate README.*.

Its purpose could be just to let you know something
basic about the package - perhaps even including
instructions for how to display a more elaborate
README.*.

To me, a plain README could serve as a minimal
place to start, requiring next to no paraphernalia
or infrastructure to read it.  It need not be long
or in any way complete/definitive.  It should at
least serve as minimal help to get going.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 13:57                     ` Tim Cross
@ 2022-06-06 16:02                       ` Eli Zaretskii
  2022-06-07  6:14                         ` Tim Cross
  2022-06-06 19:19                       ` Alan Mackenzie
  2022-06-07  0:43                       ` Po Lu
  2 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-06 16:02 UTC (permalink / raw)
  To: Tim Cross; +Cc: acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Jun 2022 23:57:55 +1000
> 
> > There are 784 key bindings in org mode.  How can you say that this isn't
> > complex and difficult to learn?
> >
> Very simply because the vast majority of  those key bindings only come
> into play when yhou use advanced features of org mode, such as the
> agenda or table editing or noweb mode. If your just using basic org mode
> features, very very few of those key bindings even come into play. Just
> go throgh that list and remove any bindings relating to agenda mode,
> table editing mode, source block modes, clock table mode, list editing
> mode, etc and you end up with very few key bindings (all of which are
> udner the C-c prefix, unlike your previous claim). 

How about if you do this exercise and walk us through the results?
Visit an Org file, type "C-h b", then show us the list, and tell which
bindings you think are "basic" and which are "advanced", and why you
think so.

I can tell you what I see with my very simple 1570-line Org file,
which just has some todo items organized in 4-level hierarchy:

  . ~100 keys that look "general Org" to me
  . ~30 keys that Org remaps to its s own commands, like 'yank' and
    'backward-sentence'
  . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
    bound by default, but they are there
  . ~60 more keys that I'm not sure whether they are "basic" or not,
    but they are all bound by default, just because I visited an Org
    file 

So "just" 230 key bindings.  And this is without any minor/add-on Org
mode/feature enabled, at least according to "C-h m".

Do you see something different?  Are you still saying that it's not a
lot, or that it's "based on ignorance and paranoia"?  If so, please
point out where I'm ignorant and/or paranoiac, because I'd really like
to know.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:19                 ` Tim Cross
  2022-06-06  9:59                   ` Philip Kaludercic
  2022-06-06 11:33                   ` Alan Mackenzie
@ 2022-06-06 16:56                   ` Michael Albinus
  2022-06-07 20:57                   ` Jean Louis
  3 siblings, 0 replies; 376+ messages in thread
From: Michael Albinus @ 2022-06-06 16:56 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

Hi Tim,

>> I have no problem if there are structured README.org or README.md files
>> in parallel. But a README file should be plain text.
>
> I've seen this mentioned multiple times in this thread and it doesn't
> make sense to me.
>
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode. They don't use any form of 'binary' data and can be
> read just fine in any text editor or just using cat/less/more whatever.
> They may look a little *ugly*, especially with respect to URLs, but are
> still quite readable - a lot more readable than other plain text formats
> such as xml or html or json etc.

My point in this discussion is, that package's README files are also
visited by people not using Emacs. I'm confident that Emacs is able to
present an org-mode structure nicely, but this isn't true outside the
Emacs world.

Yes, we're speaking about Emacs packages first-hand. But even they are
distributed outside the Emacs or GitHub/GitLab infrastructure. So we
need a single starting point for users, w/o org-mode dependencies.

Best regards, Michael.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 13:57                     ` Tim Cross
  2022-06-06 16:02                       ` Eli Zaretskii
@ 2022-06-06 19:19                       ` Alan Mackenzie
  2022-06-07 10:50                         ` Protesilaos Stavrou
  2022-06-07  0:43                       ` Po Lu
  2 siblings, 1 reply; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-06 19:19 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Hello, Tim.

On Mon, Jun 06, 2022 at 23:57:55 +1000, Tim Cross wrote:

> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Tim.

> > On Mon, Jun 06, 2022 at 10:19:38 +1000, Tim Cross wrote:

[ .... ]

> >> I also find arguments based around org being complex and difficult
> >> to learn to be somewhat overstated.

> > There are 784 key bindings in org mode.  How can you say that this
> > isn't complex and difficult to learn?

> Very simply because the vast majority of  those key bindings only come
> into play when yhou use advanced features of org mode, such as the
> agenda or table editing or noweb mode. If your just using basic org
> mode features, very very few of those key bindings even come into play.
> Just go throgh that list and remove any bindings relating to agenda
> mode, table editing mode, source block modes, clock table mode, list
> editing mode, etc and you end up with very few key bindings (all of
> which are udner the C-c prefix, unlike your previous claim). 

> >> Org is powerful and very configurable, ....

> > That contradicts your previous statement to some extent.  Any program
> > which is very configurable _needs_ to be configured.

> Absolute nonsense. Just because you can configure somethig doesn't
> automatically mean you have to configure it. Emacs is extremely
> configurable, but you can use it just fine without doing any
> configuration at all. 

I find emacs -Q to be all but unusable, except for testing things with.
Only with my configurations does it become (for me) usable.  I expect the
same to hold for other users.

> >> .... which means there can be a lot to learn if you want to leverage
> >> off all it has to offer.

> > I don't want to do this.  I want to avoid org mode being loaded into my
> > Emacs; it fouls up my key bindings to a significant extent.  Particularly
> > if I just want to read a 50 line README.

> Other posts have already explained this is NOT what is being proposed
> and that it would not affect your key bindings in the slightest. All
> that is being proposed here is that a read only buffer will use org mode
> to format/display the content. Very simple and not the monster you
> insist. 

"Very simple" has a habit of becoming increasingly complicated over time.

> >> However, like emacs, the basics are very simple and easy to learn. 

> > I don't agree that the basics of Emacs are simple and easy to learn at
> > all.  That's a strong reason why other editors have become popular.  Why
> > should I have to spend time learning a super-complicated mode just to
> > read a 50 line README?

> Well, basically, you don't. That has already bene explained and does not
> need to be reitterated here. However, to be clear, all that is being
> proposed is that org formatting is applied to the contents of this
> read-only buffer. There will not be huge numbers of
> unwanted/unexpected key bindings .....

For how long?  Politically, org mode is not part of the Emacs core - it
is an add on application for those that want it.  If the maintainers of
org mode wanted to push it into the Emacs core, then they would proceed,
firstly, by getting a foot in the door, then steadily increasing pressure
to add more and more until org mode was in the position, say, that Info
currently is.  This must not happen (for reasons discussed below).

> .... and there won't be a huge number of other changes. You will likely
> have some additional key bindings which may make navigating larger
> README files easier and that is about it. All that is really being
> proposed here is org font locking and outline mode navigation. All very
> simple really. 

Hmm.

> >> While I'm not arguing that org should be forced upon everyone ....

> > If a README file is README.org, then org mode is forced upon anybody
> > wishing to read it in Emacs.  This is why README files should be plain
> > text.

> >> .... and I would agree we need to keep potential load time issues in
> >> mind, there seems to be lots in this thread over stating the issues
> >> and jumping to extremes.

> > I think org mode is an extreme, with its 784 key bindings which seem
> > to violate written and unwritten conventions.

> That 750 key bindings is an extreme over statement and not what is
> being proposed. Yet again, people jumping to extremes based on
> ignorance and paranoia. Spend the time to go through the key bindings
> listed in the org manual and you will soon realise that the majority of
> those bindings only come into effect in specialised modes - none of
> which are relevant to a READM.org ile (for example, all the key
> bindings relating to agenda mode). 

I see many of these bindings as context dependent, with the name of the
function named after the key sequence, not the functionality - e.g.
org-shiftcontroldown.  That is not the way the rest of Emacs is
constructed.  I hate context dependence (except when it is properly
implemented, e.g. by major modes).

> I find it odd that someone can on one hand argue so hard against using a
> mode based on the complexity and vast nubmer of key bindings and at the
> same time admit they have never spent the time to learn the mode.

There are only so many hours in the day.  I do not "admit" not spending
the time learning it, since there is nothing shameful about this.  I
state it as a fact.  I do not want to spend/waste time on it, and it is
my fear I will be forced into it by a stealthy increase in org mode's
prevelance, if I do not protest.

> This seems like an arguument based on fear and uncertainty about
> something you don't know rather thaa one based on fact.

I put org.org into a buffer.  I saw lots of lines ending in ..., which is
fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
in a new window.  Brilliant!  Did I mention I hate context dependent
commands?  So I have to type M-x outline-show-subtree, which works.  I
read the screenful of text, then type C-S-<down> to scroll it up.
Instead of my standard scrolling command, I get an error message about
"Not at a clock log".  Org mode has stolen the key sequences
C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

And so on.  Without some serious configuration, org mode is for me
unusable.  Moving commands around key sequences would be a nightmare,
given the 784 commands (or even the 230 counted by Eli).  And moving
org-shiftcontroldown onto a different key sequence, assuming I could find
one, would make the function name meaningless.

> Perhaps look more closely at what was actually being proposed (which
> was not a full org mode situation) and spend enough time to at least
> understand the basics before condemnation. 

I see what looks like an attempt to force org mode into the Emacs core by
stealth, a bit at a time.  Already you are talking about adding
functionality for folding.

> >> All that seems to really be under consideration is to enable rendering
> >> of *org files in help buffers using org font locking and perhaps
> >> enabling folding, which could be very beneficial for large readme files
> >> and would not matter for small ones. I also suspect this is something
> >> which could be disabled with a simple variable setting for those who
> >> really don't like it. 

And then after adding folding (by some mysterious key sequence), some
bright spark will want to add something else "for large readme files".
The consequence will be large READMEs which cannot be conveniently read
without this new feature.  And so it goes on.

We already have a format in Emacs for big documentation, Info.  Info has
about 40 commands, not 230 or 784.

> > "It" could best be avoided by having plain text README files.

I think we should strongly encourage package writers to include plain
text READMEs in their packages.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 13:57                     ` Tim Cross
  2022-06-06 16:02                       ` Eli Zaretskii
  2022-06-06 19:19                       ` Alan Mackenzie
@ 2022-06-07  0:43                       ` Po Lu
  2022-06-07  6:46                         ` Kévin Le Gouguec
  2 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-07  0:43 UTC (permalink / raw)
  To: Tim Cross; +Cc: Alan Mackenzie, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> That 750 key bindings is an extreme over statement and not what is being
> proposed. Yet again, people jumping to extremes based on ignorance and
> paranoia. Spend the time to go through the key bindings listed in the
> org manual and you will soon realise that the majority of those bindings
> only come into effect in specialised mdoes - none of which are relevant
> to a READM.org ile (for example, all the key bindings relating to agenda
> mode). 
>
> I find it odd that someone can on one hand argue so hard against using a
> mode based on the complexity and vast nubmer of key bindings and at the
> same time admit they have never spent the time to learn the mode. This
> seems like an arguument based on fear and uncertainty about something
> you don't know rather thaa one based on fact. Perhaps look more closely
> at what was actually being proposed (which was not a full org mode
> situation) and spend enough time to at least udnerstand the basics
> before condemnation. 

What's more odd is expecting someone to go through the entire Org manual
in order to write and edit README files.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 16:02                       ` Eli Zaretskii
@ 2022-06-07  6:14                         ` Tim Cross
  2022-06-07 11:21                           ` Eli Zaretskii
  2022-06-07 16:02                           ` Alan Mackenzie
  0 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-07  6:14 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: acm, emacs-devel


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 06 Jun 2022 23:57:55 +1000
>> 
>> > There are 784 key bindings in org mode.  How can you say that this isn't
>> > complex and difficult to learn?
>> >
>> Very simply because the vast majority of  those key bindings only come
>> into play when yhou use advanced features of org mode, such as the
>> agenda or table editing or noweb mode. If your just using basic org mode
>> features, very very few of those key bindings even come into play. Just
>> go throgh that list and remove any bindings relating to agenda mode,
>> table editing mode, source block modes, clock table mode, list editing
>> mode, etc and you end up with very few key bindings (all of which are
>> udner the C-c prefix, unlike your previous claim). 
>
> How about if you do this exercise and walk us through the results?
> Visit an Org file, type "C-h b", then show us the list, and tell which
> bindings you think are "basic" and which are "advanced", and why you
> think so.
>
> I can tell you what I see with my very simple 1570-line Org file,
> which just has some todo items organized in 4-level hierarchy:
>
>   . ~100 keys that look "general Org" to me
>   . ~30 keys that Org remaps to its s own commands, like 'yank' and
>     'backward-sentence'
>   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
>     bound by default, but they are there
>   . ~60 more keys that I'm not sure whether they are "basic" or not,
>     but they are all bound by default, just because I visited an Org
>     file 
>
> So "just" 230 key bindings.  And this is without any minor/add-on Org
> mode/feature enabled, at least according to "C-h m".
>
> Do you see something different?  Are you still saying that it's not a
> lot, or that it's "based on ignorance and paranoia"?  If so, please
> point out where I'm ignorant and/or paranoiac, because I'd really like
> to know.

I think Alan's response has pretty much confirmed what I was saying. Alan has
made it fairly clear that his real underlying concern is about a stealthy
strategy to get org mode more intrinsically tied into Emacs and in the end,
making it so critical to normal operation that everyone will be forced to learn
org mode regardless. I don't necessarily agree with such an argument, but think
having a debate around that would be more up-front and in some ways honest
compared to what appear to be somewhat contrived alternative arguments about
excessive key bindings and added complexity. My characterisation of this all
being fear and paranoia may have been overstating things, but then again I'm not
sure. There certainly does seem to be considerable fear involved. I don't know
about paranoia, but I have not seen any evil plan to have org mode assimilate
Emacs as if it was the Borg and I've seen no discussions on the org list about
ways to get org more ingrained in Emacs or make Emacs more dependent on org.

With respect to ignorance, Alan made it quite clear he has not spent
the time or effort to learn org mode, so he is ignorant about its
operation and lacks any real understanding of its operation. He has made
assertions about the number of key bindings which are wildly over
inflated and about required levels of complexity which simply would not
be necessary with the org mode rendering of a read only buffer. In
short, he has made a lot of assumptions about both the key bindings and
complexity based on only a fairly cursory scan of the manual and very
limited experience. 

All that appears to have occurred here is that someone noted that packages often
had a readme.org file and wondered if this should be translated into just a
plain readme file. Someone else suggested we could have package descriptions
render the readme.org file using org and not require export into plain readme
file and mentioned some possible advantages this could have wrt navigation,
rendering of tables, links and images or inclusion of source code snippets,
examples etc.

From this point, some somewhat extreme claims and arguments have been thrown in.
One was Alan's claim that org would introduce 785 new key bindings, many of
which not bounmd to C-c and which would presumably either pollute the key space
or conflict with existing bindings. All I have done is point out this claim was
inaccurate and overstated due to an overly simplistic analysis of what is in the
key index of the org manual. 

it should be noted that if we were to add 'native' rendering of readme.org
files, this would likely be done with an org minor mode rather than a full blown
org mode as very little of the overall org functionality would be required in
this situation. All that would be required is org rendering support and org
support for navigation (folding/unfolding, following links and possibly
list/table navigation). All the advanced org functionality relating to babel,
noweb, todo and time management, agendas, etc have no relevance in this
scenario. LIkewise, if this functionality was provided via a minor mode, it
would also be possible for people to disable that mode and be left with just the
readme.org file, which is just plain text and quite readable.

Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Running Emacs wiht -q and opeing up an org file, I find the following 

- 236 org related key bindings - far less than 785 Of these, a number are
- actually remapping of existing bindings, so not new ones. This leaves 208.

I said that for a read only buffer, many of the org key bindings are not
relevant as they relate to features which are not pertinent to a read only org
buffer. Any bindings relating to babel, todo management, time management,
agendas etc have no relevance when reading a readme.org file. Really, the only
key bindings of any relevance are navigation related bindings (including
opening/closing folds, following links, etc). Removing all unnecessary org
bindings leave us with just 77 new bindings. Of these remaining bindings, quite
a few could also be removed, but I've left them in as I only wanted to remove
those which would clearly be of no benefit in a read only buffer rendering of a
readme.org file. In an org minor mode setup for rendering of readme.org (and
possibly other read-only rendering of org files), I estimate at least another 30
bindings could be removed because they have no real benefit or are of only
marginal benefit. Creating an org minor mode for this purpose which only
introduced 20 or so new bindings would easily be possible. The key point is that
claims of 785 new bindings arfe incorrect and misleading. Even being very
conservative, we are really talking about 10% of the number being incorrectly claimed.  
if we introduced a minor mode for this, we are talking likely less than 20 or so.

Arguments regarding this being a 'slippery slope' or 'edge of the wedge' in an
org takeover are a completely different argument and there may well be some
concerns which may need to be considered. However, if that is the real concern,
then argue based on that rather than exaggerated claims about excessive key
bindings, key binding polution/conflicts and excessive complexity.

In a similar manner, lets not jump to conclusions or throw in wild speculation
about things which have not been proposed. For example, nobody has suggested
everyone would be required to provide readme.org files, nobody has suggested you
could not just have a readme or a readme.txt file, nobody has suggested making
anyone provide any specific format. All that has been rasied is whether, when a
package has a readme.org file, should it be translated into a 'plain text'
(ASCII/UTF-8) format or whether it might be useful to have the file formatted
and nicely displayed using existing org-mode formatting capabilities. It could
even be that the ability to have 'native' rendering of readme.org files could be
n optional feature which those who want it can enable.

All of this now seems to be irrelevant as Stefan has indicated he will
just automatically convert the readme.org to a plain readme. I hope that
the original readme.org file will still be included for those who do
prefer to use org to render and navigate a well formatted readme file
which includes working and concise links, source code syntax
highlighting, well formatted examples and fast navigation. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07  0:43                       ` Po Lu
@ 2022-06-07  6:46                         ` Kévin Le Gouguec
  2022-06-07  7:53                           ` Po Lu
  0 siblings, 1 reply; 376+ messages in thread
From: Kévin Le Gouguec @ 2022-06-07  6:46 UTC (permalink / raw)
  To: Po Lu; +Cc: Tim Cross, Alan Mackenzie, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> That 750 key bindings is an extreme over statement and not what is being
>> proposed. Yet again, people jumping to extremes based on ignorance and
>> paranoia. Spend the time to go through the key bindings listed in the
>> org manual and you will soon realise that the majority of those bindings
>> only come into effect in specialised mdoes - none of which are relevant
>> to a READM.org ile (for example, all the key bindings relating to agenda
>> mode). 
>>
>> I find it odd that someone can on one hand argue so hard against using a
>> mode based on the complexity and vast nubmer of key bindings and at the
>> same time admit they have never spent the time to learn the mode. This
>> seems like an arguument based on fear and uncertainty about something
>> you don't know rather thaa one based on fact. Perhaps look more closely
>> at what was actually being proposed (which was not a full org mode
>> situation) and spend enough time to at least udnerstand the basics
>> before condemnation. 
>
> What's more odd is expecting someone to go through the entire Org manual
> in order to write and edit README files.

This thread has been a bit disheartening to read from the sidelines; as
much as I empathize with wanting to keep Emacs accessible by avoiding
code bloat and keeping cognitive load low, it is hard to know what to
reply to statements such as this one.

No one has ever had to "go through the entire Org manual in order to
write and edit README files".  Just like no one has ever had to read the
Emacs manual cover-to-cover in order to do anything with it (they _can_,
of course, but they don't _have to_).

My first years with Org were spent using (1) TAB folding (2) outline
navigation commands, and that's it.  We already have *Help* commands
which combine those features on the development branch: see C-h b with
the recently added describe-bindings-outline user option.


I don't see how enabling these section-folding and navigation features
could hurt the accessibility of package READMEs written in a structured
format.  Alan rightly points out that Org overreaches with unhelpful key
bindings, and the benchmarks posted earlier show that loading Org
wholesale for displaying READMEs would be irksome; to me that merely
suggests that whatever subset of Org features we find relevant for
_consulting_ READMEs should be better modularized.

So I would expect the discussion to focus on:

(1) what this subset should be (e.g. should it include font-locking),

(2) whether we want to add "escape hatches" for whoever struggles with
    those "rich" READMEs.

    E.g. a package-show-plain-readmes option, which could do any number
    of things depending on the underlying markup format.  For Org
    specifically, it could pass the README through the (ill-named, and
    not limited to ASCII) org-ascii-export-as-ascii, which converts Org
    markup to something more appropriate for font-lock-less viewing
    (sections underlined with equals and hyphens; inline links moved to
    their own paragraphs).

    It might even make sense for (Non)GNU ELPA to automatically run that
    function on Org READMEs in order to ensure the "bare" variant is
    always instantly available to package users.


Instead it feels like a lot of words are spent on

(1) Org is bloated and complex: yes, agreed,

(2) Package maintainers should give users the courtesy of maintaining
    two READMEs: no, I don't see a universe where that will fly.


I hope at the end of all this, we can find some middle ground between
maintainers who happen to find their productivity increased by using
Org, and maintainers who cringe at every cycle their CPU spends on Org
code.  I believe this middle ground exists.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07  6:46                         ` Kévin Le Gouguec
@ 2022-06-07  7:53                           ` Po Lu
  2022-06-07 22:15                             ` Kévin Le Gouguec
  0 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-07  7:53 UTC (permalink / raw)
  To: Kévin Le Gouguec; +Cc: Tim Cross, Alan Mackenzie, emacs-devel

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> This thread has been a bit disheartening to read from the sidelines; as
> much as I empathize with wanting to keep Emacs accessible by avoiding
> code bloat and keeping cognitive load low, it is hard to know what to
> reply to statements such as this one.
>
> No one has ever had to "go through the entire Org manual in order to
> write and edit README files".  Just like no one has ever had to read the
> Emacs manual cover-to-cover in order to do anything with it (they _can_,
> of course, but they don't _have to_).

So what was the purpose of the following statement?

>> Spend the time to go through the key bindings listed in the org
>> manual

> I don't see how enabling these section-folding and navigation features
> could hurt the accessibility of package READMEs written in a structured
> format.  Alan rightly points out that Org overreaches with unhelpful key
> bindings, and the benchmarks posted earlier show that loading Org
> wholesale for displaying READMEs would be irksome; to me that merely
> suggests that whatever subset of Org features we find relevant for
> _consulting_ READMEs should be better modularized.

IMO, packages should provide a plain text README, alongside a rich text
version in a relatively standard format like HTML.

By using HTML, it becomes readable not only in Emacs, but also any other
program that understands HTML.  HTML is also understood by many more
people than Org mode.

It's okay for package maintainers to write their documentation using
Org, and then to export it to HTML.  But please don't insist that all
documentation be written using Org mode, or that it's okay for packages
to only provide documentation in that format.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 19:19                       ` Alan Mackenzie
@ 2022-06-07 10:50                         ` Protesilaos Stavrou
  2022-06-07 12:07                           ` Philip Kaludercic
  2022-06-07 12:27                           ` Stefan Monnier
  0 siblings, 2 replies; 376+ messages in thread
From: Protesilaos Stavrou @ 2022-06-07 10:50 UTC (permalink / raw)
  To: Alan Mackenzie, Tim Cross; +Cc: emacs-devel

Hello Alan, Tim, everyone,

> From: Alan Mackenzie <acm@muc.de>
> Date: Mon, 06 Jun 2022 19:19:41 +0000
>
>> > "It" could best be avoided by having plain text README files.
>
> I think we should strongly encourage package writers to include plain
> text READMEs in their packages.

As the author of several packages on GNU ELPA which use a README.org:

1. I agree with Alan's main point: we do not need Org in Help buffers.
   Neither directly nor indirectly (if we are having that discussion).

2. I write all my documentation using the .org format because it can be
   exported to HTML and Info, among others.

3. Org is efficient for large files as it can, for example, create
   persistent links/references between headings.  Those are preserved in
   .info and .html exports.  Org also has nice features, such as the
   ability to declare a buffer-local macro which expands into, say,
   "version1.2.3" or "exported on DATE" where DATE is dynamic.  There
   are some cases where such functionality is helpful.

Concerning point 2, the GNU ELPA machinery converts the Org source into
an Info file.  (Those who install the package can access it with C-h i.)
While I handle the export process into HTML.

This arrangement provides two viable options for users who (i) want to
read the documentation yet (ii) do not wish to load org.el and its
accoutrements.  A third option, in the form of a plain README, is a
useful addition, provided it is done automatically by GNU ELPA.

As far as I am concerned, the README.org is a fully fledged resource,
not a generic, "here are the absolute basics" kind of file (not that
there is anything wrong with those).  As such, writing a separate
README, instead of exporting README.org into plain text, would add to
the maintenance burden---I would rather avoid that.

All the best,
Protesilaos (or simply "Prot")

-- 
Protesilaos Stavrou
https://protesilaos.com



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07  6:14                         ` Tim Cross
@ 2022-06-07 11:21                           ` Eli Zaretskii
  2022-06-08 13:12                             ` Ihor Radchenko
  2022-06-08 19:34                             ` Convert README.org to plain text README while installing package Tim Cross
  2022-06-07 16:02                           ` Alan Mackenzie
  1 sibling, 2 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-07 11:21 UTC (permalink / raw)
  To: Tim Cross; +Cc: acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 16:14:52 +1000
> 
> > I can tell you what I see with my very simple 1570-line Org file,
> > which just has some todo items organized in 4-level hierarchy:
> >
> >   . ~100 keys that look "general Org" to me
> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
> >     'backward-sentence'
> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
> >     bound by default, but they are there
> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
> >     but they are all bound by default, just because I visited an Org
> >     file 
> >
> > So "just" 230 key bindings.  And this is without any minor/add-on Org
> > mode/feature enabled, at least according to "C-h m".
> >
> > Do you see something different?  Are you still saying that it's not a
> > lot, or that it's "based on ignorance and paranoia"?  If so, please
> > point out where I'm ignorant and/or paranoiac, because I'd really like
> > to know.
> 
> I think Alan's response has pretty much confirmed what I was saying. Alan has
> made it fairly clear that his real underlying concern is about a stealthy
> strategy to get org mode more intrinsically tied into Emacs and in the end,
> making it so critical to normal operation that everyone will be forced to learn
> org mode regardless. I don't necessarily agree with such an argument, but think
> having a debate around that would be more up-front and in some ways honest
> compared to what appear to be somewhat contrived alternative arguments about
> excessive key bindings and added complexity. My characterisation of this all
> being fear and paranoia may have been overstating things, but then again I'm not
> sure. There certainly does seem to be considerable fear involved. I don't know
> about paranoia, but I have not seen any evil plan to have org mode assimilate
> Emacs as if it was the Borg and I've seen no discussions on the org list about
> ways to get org more ingrained in Emacs or make Emacs more dependent on org.
> 
> With respect to ignorance, Alan made it quite clear he has not spent
> the time or effort to learn org mode, so he is ignorant about its
> operation and lacks any real understanding of its operation. He has made
> assertions about the number of key bindings which are wildly over
> inflated and about required levels of complexity which simply would not
> be necessary with the org mode rendering of a read only buffer. In
> short, he has made a lot of assumptions about both the key bindings and
> complexity based on only a fairly cursory scan of the manual and very
> limited experience. 
> 
> All that appears to have occurred here is that someone noted that packages often
> had a readme.org file and wondered if this should be translated into just a
> plain readme file. Someone else suggested we could have package descriptions
> render the readme.org file using org and not require export into plain readme
> file and mentioned some possible advantages this could have wrt navigation,
> rendering of tables, links and images or inclusion of source code snippets,
> examples etc.
> 
> >From this point, some somewhat extreme claims and arguments have been thrown in.
> One was Alan's claim that org would introduce 785 new key bindings, many of
> which not bounmd to C-c and which would presumably either pollute the key space
> or conflict with existing bindings. All I have done is point out this claim was
> inaccurate and overstated due to an overly simplistic analysis of what is in the
> key index of the org manual. 
> 
> it should be noted that if we were to add 'native' rendering of readme.org
> files, this would likely be done with an org minor mode rather than a full blown
> org mode as very little of the overall org functionality would be required in
> this situation. All that would be required is org rendering support and org
> support for navigation (folding/unfolding, following links and possibly
> list/table navigation). All the advanced org functionality relating to babel,
> noweb, todo and time management, agendas, etc have no relevance in this
> scenario. LIkewise, if this functionality was provided via a minor mode, it
> would also be possible for people to disable that mode and be left with just the
> readme.org file, which is just plain text and quite readable.
> 
> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

The above is completely irrelevant to my questions.

> Running Emacs wiht -q and opeing up an org file, I find the following 
> 
> - 236 org related key bindings - far less than 785 Of these, a number are
> - actually remapping of existing bindings, so not new ones. This leaves 208.

??? How is remapping "not new"?  It takes some very common Emacs
commands and redirects them to different commands.  E.g., 'open-line'
now does something quite different.  This means the user should either
go learn what the Org commands do, or be prepared to be surprised.

> I said that for a read only buffer, many of the org key bindings are not
> relevant as they relate to features which are not pertinent to a read only org
> buffer. Any bindings relating to babel, todo management, time management,
> agendas etc have no relevance when reading a readme.org file.

Then why does Org define them in that case?

> Really, the only
> key bindings of any relevance are navigation related bindings (including
> opening/closing folds, following links, etc). Removing all unnecessary org
> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
> a few could also be removed, but I've left them in as I only wanted to remove
> those which would clearly be of no benefit in a read only buffer rendering of a
> readme.org file. In an org minor mode setup for rendering of readme.org (and
> possibly other read-only rendering of org files), I estimate at least another 30
> bindings could be removed because they have no real benefit or are of only
> marginal benefit. Creating an org minor mode for this purpose which only
> introduced 20 or so new bindings would easily be possible. The key point is that
> claims of 785 new bindings arfe incorrect and misleading. Even being very
> conservative, we are really talking about 10% of the number being incorrectly claimed.  
> if we introduced a minor mode for this, we are talking likely less than 20 or so.

I'm sorry, but IMO this hand-waves away a real problem by ignoring the
problematic UX of suddenly having more than 200 keybindings defined
just because I visited a small file.  I hoped you will present some
serious analysis of the issue, and perhaps even show me where I'm
wrong in my conclusions.  Sadly, it sounds like all you wanted to do
is to prove that you are right, the facts notwithstanding.

You accuse Alan in extremism, but your own argumentation is another
example of a similar extremism -- just in the opposite direction.

This doesn't facilitate useful discussions of what I think is a real
problem.  Any unbiased observer should agree that adding 200+ key
bindings when a small file is visited is a problem that needs to be
solved.  Org is too large for such simple jobs, and should be
refactored to avoid loading everything under the sun when a README is
visited.  And we should work towards solving it, not sweep it under
the carpet.

P.S. Btw, here's one more demonstration that Org loads too much by
default:

  C-x C-f ~/org/Plan.org RET
   => "Package vc-mtn is deprecated"

Please believe me that NOTHING in my file refers or requires mtn.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 10:50                         ` Protesilaos Stavrou
@ 2022-06-07 12:07                           ` Philip Kaludercic
  2022-06-07 12:23                             ` Protesilaos Stavrou
  2022-06-07 12:27                           ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Philip Kaludercic @ 2022-06-07 12:07 UTC (permalink / raw)
  To: Protesilaos Stavrou; +Cc: Alan Mackenzie, Tim Cross, emacs-devel


Hi Prot,

Protesilaos Stavrou <info@protesilaos.com> writes:

> As far as I am concerned, the README.org is a fully fledged resource,
> not a generic, "here are the absolute basics" kind of file (not that
> there is anything wrong with those).  As such, writing a separate
> README, instead of exporting README.org into plain text, would add to
> the maintenance burden---I would rather avoid that.

Most packages already come with a commentary section.  While these
sometimes just say "read the readme" and others contain unnecessary
information like how to install the package, many are well written and
give the appropriate, initial pointers for getting started with a new
package (basic configuration recommendations, what the main entry points
are, what options to look out for).

As mentioned before in this thread, I don't think C-h P should give the
same information as the manual -- be in in org-mode or rendered as
ASCII/Unicode.  This is too much information for someone who wants to
just wants to know what does a package "foobar" do (this is especially
critical when package names are not self explanatory/generic, as was
discussed a few weeks back).

So it might make sense to instruct the ELPA build system to not use
README files, and instead just rely on the commentary section of the
main file.  If you check out the elpaa--get-README function
elpa-admin.el, you'll already see that "README.md" files are commented
out, because

        ;; Most README.md files seem to be
        ;; currently worse than the Commentary:
        ;; section :-(

which is indicative of a general problem, of how README files are used
since GitHub (I have commented previously on the issue in this thread
https://news.ycombinator.com/item?id=30336703).

Once again I would recommend publishing guidelines and recommendations
on good packaging practices on elpa.gnu.org, with tips and examples on
what a good commentary section looks like.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 12:07                           ` Philip Kaludercic
@ 2022-06-07 12:23                             ` Protesilaos Stavrou
  0 siblings, 0 replies; 376+ messages in thread
From: Protesilaos Stavrou @ 2022-06-07 12:23 UTC (permalink / raw)
  To: Philip Kaludercic; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

Hello Philip,

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Tue, 07 Jun 2022 12:07:21 +0000
>
>> As far as I am concerned, the README.org is a fully fledged resource,
>> not a generic, "here are the absolute basics" kind of file (not that
>> there is anything wrong with those).  As such, writing a separate
>> README, instead of exporting README.org into plain text, would add to
>> the maintenance burden---I would rather avoid that.
>
> Most packages already come with a commentary section.  While these
> sometimes just say "read the readme" and others contain unnecessary
> information like how to install the package, many are well written and
> give the appropriate, initial pointers for getting started with a new
> package (basic configuration recommendations, what the main entry points
> are, what options to look out for).
>
> As mentioned before in this thread, I don't think C-h P should give the
> same information as the manual -- be in in org-mode or rendered as
> ASCII/Unicode.  This is too much information for someone who wants to
> just wants to know what does a package "foobar" do (this is especially
> critical when package names are not self explanatory/generic, as was
> discussed a few weeks back).
>
> [... 14 lines elided]
>
> Once again I would recommend publishing guidelines and recommendations
> on good packaging practices on elpa.gnu.org, with tips and examples on
> what a good commentary section looks like.

Agreed!

I do use the Commentary for this purpose and am happy to conform with
such guidelines.

-- 
Protesilaos Stavrou
https://protesilaos.com



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 10:50                         ` Protesilaos Stavrou
  2022-06-07 12:07                           ` Philip Kaludercic
@ 2022-06-07 12:27                           ` Stefan Monnier
  1 sibling, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-07 12:27 UTC (permalink / raw)
  To: Protesilaos Stavrou; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

> Concerning point 2, the GNU ELPA machinery converts the Org source into
> an Info file.  (Those who install the package can access it with C-h i.)
> While I handle the export process into HTML.

Side note: elpa.gnu.org also converts the docs to HTML (both for Org
and for Texinfo docs), and puts a link to it under "Manual" as can be
seen on http://elpa.gnu.org/packages/modus-themes.html

[ This HTML isn't ideal, admittedly.  Help welcome.  ]

> This arrangement provides two viable options for users who (i) want to
> read the documentation yet (ii) do not wish to load org.el and its
> accoutrements.  A third option, in the form of a plain README, is a
> useful addition, provided it is done automatically by GNU ELPA.

I have the impression that there's a bit of misunderstanding here.  There
are two pieces of "docs" that elpa.gnu.org handles: the `:readme` and
the `:doc`.  One is meant for an actual manual that can be browsed with Info,
while the other is expected to be a short blurb, to help prospective
users understand what the package is about.
Both accept the Org format, but the `:doc` does not accept "plain text"
(nor Markdown).

Many packages currently use the same file for both (typically called
`README.org`), which tends to be suboptimal (it's often neither a great
`:readme` nor a great `:doc`), but works fine.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07  6:14                         ` Tim Cross
  2022-06-07 11:21                           ` Eli Zaretskii
@ 2022-06-07 16:02                           ` Alan Mackenzie
  2022-06-07 18:14                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Stefan Monnier
  2022-06-07 21:51                             ` Convert README.org to plain text README while installing package Tim Cross
  1 sibling, 2 replies; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-07 16:02 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

Hello, Tim.

On Tue, Jun 07, 2022 at 16:14:52 +1000, Tim Cross wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

> >> From: Tim Cross <theophilusx@gmail.com>
> >> Cc: emacs-devel@gnu.org
> >> Date: Mon, 06 Jun 2022 23:57:55 +1000

> >> > There are 784 key bindings in org mode.  How can you say that this isn't
> >> > complex and difficult to learn?

> >> Very simply because the vast majority of  those key bindings only come
> >> into play when you use advanced features of org mode, such as the
> >> agenda or table editing or noweb mode. If your just using basic org mode
> >> features, very very few of those key bindings even come into play. Just
> >> go throgh that list and remove any bindings relating to agenda mode,
> >> table editing mode, source block modes, clock table mode, list editing
> >> mode, etc and you end up with very few key bindings (all of which are
> >> udner the C-c prefix, unlike your previous claim). 

> > How about if you do this exercise and walk us through the results?
> > Visit an Org file, type "C-h b", then show us the list, and tell which
> > bindings you think are "basic" and which are "advanced", and why you
> > think so.

> > I can tell you what I see with my very simple 1570-line Org file,
> > which just has some todo items organized in 4-level hierarchy:

> >   . ~100 keys that look "general Org" to me
> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
> >     'backward-sentence'
> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
> >     bound by default, but they are there
> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
> >     but they are all bound by default, just because I visited an Org
> >     file 

> > So "just" 230 key bindings.  And this is without any minor/add-on Org
> > mode/feature enabled, at least according to "C-h m".

> > Do you see something different?  Are you still saying that it's not a
> > lot, or that it's "based on ignorance and paranoia"?  If so, please
> > point out where I'm ignorant and/or paranoiac, because I'd really like
> > to know.

> I think Alan's response has pretty much confirmed what I was saying. Alan has
> made it fairly clear that his real underlying concern is about a stealthy
> strategy to get org mode more intrinsically tied into Emacs and in the end,
> making it so critical to normal operation that everyone will be forced to learn
> org mode regardless. I don't necessarily agree with such an argument, but think
> having a debate around that would be more up-front and in some ways honest
> compared to what appear to be somewhat contrived alternative arguments about
> excessive key bindings and added complexity.

You've misread the situation.  My objections to org mode are almost
entirely due to its excessive complexity as measured by the number of key
bindings.

[ .... ]

> With respect to ignorance, Alan made it quite clear he has not spent
> the time or effort to learn org mode, so he is ignorant about its
> operation and lacks any real understanding of its operation.

This is indeed the case.

> He has made assertions about the number of key bindings which are
> wildly over inflated .....

You have said this several times in your post without any justification.
There are 784 entries in the key index in org.info.  You're saying it's
"wildly over inflated" to use this number, at least as a first estimate.
Why are you making such a counter intuitive claim without any
justification? 

Eli counted around 230 bindings from C-h m.  It thus seems likely that on
average, an org mode key sequence is used for 3½ distinct functions.
This is not a good thing.

> .... and about required levels of complexity which simply would not be
> necessary with the org mode rendering of a read only buffer.

You're not saying that loading a RO readme.org would somehow only load a
small part of org mode, surely?

> In short, he has made a lot of assumptions about both the key bindings
> and complexity based on only a fairly cursory scan of the manual and
> very limited experience. 

And now you're trying to be patronising, but failing.  ;-)

Why do you make no attempt to explain why my assumptions are invalid?

> All that appears to have occurred here is that someone noted that
> packages often had a readme.org file and wondered if this should be
> translated into just a plain readme file. Someone else suggested we
> could have package descriptions render the readme.org file using org
> and not require export into plain readme file and mentioned some
> possible advantages this could have wrt navigation, rendering of
> tables, links and images or inclusion of source code snippets, examples
> etc.

> From this point, some somewhat extreme claims and arguments have been
> thrown in.  One was Alan's claim that org would introduce 785 new key
> bindings, many of which not bounmd to C-c and which would presumably
> either pollute the key space or conflict with existing bindings. All I
> have done is point out this claim was inaccurate and overstated due to
> an overly simplistic analysis of what is in the key index of the org
> manual.

You have made an unfounded assertion of my inaccuracy.  Are you
suggesting, perhaps, that of these 784 entries in org.info's key index,
the vast bulk are dummies?  Some of these bindings (I don't think I used
the word "many", but may have done) are not in the standard major mode
space C-c C-<letter>.  Some of these clash with existing bindings,
examples of which I gave last night.

> it should be noted that if we were to add 'native' rendering of readme.org
> files, this would likely be done with an org minor mode rather than a full blown
> org mode as very little of the overall org functionality would be required in
> this situation.

In other words, an attempt would be made to fix the problems which I am
drawing to people's attention.

> All that would be required is org rendering support and org support for
> navigation (folding/unfolding, following links and possibly list/table
> navigation). All the advanced org functionality relating to babel,
> noweb, todo and time management, agendas, etc have no relevance in this
> scenario. LIkewise, if this functionality was provided via a minor
> mode, it would also be possible for people to disable that mode and be
> left with just the readme.org file, which is just plain text and quite
> readable.

> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 

Actually it was 784, 28².  Again, why is this number invalid?

> Running Emacs wiht -q and opeing up an org file, I find the following 

> - 236 org related key bindings - far less than 785 Of these, a number are
> - actually remapping of existing bindings, so not new ones. This leaves 208.

> I said that for a read only buffer, many of the org key bindings are not
> relevant as they relate to features which are not pertinent to a read only org
> buffer. Any bindings relating to babel, todo management, time management,
> agendas etc have no relevance when reading a readme.org file. Really, the only
> key bindings of any relevance are navigation related bindings (including
> opening/closing folds, following links, etc). Removing all unnecessary org
> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
> a few could also be removed, but I've left them in as I only wanted to remove
> those which would clearly be of no benefit in a read only buffer rendering of a
> readme.org file. In an org minor mode setup for rendering of readme.org (and
> possibly other read-only rendering of org files), I estimate at least another 30
> bindings could be removed because they have no real benefit or are of only
> marginal benefit. Creating an org minor mode for this purpose which only
> introduced 20 or so new bindings would easily be possible.

You're describing what might be rather than what is.

> The key point is that claims of 785 new bindings are incorrect and
> misleading.

I resent you continally repeating this unfounded assertion.  There are
784 entries in org.info's key index.  That is the current state.  That is
the truth.  How can that be "incorrect"?  Even if 750 of the 784 bindings
somehow "don't come into play", they still exist, and are still
problematic.  Are you saying that only every tenth entry actually
signifies an actual binding, or something like that?

> Even being very conservative, we are really talking about 10% of the
> number being incorrectly claimed.  if we introduced a minor mode for
> this, we are talking likely less than 20 or so.

> Arguments regarding this being a 'slippery slope' or 'edge of the
> wedge' in an org takeover are a completely different argument and there
> may well be some concerns which may need to be considered. However, if
> that is the real concern, then argue based on that rather than
> exaggerated claims about excessive key bindings, key binding
> polution/conflicts and excessive complexity.

If it weren't for org mode's excessive key bindings and complexity, the
"takeover scenario" would scarcely be a worry.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 10:35                         ` Philip Kaludercic
@ 2022-06-07 17:55                           ` Stefan Monnier
  2022-06-24  7:18                             ` Akib Azmain Turja
  2022-06-24 15:42                             ` Philip Kaludercic
  0 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-07 17:55 UTC (permalink / raw)
  To: Philip Kaludercic
  Cc: Akib Azmain Turja, Ihor Radchenko, Po Lu, Tassilo Horn,
	Michael Albinus, Alan Mackenzie, Stefan Kangas, emacs-devel

> From 9d973044a346860d3fc6164fe75ad8cd9721a595 Mon Sep 17 00:00:00 2001
> From: Philip Kaludercic <philipk@posteo.net>
> Date: Mon, 6 Jun 2022 12:34:40 +0200
> Subject: [PATCH] Render Org documentation in a plain-text README-elpa file
>
> * elpa-admin.el (elpaa--make-one-tarball-1): Call elpaa--write-plain-readme.
> (elpaa--write-plain-readme): Add new function.

That looks OK, feel free to push that to `elpa-admin`.
Comments:
- you'd get more bonus points by arranging for the code
  to (re)use the existing plain text (which is currently produced later
  when (re)generating the HTML page, so it would take a fair bit of
  code reorg to get that.  In the mean time, a FIXME comment seems
  in order).
- Your code may end up "rendering" a Markdown file for no obvious benefit
  since the rendering is a no-op (tho I think this is very hypothetical
  and likely can't happen because of other constraints).
- We might also add a `README-elpa` file when the readme file has a name
  not recognized by `package--get-description`.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-07 16:02                           ` Alan Mackenzie
@ 2022-06-07 18:14                             ` Stefan Monnier
  2022-06-07 18:26                               ` Org mode and Emacs Lars Ingebrigtsen
                                                 ` (2 more replies)
  2022-06-07 21:51                             ` Convert README.org to plain text README while installing package Tim Cross
  1 sibling, 3 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-07 18:14 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Tim Cross, emacs-devel

Rather than try and figure out who's right or wrong, I hope we can take
advantage of this discussion to get some code out that try to solve
the problem.

AFAICT the problem seen from Emacs, is that Org is large (even for
a basic uses, which occasionally leads to high load times) and that it
doesn't follow all the usual Emacs conventions, such as
overriding/remapping standard key-bindings (the resulting behaviors may
sometimes be similar to the standard one, but even when that's the case
the redefinition can easily bite the user).

The problem seen from Org is that Emacs doesn't use Org enough :-)
[ This paragraph's shorter length probably reflects the fact that I'm
  less familiar with Org than with Emacs.  ]

I think the way forward is to define a "basic org-mode".  This one could
be used at many more places where there's currently an occasional desire
to use Org that's resisted because of the above problems.
Ideally, org-mode would then be defined as an extension of this "basic
org-mode".  Also ideally some of those extensions would be reworked so
they can be used "Emacs-wide" rather than only in Org (obviously, that
can only make sense for some of them).

We could start this "basic org-mode" as a trivial copycat of
`outline-mode` and then start adding Org features to it.  The driving
constraint is to keep it lightweight and rules-abiding enough that there
won't be any resistance to using it, while at the same time making sure
that the features added to it can be removed from the "org-mode
extension", so that org-mode and "basic org-mode" don't diverge.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 18:14                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Stefan Monnier
@ 2022-06-07 18:26                               ` Lars Ingebrigtsen
  2022-06-07 18:48                                 ` Stefan Monnier
  2022-06-07 22:10                               ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Tim Cross
  2022-06-08 13:22                               ` Ihor Radchenko
  2 siblings, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-07 18:26 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.

I think that people that use Org mode wouldn't be very interested in
that, and people that don't use Org can just continue editing those
files with `M-x find-file-literally', really.  Doing so is fine.

But I don't understand the discussion this thread warped out of.  When
we display a README from Package, we shouldn't be showing the raw text
of README.org or README.md or README.html, lightly fontified -- we
should be showing a rendering of it.  People that read the README aren't
supposed to edit it, after all.

Which would make the 40K key bindings that Org apparently has completely
irrelevant for non-Org users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 18:26                               ` Org mode and Emacs Lars Ingebrigtsen
@ 2022-06-07 18:48                                 ` Stefan Monnier
  2022-06-07 18:54                                   ` Eli Zaretskii
  2022-06-07 20:54                                   ` Lars Ingebrigtsen
  0 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-07 18:48 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

> I think that people that use Org mode wouldn't be very interested in
> that, and people that don't use Org can just continue editing those
> files with `M-x find-file-literally', really.  Doing so is fine.

I was thinking of files like etc/NEWS where having "a bit more than
outline-mode" would be welcome.

> But I don't understand the discussion this thread warped out of.  When
> we display a README from Package, we shouldn't be showing the raw text
> of README.org or README.md

Why not?  If it can be prettified on the fly by font-lock?

Separating "viewing" from "editing" is un-Emacsish, because it puts
a barrier between the "consumers" and the "producers" whereas Emacs
likes to try and get end-users exposed to the source as mush as possible
so it takes little effort for them to change role.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 18:48                                 ` Stefan Monnier
@ 2022-06-07 18:54                                   ` Eli Zaretskii
  2022-06-07 19:38                                     ` Stefan Monnier
  2022-06-07 20:54                                   ` Lars Ingebrigtsen
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-07 18:54 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: larsi, acm, theophilusx, emacs-devel

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 14:48:31 -0400
> 
> > But I don't understand the discussion this thread warped out of.  When
> > we display a README from Package, we shouldn't be showing the raw text
> > of README.org or README.md
> 
> Why not?

Because it's butt-ugly!



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 18:54                                   ` Eli Zaretskii
@ 2022-06-07 19:38                                     ` Stefan Monnier
  0 siblings, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-07 19:38 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: larsi, acm, theophilusx, emacs-devel

>> > But I don't understand the discussion this thread warped out of.  When
>> > we display a README from Package, we shouldn't be showing the raw text
>> > of README.org or README.md
>> Why not?
> Because it's butt-ugly!

I don't think it has to be.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 18:48                                 ` Stefan Monnier
  2022-06-07 18:54                                   ` Eli Zaretskii
@ 2022-06-07 20:54                                   ` Lars Ingebrigtsen
  1 sibling, 0 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-07 20:54 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But I don't understand the discussion this thread warped out of.  When
>> we display a README from Package, we shouldn't be showing the raw text
>> of README.org or README.md
>
> Why not?  If it can be prettified on the fly by font-lock?
>
> Separating "viewing" from "editing" is un-Emacsish, because it puts
> a barrier between the "consumers" and the "producers" whereas Emacs
> likes to try and get end-users exposed to the source as mush as possible
> so it takes little effort for them to change role.

That's true, but on the other hand, we don't show texinfo files to
people for a good reason.  That's about as "plain text" as normal Org
files, or as some Markdown files get.  (Most Markdown files are kinda
pleasant, but there's plenty of scope for unreadability -- it drops down
into HTML for tables, for instance.)

Or take doc strings as an example -- it's always been a markup language,
too (\\ and \\[] and \\<>), but it's moving more clearly in that
direction -- because we want to make the documentation we present the
user pretty and easy to read.  And we have one-key commands to take us
to the source code, which we could have in the Package README rendering,
too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  0:19                 ` Tim Cross
                                     ` (2 preceding siblings ...)
  2022-06-06 16:56                   ` Michael Albinus
@ 2022-06-07 20:57                   ` Jean Louis
  2022-06-08  6:50                     ` Tim Cross
  3 siblings, 1 reply; 376+ messages in thread
From: Jean Louis @ 2022-06-07 20:57 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

* Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
> Org files *are* plain text. This is one of (perhaps the biggest) selling
> points for org mode.

We may call it "plain text" and problem is not that we can open Org
files with any editor as plain text, but in formatting. People format
Org files in such ways that they are not readable, they may not make
spacing where it would be otherwise required, in other words, plain
text files do not look nearly as readable as for example RFC text,
like this one: https://www.rfc-editor.org/rfc/rfc1.txt

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 16:02                           ` Alan Mackenzie
  2022-06-07 18:14                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Stefan Monnier
@ 2022-06-07 21:51                             ` Tim Cross
  2022-06-08 19:26                               ` chad
  1 sibling, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-07 21:51 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: emacs-devel


Alan Mackenzie <acm@muc.de> writes:

> I resent you continally repeating this unfounded assertion.  There are
> 784 entries in org.info's key index.  That is the current state.  That is
> the truth.  How can that be "incorrect"?  Even if 750 of the 784 bindings
> somehow "don't come into play", they still exist, and are still
> problematic.  Are you saying that only every tenth entry actually
> signifies an actual binding, or something like that?
>

I will make a last attempt to clarify. 

When this thread started, the discussion was around whether readme.org
files should be converted to or replaced with just plain 'traditional'
readme files - waht would once probably have been called ASCII files,
though now these are utf8 I'm not sure calling them ascii is accurate.
It was then suggested an alternative would be to have the readme.org
files formatted using org-mode formatting - enabling org mode features
for folding, formatting of tables, links, source code examples etc. It
was also stated this would be a read-only buffer. 

The point which has been missed an probably should have been made
clearer is that nobody was talking about a full blown org mode. The
suggestion was to use primarily org mode rendering/presentation
capabilities in a read-only buffer. 

My point was that most of the key bindings and complexity you were
referring to were not relevant in this scenario. Of the 784 key bindings
you refer to, the majority of them would never come into play because
they are bindings which only exist is specialised sub-modes of org mode,
like the agenda, which would not appear in the rendering of a readme.org
file. When you open an org file, you get 236 org related key bindings,
not 7984. All those additional key bindings only get loaded/defined if
you open specialised org modes like the agenda, which does not have a
key binding by default, so your unlikely to accidentally initiate that
mode. 

So going back to my original post, which was simply saying that
implying you would get 784 additional key bindings just because you open
an org file was misleading because you simply don't get that many
additional key bindings when you open an org file - in fact, you get
only about 1/3 of that many. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-07 18:14                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Stefan Monnier
  2022-06-07 18:26                               ` Org mode and Emacs Lars Ingebrigtsen
@ 2022-06-07 22:10                               ` Tim Cross
  2022-06-08  6:06                                 ` Org mode and Emacs Visuwesh
  2022-06-09 22:31                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  2022-06-08 13:22                               ` Ihor Radchenko
  2 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-07 22:10 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Alan Mackenzie, emacs-devel


Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Rather than try and figure out who's right or wrong, I hope we can take
> advantage of this discussion to get some code out that try to solve
> the problem.
>
> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).
>
> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.
>
>

Thanks Stefan and I agree. I also think this is in general exactly the
direction current org maintainers/developers are taking. 

At this point, considerable work is being done (by Ihor and others) to 

- Implement a concise and efficient parser. This has also involved
  refining and clarifying org syntax, which has had some holes and
  ambiguities. 

- Replace org's largely regexp based font locking mechanism with one
  based on the parsing and tagging made possible by the org parser. This
  should improve both performance and accuracy in parsing. 

- Improving efficiency of folding etc

- Improving efficiency with respect to larger org files through the use
  of caching etc. 

- Increasing org's use of built-in Emacs capabilities rather than using
  org specific implementations. For example, adopting transient instead
  of an org implemented module which replicates similar functionality. 

- Increase modularity an enable loading of the functionality that you
  want. 

In particular, the important work being done by Ihor wrt folding,
parsing and caching will provide a core of functionality which can be
used to define something like an org minor mode which could be used in
those situations where we want some basic org functionality, like
folding, navigation and formatting/presentation, but we don't need all
the additional advanced features, such as babel, todo and time
management, spreadsheets, tables and table formulas etc. 

Of course, to what extent Emacs will/should leverage off org mode is a
completely different debate. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07  7:53                           ` Po Lu
@ 2022-06-07 22:15                             ` Kévin Le Gouguec
  2022-06-08  0:36                               ` Po Lu
  0 siblings, 1 reply; 376+ messages in thread
From: Kévin Le Gouguec @ 2022-06-07 22:15 UTC (permalink / raw)
  To: Po Lu; +Cc: Tim Cross, Alan Mackenzie, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> No one has ever had to "go through the entire Org manual in order to
>> write and edit README files".  Just like no one has ever had to read the
>> Emacs manual cover-to-cover in order to do anything with it (they _can_,
>> of course, but they don't _have to_).
>
> So what was the purpose of the following statement?
>
>>> Spend the time to go through the key bindings listed in the org
>>> manual

Cannot speak for Alan nor Tim, but my understanding of the exchange that
led to the bit you quote is that Tim was inviting Alan to check how many
of these 784 bindings had any relevance to the topic under discussion
(navigating and, to a lesser extent, authoring Org READMEs) in order to
show that "M-x org TAB" was not a good heuristic to assess how much
complexity _needs_ to be shoved onto an unsuspecting README reader.

In Tim's own words (a bit before the one you quote):

> That 750 key bindings is an extreme over statement and not what is being
> proposed.

So your paraphrasing ("expecting someone to go through the entire Org
manual in order to write and edit README files") struck me as
disconnected from (my own understanding of) Tim's argument: AFAIU,

* he wasn't saying _Org users_ need to go through the manual,

* he was confronting the "784 bindings" argument by saying it was not a
  particularly useful heuristic, inviting _us_ (participants of the
  present discussion) to check for ourselves what proportion of these
  bindings are actually relevant to the discussion.

Again though, cannot speak for Tim; I can only explain my undertanding
of the discussion, and why I was dumbfounded by your reply.

> IMO, packages should provide a plain text README, alongside a rich text
> version in a relatively standard format like HTML.
>
> By using HTML, it becomes readable not only in Emacs, but also any other
> program that understands HTML.  HTML is also understood by many more
> people than Org mode.
>
> It's okay for package maintainers to write their documentation using
> Org, and then to export it to HTML.  But please don't insist that all
> documentation be written using Org mode, or that it's okay for packages
> to only provide documentation in that format.

If there's been advocating for all documentation to be written in Org,
I've (dis)missed that, so sorry for not following closely enough.

As for packages only providing documentation in Org: again, I think the
way forward is automation at the package archive level.  Asking authors
to maintain two sync'ed READMEs stands a small chance of success IMO.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 22:15                             ` Kévin Le Gouguec
@ 2022-06-08  0:36                               ` Po Lu
  2022-06-08  6:41                                 ` Kévin Le Gouguec
  0 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-08  0:36 UTC (permalink / raw)
  To: Kévin Le Gouguec; +Cc: Tim Cross, Alan Mackenzie, emacs-devel

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> So your paraphrasing ("expecting someone to go through the entire Org
> manual in order to write and edit README files") struck me as
> disconnected from (my own understanding of) Tim's argument: AFAIU,
>
> * he wasn't saying _Org users_ need to go through the manual,
>
> * he was confronting the "784 bindings" argument by saying it was not a
>   particularly useful heuristic, inviting _us_ (participants of the
>   present discussion) to check for ourselves what proportion of these
>   bindings are actually relevant to the discussion.
>
> Again though, cannot speak for Tim; I can only explain my undertanding
> of the discussion, and why I was dumbfounded by your reply.

Going through the key bindings in the Org manual means to essentially
read the entire manual.

> If there's been advocating for all documentation to be written in Org,
> I've (dis)missed that, so sorry for not following closely enough.

That's how this discussion started, with Stefan K asking for all
documentation to be written in Org mode.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-07 22:10                               ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Tim Cross
@ 2022-06-08  6:06                                 ` Visuwesh
  2022-06-08  6:58                                   ` Tim Cross
  2022-06-09 22:31                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  1 sibling, 1 reply; 376+ messages in thread
From: Visuwesh @ 2022-06-08  6:06 UTC (permalink / raw)
  To: Tim Cross; +Cc: Stefan Monnier, Alan Mackenzie, emacs-devel

[புதன் ஜூன் 08, 2022] Tim Cross wrote:

> [...]
> - Increasing org's use of built-in Emacs capabilities rather than using
>   org specific implementations. For example, adopting transient instead
>   of an org implemented module which replicates similar functionality. 

Where can I read more about this?  I see it being mentioned a few times
in the org-mode mailing list, and in the matrix room.  Is the plan to
completely remove the org-mks interface and replace it with transient
without having an option to use the former?  I find org-mks perfectly
fine to use and would be sad if it was replaced with transient.
Transient needs time to get used to, and the default settings is quite
un-Emacsy; I'm not too excited about configuring yet another package
that has a hard-to-understand manual TBH.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  0:36                               ` Po Lu
@ 2022-06-08  6:41                                 ` Kévin Le Gouguec
  0 siblings, 0 replies; 376+ messages in thread
From: Kévin Le Gouguec @ 2022-06-08  6:41 UTC (permalink / raw)
  To: Po Lu; +Cc: Tim Cross, Alan Mackenzie, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
>
>> So your paraphrasing ("expecting someone to go through the entire Org
>> manual in order to write and edit README files") struck me as
>> disconnected from (my own understanding of) Tim's argument: AFAIU,
>>
>> * he wasn't saying _Org users_ need to go through the manual,
>>
>> * he was confronting the "784 bindings" argument by saying it was not a
>>   particularly useful heuristic, inviting _us_ (participants of the
>>   present discussion) to check for ourselves what proportion of these
>>   bindings are actually relevant to the discussion.
>>
>> Again though, cannot speak for Tim; I can only explain my undertanding
>> of the discussion, and why I was dumbfounded by your reply.
>
> Going through the key bindings in the Org manual means to essentially
> read the entire manual.

Yes, going through keybindings ≈ reading the entire manual.  My point is
that

    asking _participants of a discussion_ to do X in order to make the
    stakes clearer (my understanding of Tim's position)
≪
    asking anyone who wants to write and edit README files to do X (your
    paraphrasing)

>> If there's been advocating for all documentation to be written in Org,
>> I've (dis)missed that, so sorry for not following closely enough.
>
> That's how this discussion started, with Stefan K asking for all
> documentation to be written in Org mode.

Assuming you are referring to
<CADwFkm=6_xP6zRSkDP1kreMKCXGBmp8Ze2WtUMeqq9-gEuGcxw@mail.gmail.com>?

The actual OP, <87leuca7v7.fsf@disroot.org>, started with something much
more modest: converting Org READMEs to plaintext.

Then, AFAIU, came ideas to (in no particular order, and not mutually
exclusive) (1) render the Org syntax rather than converting it "down" to
plaintext, (2) enable Org wholesale, (3) break down Org into smaller
pieces in order to make it less of a pain to fit the "display a rich
README" use-case.

I admit to focusing on those points of the discussion, forgetting about
StefanK's message, and then getting very confused why Org's complexity
_as an authorship tool_ mattered so much.  Apologies for not paying
enough attention.


Anyway, Philip posted a patch to the effect of what the OP suggested in
<877d5ut6z6.fsf@posteo.net>.  Digging into the ELPA admin branch, I also
see that the machinery for the conversion is exactly what I expected: it
lets Org's export library do the heavy lifting.

So AFAICT reason prevailed; package maintainers who enjoy writing
READMEs in Org can keep doing so without impacting their users; and
despite the forecast for today being pretty rainy over here, I'm
starting my day in a resolutely sunny mood.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 20:57                   ` Jean Louis
@ 2022-06-08  6:50                     ` Tim Cross
  2022-06-08  7:25                       ` Ihor Radchenko
                                         ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-08  6:50 UTC (permalink / raw)
  To: Jean Louis; +Cc: emacs-devel


Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>> points for org mode.
>
> We may call it "plain text" and problem is not that we can open Org
> files with any editor as plain text, but in formatting. People format
> Org files in such ways that they are not readable, they may not make
> spacing where it would be otherwise required, in other words, plain
> text files do not look nearly as readable as for example RFC text,
> like this one: https://www.rfc-editor.org/rfc/rfc1.txt

Please give me an example of org mode 'not make space where it would be
otherwise required'. Can you provide a single example of org mode
syntax which is not readable in any text editor. There are quite a few
projects on Github/Gitlab which have readme.org files - can you point to
one which cannot be read with a plain text editor? 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-08  6:06                                 ` Org mode and Emacs Visuwesh
@ 2022-06-08  6:58                                   ` Tim Cross
  0 siblings, 0 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-08  6:58 UTC (permalink / raw)
  To: Visuwesh; +Cc: Stefan Monnier, Alan Mackenzie, emacs-devel


Visuwesh <visuweshm@gmail.com> writes:

> [புதன் ஜூன் 08, 2022] Tim Cross wrote:
>
>> [...]
>> - Increasing org's use of built-in Emacs capabilities rather than using
>>   org specific implementations. For example, adopting transient instead
>>   of an org implemented module which replicates similar functionality. 
>
> Where can I read more about this?  I see it being mentioned a few times
> in the org-mode mailing list, and in the matrix room.  Is the plan to
> completely remove the org-mks interface and replace it with transient
> without having an option to use the former?  I find org-mks perfectly
> fine to use and would be sad if it was replaced with transient.
> Transient needs time to get used to, and the default settings is quite
> un-Emacsy; I'm not too excited about configuring yet another package
> that has a hard-to-understand manual TBH.


The specifics of what is planned are still being worked out. Initially,
the likely initial candidate for change will be to the export menu.
Unless you have extensive low level customisation, it should not be a
change which has significant impact on users. In fact, maintaining
backwards compatibility and consistency for end users is important to
the org developers. It is also quite possible that after an initial
investigation, it may be decided transient is not a good option for org
mode or perhaps it will be a good option once additonal functionality or
enhancements are added. Right now, all that has been agreed is that it
would be worthwhile looking at it to see if it can be of benefit in
helping to reduce org maintenance overheads and/or increase org's
consistency with other emacs packages. 

What will determine what remains and what choices are available will
depend on what involvement people have in the development and what will
be maintainable. There are a number of areas in org mode where
functionality has been implemented that is 80+% equivalent to
functionality which exists or has been added to core emacs. Having this
duplication of effort is adding to the burden of maintenance, which is
already significant. In general, org maintainers keep an eye on what is
being added/expanded in Emacs core and when things are added, like
transient mode, they are assessed to see if adopting that functionality
would reduce the org maintenance load and improve org's consistency
withi the rest of Emacs. At this stage, transient has been added as
something to look at in the backlog. When this task makes if off the
backlog, the typical process would be for it to be discussed on the org
devel list, for initial implementations to be done either in its own
branch (if considered a significantly large enough change) or on the
development branch otherwise and people will be asked to try it out. 

So, if your interested in this area, the first thing would be to get on
the org devel mail list. Anyone who is particularly keen to see such
things added might also initiate discussions and development in their
own branch, which could then be added in as a PR. However, like Emacs,
any significant development work on org mode also requires FSF copyright
assignment. 

At this specific point in time, the main focus with org has been on
stability, performance improvements, especially for large org files with
many babel blocks and sections and improvements to the org syntax and provision of an
org parser,  

In particular, I think the work being done by Ihor on folding and
the org parser will have huge benefits. In addition to making the code
base easier to maintain and improving performance, it will help reduce
org's current dependency on complicated and difficult to maintain
regular expressions in the font-locking layer. This should improve
performance and reduce errors or unexpected results that sometimes occur
because of regexp errors etc and reduce time spent by maintainers in
tracking down such errors and maintenance of those complex regexps.   



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  6:50                     ` Tim Cross
@ 2022-06-08  7:25                       ` Ihor Radchenko
  2022-06-08  7:43                         ` Tim Cross
  2022-06-08 11:39                       ` Jean Louis
  2022-06-11  6:40                       ` Jean Louis
  2 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08  7:25 UTC (permalink / raw)
  To: Tim Cross; +Cc: Jean Louis, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> Jean Louis <bugs@gnu.support> writes:
>
>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>> points for org mode.
>>
>> We may call it "plain text" and problem is not that we can open Org
>> files with any editor as plain text, but in formatting. People format
>> Org files in such ways that they are not readable, they may not make
>> spacing where it would be otherwise required, in other words, plain
>> text files do not look nearly as readable as for example RFC text,
>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>
> Please give me an example of org mode 'not make space where it would be
> otherwise required'. Can you provide a single example of org mode
> syntax which is not readable in any text editor. There are quite a few
> projects on Github/Gitlab which have readme.org files - can you point to
> one which cannot be read with a plain text editor? 

I'd say that Jean made a fair point.
In Org, authors may not care as much about, for example, indentation.
Especially when the org files are written with org-indent-mode turned
on.

If you look at https://www.rfc-editor.org/rfc/rfc1.txt, the paragraphs a
nicely indented and can be distinguished from the headers. The top-level
headers are numbered and can be easily distinguished from the
lower-level headers. The paragraph text is also filled appropriately,
unlike some Org documents written with truncate-lines set to nil.

So, without Emacs, Org files (some of them!) may be harder to read
compared to properly formatted ASCII.

I am not 100% sure if we need to do anything about this observation.
One practical conclusion that can be made is that we might incorporate
filling, numbering, and indentation into ox-org.el.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  7:25                       ` Ihor Radchenko
@ 2022-06-08  7:43                         ` Tim Cross
  2022-06-08 11:27                           ` Jean Louis
  2022-06-08 13:24                           ` Ihor Radchenko
  0 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-08  7:43 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Jean Louis, emacs-devel


Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> Jean Louis <bugs@gnu.support> writes:
>>
>>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>>> Org files *are* plain text. This is one of (perhaps the biggest) selling
>>>> points for org mode.
>>>
>>> We may call it "plain text" and problem is not that we can open Org
>>> files with any editor as plain text, but in formatting. People format
>>> Org files in such ways that they are not readable, they may not make
>>> spacing where it would be otherwise required, in other words, plain
>>> text files do not look nearly as readable as for example RFC text,
>>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>>
>> Please give me an example of org mode 'not make space where it would be
>> otherwise required'. Can you provide a single example of org mode
>> syntax which is not readable in any text editor. There are quite a few
>> projects on Github/Gitlab which have readme.org files - can you point to
>> one which cannot be read with a plain text editor? 
>
> I'd say that Jean made a fair point.
> In Org, authors may not care as much about, for example, indentation.
> Especially when the org files are written with org-indent-mode turned
> on.
>
> If you look at https://www.rfc-editor.org/rfc/rfc1.txt, the paragraphs a
> nicely indented and can be distinguished from the headers. The top-level
> headers are numbered and can be easily distinguished from the
> lower-level headers. The paragraph text is also filled appropriately,
> unlike some Org documents written with truncate-lines set to nil.
>
> So, without Emacs, Org files (some of them!) may be harder to read
> compared to properly formatted ASCII.
>
> I am not 100% sure if we need to do anything about this observation.
> One practical conclusion that can be made is that we might incorporate
> filling, numbering, and indentation into ox-org.el.
>

On one level, fair enough. However, this has nothing to do with org mode
in the sense it is not enforced or caused by org mode. My point is that
there is nothing in the org mode syntax which makes it impossible to
read the contents with any plain text editor. You can also write a
poorly formatted ascii file which lacks indentation, spaces etc in any editor. 

There is nothing in org syntax which forces people to set truncate lines
to nil or prevents them from indenting or wrapping or adding underlines
or extra spaces around headings or .... If you want to write rfc
compliant files with org mode, you can. It isn't the org mode syntax
which is the issue. Besides, just because a plain text file doesn't look
as nice or isn't a subjectively readable doesn't mean it isn't a plain
text file anymore. 




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  7:43                         ` Tim Cross
@ 2022-06-08 11:27                           ` Jean Louis
  2022-06-08 13:24                           ` Ihor Radchenko
  1 sibling, 0 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-08 11:27 UTC (permalink / raw)
  To: emacs-devel, Tim Cross, Ihor Radchenko

I think that one of points conveyed with the term "plain text" means readable. That is what README should be about. Even the title of the file (README) indicates it is for reading. Org files are source files, they shall for public be exported to the actual text. The term "plain text" is not beneficial in this subject. Those files should be readable without Emacs.

Take as analogy the TeX system, or Texinfo, that is also plain text but not readable. 
 

Jean



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  6:50                     ` Tim Cross
  2022-06-08  7:25                       ` Ihor Radchenko
@ 2022-06-08 11:39                       ` Jean Louis
  2022-06-11  6:40                       ` Jean Louis
  2 siblings, 0 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-08 11:39 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

It was not the point if file is readable by computer program named text editor.

The point is if it is readable for human.

There are many Emacs users who don't use Org. They are not supposed to decipher brackets, properties, links, and lack of authors' sense in writing Org files in human readable manner.

This whole subject is related to accessibility, Org files could be accessible and thus well readable if they would be just simple text without any markup, without  headings, but they are practically not so. Authors write headings without space to its paragraph, it destroys the original meaning of the heading, file does not convey the intended idea unless you are Emscd wizard.

Difficulties in reading reject users.

I will give you examples later.

On June 8, 2022 6:50:24 AM UTC, Tim Cross <theophilusx@gmail.com> wrote:
>
>Jean Louis <bugs@gnu.support> writes:
>
>> * Tim Cross <theophilusx@gmail.com> [2022-06-06 15:57]:
>>> Org files *are* plain text. This is one of (perhaps the biggest)
>selling
>>> points for org mode.
>>
>> We may call it "plain text" and problem is not that we can open Org
>> files with any editor as plain text, but in formatting. People format
>> Org files in such ways that they are not readable, they may not make
>> spacing where it would be otherwise required, in other words, plain
>> text files do not look nearly as readable as for example RFC text,
>> like this one: https://www.rfc-editor.org/rfc/rfc1.txt
>
>Please give me an example of org mode 'not make space where it would be
>otherwise required'. Can you provide a single example of org mode
>syntax which is not readable in any text editor. There are quite a few
>projects on Github/Gitlab which have readme.org files - can you point
>to
>one which cannot be read with a plain text editor? 


Jean



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 17:40               ` Stefan Monnier
@ 2022-06-08 12:51                 ` Ihor Radchenko
  2022-06-08 14:17                   ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 12:51 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Po Lu, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Tassilo Horn, Akib Azmain Turja, Emacs developers

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I've heard about the problem with loading multiple times and I do agree
>> that loading can be slow, especially when various kinds of extra
>> functionality is requested. However, I am wondering about the actual
>> numbers.  How long is long?
>
> I think it's fairly hard to quantify accurately: last time I tried to
> benchmark it, the numbers (around 1s) were much lower than what I was
> experiencing subjectively (I'd estimate 5s or more).  So, AFAICT, the
> time to load Org mode is strongly affected by the size of the running
> session or maybe by the presence of various third party packages, which
> means benchmarking
>
>     emacs --batch  --eval '(require `org)'
>
> doesn't give a representative picture of the problem.

I suspect that garbage collection is strongly affecting your loading
times. It is taking half of the time even with clean Emacs. I imagine
that GC sweeps are even more costly when you have a sizable allocated
memory.

Loading ol-* staff appears to generate a lot of GC runs because of
mapconcats over all the links types with O(N_links^2) strings being
allocated. I took a note of this. I am sure that memory usage can be
improved on loading.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06 12:56                         ` Lars Ingebrigtsen
@ 2022-06-08 12:55                           ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 12:55 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Tassilo Horn, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> In this specific paragraph, I was referring to file:, id:, help:, and
>> https: links.
>
> So you were talking about these?
>
> [[http://angg.twu.net/emacsconf2019.html][How to record executable notes with eev - and how to play them back]]
>
> Then I posit that it's trivial to get to a speedy 90% solution for these.

If you are interested to do this, feel free to go ahead. If you need any
help to get started, feel free to ask me or post a question in Org ML.

I am a bit biased about fontification because of
https://orgmode.org/list/87ee7c9quk.fsf@localhost
I keep thinking about various edge cases.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-06  7:39                         ` Tassilo Horn
@ 2022-06-08 12:56                           ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 12:56 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Lars Ingebrigtsen, Po Lu, Michael Albinus, Alan Mackenzie,
	Stefan Kangas, Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>>>> In short, no.
>>>
>>> Is that a challenge?  😀
>>
>> Sorry, I do not understand what you mean here.
>
> Lars is asking if he should hack up a solution which will probably work
> well enough for 95% of all existing README.org/md files, so just say
> "yes"! ;-)

:) Done.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 11:21                           ` Eli Zaretskii
@ 2022-06-08 13:12                             ` Ihor Radchenko
  2022-06-08 14:15                               ` Eli Zaretskii
  2022-06-08 19:34                             ` Convert README.org to plain text README while installing package Tim Cross
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 13:12 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Tim Cross, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> - 236 org related key bindings - far less than 785 Of these, a number are
>> - actually remapping of existing bindings, so not new ones. This leaves 208.
>
> ??? How is remapping "not new"?  It takes some very common Emacs
> commands and redirects them to different commands.  E.g., 'open-line'
> now does something quite different.  This means the user should either
> go learn what the Org commands do, or be prepared to be surprised.

There are 3 main purposes of remappings in Org:
1. To implement some optional functionality. That functionality is not
   enabled by default and no surprises will be given to user.
2. To implement the original functionality where the existing Emacs
   customization is not sufficient. For example, org-backward-paragraph
   has to re-implement backward-paragraph as the original function only
   supports regexp-bases paragraph boundaries. Regexps are not
   sufficient in Org syntax.
3. Like (1), but deliberately changing the default behavior to what Org
   users prefer historically. This is usually not much different from
   customisations that are usually made in define-derived-mode, except
   that Emacs does not have enough flexibility to do it there.

>> I said that for a read only buffer, many of the org key bindings are not
>> relevant as they relate to features which are not pertinent to a read only org
>> buffer. Any bindings relating to babel, todo management, time management,
>> agendas etc have no relevance when reading a readme.org file.
>
> Then why does Org define them in that case?

I am not very sure what is the problem with the number of bindings. It
is not different from Emacs itself.

For example, I am sometimes puzzled by <F2> prefix key in Emacs defining
bindings for two-column.el. Over years of using Emacs I only used them
by accident and always felt like the "break" my Emacs.

Yet, I am fine with those hundreds of default Emacs bindings I never
use. I do not have to read the full Emacs manual to understand those
bindings (they are even not all described in the Emacs manual!).

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-07 18:14                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Stefan Monnier
  2022-06-07 18:26                               ` Org mode and Emacs Lars Ingebrigtsen
  2022-06-07 22:10                               ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Tim Cross
@ 2022-06-08 13:22                               ` Ihor Radchenko
  2022-06-08 13:33                                 ` Stefan Kangas
       [not found]                                 ` <jwvr13z452j.fsf-monnier+emacs@gnu.org>
  2 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 13:22 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAICT the problem seen from Emacs, is that Org is large (even for
> a basic uses, which occasionally leads to high load times) and that it
> doesn't follow all the usual Emacs conventions, such as
> overriding/remapping standard key-bindings (the resulting behaviors may
> sometimes be similar to the standard one, but even when that's the case
> the redefinition can easily bite the user).

I am not sure if I follow the argument. Major modes are allowed to
change defaults. For example, special modes often change truncate-lines
value. Org mode is also tweaking defaults (yes, many of them). I do not
see any problem here in general.

If you have some specific cases when Org mode alters Emacs defaults in a
way that bites the user, please give concrete examples. Otherwise, your
criticism is not very constructive.

> The problem seen from Org is that Emacs doesn't use Org enough :-)
> [ This paragraph's shorter length probably reflects the fact that I'm
>   less familiar with Org than with Emacs.  ]
>
> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.
> Ideally, org-mode would then be defined as an extension of this "basic
> org-mode".  Also ideally some of those extensions would be reworked so
> they can be used "Emacs-wide" rather than only in Org (obviously, that
> can only make sense for some of them).
>
> We could start this "basic org-mode" as a trivial copycat of
> `outline-mode` and then start adding Org features to it.  The driving
> constraint is to keep it lightweight and rules-abiding enough that there
> won't be any resistance to using it, while at the same time making sure
> that the features added to it can be removed from the "org-mode
> extension", so that org-mode and "basic org-mode" don't diverge.

This is reasonable. RMS also asked for this years back
https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
Since we cannot start Org from scratch, factoring out individual modules
is taking a lot of time and having the hostile attitudes expressed in
some of the emails in this thread is not exactly encouraging.
If you want Org to be more modular, please help by reporting
inconsistencies or misbehavior to Org ML.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  7:43                         ` Tim Cross
  2022-06-08 11:27                           ` Jean Louis
@ 2022-06-08 13:24                           ` Ihor Radchenko
  1 sibling, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 13:24 UTC (permalink / raw)
  To: Tim Cross; +Cc: Jean Louis, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> On one level, fair enough. However, this has nothing to do with org mode
> in the sense it is not enforced or caused by org mode. My point is that
> there is nothing in the org mode syntax which makes it impossible to
> read the contents with any plain text editor. You can also write a
> poorly formatted ascii file which lacks indentation, spaces etc in any editor. 

Yes, but the poor formatting is immediately noticeable in ASCII. Not in
Org.

One may provide an org-lint checker for stylistic formatting issues
maybe?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-08 13:22                               ` Ihor Radchenko
@ 2022-06-08 13:33                                 ` Stefan Kangas
       [not found]                                 ` <jwvr13z452j.fsf-monnier+emacs@gnu.org>
  1 sibling, 0 replies; 376+ messages in thread
From: Stefan Kangas @ 2022-06-08 13:33 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Stefan Monnier, Alan Mackenzie, Tim Cross, Emacs developers

Ihor Radchenko <yantar92@gmail.com> writes:

> Since we cannot start Org from scratch, factoring out individual modules
> is taking a lot of time and having the hostile attitudes expressed in
> some of the emails in this thread is not exactly encouraging.

Thanks for working on modularizing Org mode, it sounds promising.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 13:12                             ` Ihor Radchenko
@ 2022-06-08 14:15                               ` Eli Zaretskii
  2022-06-08 15:15                                 ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-08 14:15 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 21:12:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> - 236 org related key bindings - far less than 785 Of these, a number are
> >> - actually remapping of existing bindings, so not new ones. This leaves 208.
> >
> > ??? How is remapping "not new"?  It takes some very common Emacs
> > commands and redirects them to different commands.  E.g., 'open-line'
> > now does something quite different.  This means the user should either
> > go learn what the Org commands do, or be prepared to be surprised.
> 
> There are 3 main purposes of remappings in Org:

I'm sure there are good reasons for that.  My point is that such
remappings effectively force the user to re-learn the commands he/she
is very familiar with.  So it's a non-trivial burden.

> >> I said that for a read only buffer, many of the org key bindings are not
> >> relevant as they relate to features which are not pertinent to a read only org
> >> buffer. Any bindings relating to babel, todo management, time management,
> >> agendas etc have no relevance when reading a readme.org file.
> >
> > Then why does Org define them in that case?
> 
> I am not very sure what is the problem with the number of bindings. It
> is not different from Emacs itself.

The difference is that we had years or decades to get used to the
Emacs defaults, and once Org is turned on in a buffer, one has a lot
of new stuff to get used to.  Unless Org is used constantly, you will
forget most of those changes till the next time, so this re-learning
experience will be repeated every time.

It isn't a catastrophe, of course, but we should recognize this as an
issue, especially if many of the bindings aren't needed.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 12:51                 ` Ihor Radchenko
@ 2022-06-08 14:17                   ` Eli Zaretskii
  2022-06-08 14:38                     ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-08 14:17 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: monnier, luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Po Lu <luangruo@yahoo.com>,  Michael Albinus <michael.albinus@gmx.de>,
>  Alan Mackenzie <acm@muc.de>,  Stefan Kangas <stefan@marxist.se>,  Tassilo
>  Horn <tsdh@gnu.org>,  Akib Azmain Turja <akib@disroot.org>,  Emacs
>  developers <emacs-devel@gnu.org>
> Date: Wed, 08 Jun 2022 20:51:35 +0800
> 
> >     emacs --batch  --eval '(require `org)'
> >
> > doesn't give a representative picture of the problem.
> 
> I suspect that garbage collection is strongly affecting your loading
> times. It is taking half of the time even with clean Emacs. I imagine
> that GC sweeps are even more costly when you have a sizable allocated
> memory.

Which, btw, yet another problem: why does loading Org produce so much
garbage?  Can this be kept in check somehow?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 14:17                   ` Eli Zaretskii
@ 2022-06-08 14:38                     ` Ihor Radchenko
  2022-06-08 14:43                       ` Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 14:38 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: monnier, luangruo, michael.albinus, acm, stefan, tsdh, akib, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I suspect that garbage collection is strongly affecting your loading
>> times. It is taking half of the time even with clean Emacs. I imagine
>> that GC sweeps are even more costly when you have a sizable allocated
>> memory.
>
> Which, btw, yet another problem: why does loading Org produce so much
> garbage?  Can this be kept in check somehow?

AFAIU, so much garbage is produced simply because nobody looked into
memory usage. It is not the easiest thing to track in Emacs.

Can it be kept in check? Maybe. We need to identify first which parts
are taking excess memory. org-link-set-parameters appears to be one of
the bottlenecks. I do not know any others.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 14:38                     ` Ihor Radchenko
@ 2022-06-08 14:43                       ` Stefan Monnier
  2022-06-08 15:44                         ` Ihor Radchenko
  2022-06-08 16:02                         ` Eli Zaretskii
  0 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-08 14:43 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Eli Zaretskii, luangruo, michael.albinus, acm, stefan, tsdh,
	akib, emacs-devel

>> Which, btw, yet another problem: why does loading Org produce so much
>> garbage?  Can this be kept in check somehow?
>
> AFAIU, so much garbage is produced simply because nobody looked into
> memory usage. It is not the easiest thing to track in Emacs.
>
> Can it be kept in check? Maybe. We need to identify first which parts
> are taking excess memory. org-link-set-parameters appears to be one of
> the bottlenecks. I do not know any others.

w.r.t spending time in the GC, since the GC is run based on the amount
of memory that has been allocated (as opposed to the amount of garbage
generated, for example. which we sadly can't know in advance), the "mem"
profiler (aka `M-x profilter-start RET mem RET`) can be very useful
since it does a sampling-based profiling where the "time" is measured in
number of byte allocated (or some approximation thereof).


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
       [not found]                                 ` <jwvr13z452j.fsf-monnier+emacs@gnu.org>
@ 2022-06-08 15:08                                   ` Ihor Radchenko
  2022-06-12 22:38                                   ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 15:08 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Alan Mackenzie, Tim Cross, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If you have some specific cases when Org mode alters Emacs defaults in a
>> way that bites the user, please give concrete examples. Otherwise, your
>> criticism is not very constructive.
>
> [ Alan gave one or two concrete examples of things that bit him.  ]

Alan gave two concrete examples:

>>> I put org.org into a buffer.  I saw lots of lines ending in ..., which is
>>> fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
>>> in a new window.  Brilliant!  Did I mention I hate context dependent
>>> commands?  So I have to type M-x outline-show-subtree, which works.  I
>>> read the screenful of text, then type C-S-<down> to scroll it up.
>>> Instead of my standard scrolling command, I get an error message about
>>> "Not at a clock log".  Org mode has stolen the key sequences
>>> C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

where he
1. Complains about major mode shadowing key binding reserved for major
   modes, according to D.2 Key Binding Conventions section of the Elisp
   manual:
>      • Sequences consisting of ‘C-c’ followed by a control character or a
>     digit are reserved for major modes.

2. C-S-<down>, which is a valid point. However, arrow commands is one of
   the few core concepts used in Org major mode.
   I am not sure if re-binding C-S-<down> is so much of a sin on Org
   mode side. Emacs manual 49.3 Customizing Key Bindings says:

>         Since most modes define their own key bindings, activating a mode
>      might override your custom key bindings.  A small number of keys are
>      reserved for user-defined bindings, and should not be used by modes, so
>      key bindings using those keys are safer in this regard.  The reserved
>      key sequences are those consisting of ‘C-c’ followed by a letter (either
>      upper or lower case), and function keys <F5> through <F9> without
>      modifiers (*note Modifier Keys::).

   As a compromise, Org may provide some kind of read-only mode just to
   navigate the document. The arrow bindings can be disabled then. They
   are for editing.

>>> I see many of these bindings as context dependent, with the name of the
>>> function named after the key sequence, not the functionality - e.g.
>>> org-shiftcontroldown.  That is not the way the rest of Emacs is
>>> constructed.  I hate context dependence (except when it is properly
>>> implemented, e.g. by major modes).

    Only a handful of bindings are context dependent. In particular,
    control/shift/meta-arrow keys, <TAB>, and C-c C-c. They are most
    frequently used when editing and navigating Org buffers.

    It would help if "properly implemented" were defined more precisely
    in the above statement.

> This is not a criticism, just a description of how Org is perceived from
> the side of "old-time Emacs users who aren't Org users".
> The key in what you wrote above is the "yes, many of them", which means
> that even tho those tweaks are minor they sum up to something that
> old-time users will almost inevitably bump into.
>
> It's not a problem for Org itself, and there are good reasons for each
> one of those tweaks, I'm sure, but it does create opposition to using
> Org "in core", such as in etc/NEWS.

I understand. However, such criticism is not helpful. Only concrete
examples can lead to actual Org improvements in this area.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 14:15                               ` Eli Zaretskii
@ 2022-06-08 15:15                                 ` Ihor Radchenko
  2022-06-08 16:16                                   ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 15:15 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Wed, 08 Jun 2022 21:12:11 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> - 236 org related key bindings - far less than 785 Of these, a number are
>> >> - actually remapping of existing bindings, so not new ones. This leaves 208.
>> >
>> > ??? How is remapping "not new"?  It takes some very common Emacs
>> > commands and redirects them to different commands.  E.g., 'open-line'
>> > now does something quite different.  This means the user should either
>> > go learn what the Org commands do, or be prepared to be surprised.
>> 
>> There are 3 main purposes of remappings in Org:
>
> I'm sure there are good reasons for that.  My point is that such
> remappings effectively force the user to re-learn the commands he/she
> is very familiar with.  So it's a non-trivial burden.

Not really. The remappings usually intend to re-implement the usual
expected Emacs behavior inside Org. It's just that it is not always
possible using the built-in functions. Hence, we implement a layer on
top.

There should not be anything to learn with regard to remapped commands
given that Org defaults are not changed.

The difference is that things like, say, paragraph-separate, are not
even noticed by users of various major modes. Org had to remap the
commands, which is more immediately visible.

>> >> I said that for a read only buffer, many of the org key bindings are not
>> >> relevant as they relate to features which are not pertinent to a read only org
>> >> buffer. Any bindings relating to babel, todo management, time management,
>> >> agendas etc have no relevance when reading a readme.org file.
>> >
>> > Then why does Org define them in that case?
>> 
>> I am not very sure what is the problem with the number of bindings. It
>> is not different from Emacs itself.
>
> The difference is that we had years or decades to get used to the
> Emacs defaults, and once Org is turned on in a buffer, one has a lot
> of new stuff to get used to.  Unless Org is used constantly, you will
> forget most of those changes till the next time, so this re-learning
> experience will be repeated every time.

Isn't it the same for any other major mode?

> It isn't a catastrophe, of course, but we should recognize this as an
> issue, especially if many of the bindings aren't needed.

I am not sure what you mean by aren't needed. They are needed if you use
their functionality. They are not needed if you don't.

There is no doubt that you do not need most of the bindings just to
navigate Org files or do basic editing. You do not need to learn those
other bindings either.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 14:43                       ` Stefan Monnier
@ 2022-06-08 15:44                         ` Ihor Radchenko
  2022-06-08 17:37                           ` Stefan Monnier
  2022-06-08 16:02                         ` Eli Zaretskii
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-08 15:44 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Eli Zaretskii, luangruo, michael.albinus, acm, stefan, tsdh,
	akib, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Which, btw, yet another problem: why does loading Org produce so much
>>> garbage?  Can this be kept in check somehow?
>>
>> AFAIU, so much garbage is produced simply because nobody looked into
>> memory usage. It is not the easiest thing to track in Emacs.
>>
>> Can it be kept in check? Maybe. We need to identify first which parts
>> are taking excess memory. org-link-set-parameters appears to be one of
>> the bottlenecks. I do not know any others.
>
> w.r.t spending time in the GC, since the GC is run based on the amount
> of memory that has been allocated (as opposed to the amount of garbage
> generated, for example. which we sadly can't know in advance), the "mem"
> profiler (aka `M-x profilter-start RET mem RET`) can be very useful
> since it does a sampling-based profiling where the "time" is measured in
> number of byte allocated (or some approximation thereof).

I know the 'mem profiler. It can help in some cases. That's how I
noticed org-link-set-parameter.

However, here is one recent memory profiler report:

  1,307,745,624  23%                            - org-persist-load
  1,307,726,070  23%                             - org-persist-read
  1,281,385,671  22%                              - seq-find
  1,281,385,671  22%                               - seq-do
  1,281,385,671  22%                                - mapc
  1,281,385,671  22%                                 - #<compiled 0x43febaeb8a58606>
  1,281,385,671  22%                                  - #<compiled -0x148951e07c95fd1a>
  1,281,385,671  22%                                   - run-hook-with-args-until-success
        333,712   0%                                    + org-element--cache-persist-before-read

Unless I misinterpret things, run-hook-with-args-until-success is
allocating memory. However, that hook contains a single function which
is not allocating much according to the profiler.

Moreover, I know myself that the actual memory allocation is done down
in org-perist-read when running (read (current-buffer)). I got memory
profiler wrong significant amount of time to learn not to trust it too
much.

cpu profiler is a bit more robust in this regard. At least not until I
C-g in the middle of the profiling.

Best,
Ihor





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 14:43                       ` Stefan Monnier
  2022-06-08 15:44                         ` Ihor Radchenko
@ 2022-06-08 16:02                         ` Eli Zaretskii
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-08 16:02 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: yantar92, luangruo, michael.albinus, acm, stefan, tsdh, akib,
	emacs-devel

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  luangruo@yahoo.com,
>   michael.albinus@gmx.de,  acm@muc.de,  stefan@marxist.se,  tsdh@gnu.org,
>   akib@disroot.org,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 10:43:44 -0400
> 
> w.r.t spending time in the GC, since the GC is run based on the amount
> of memory that has been allocated (as opposed to the amount of garbage
> generated, for example. which we sadly can't know in advance), the "mem"
> profiler (aka `M-x profilter-start RET mem RET`) can be very useful
> since it does a sampling-based profiling where the "time" is measured in
> number of byte allocated (or some approximation thereof).

Did you actually try this, and could indeed deduce where most of the
garbage is produced?

The problem with your idea, IME, is that Emacs calls memory-allocation
functions from code that has nothing to do with consing Lisp objects,
and it does that _a_lot_.  So given a "memory" profile it is not very
easy to see which of the parts are related to GC, and many times the
hot spots aren't.

But it doesn't hurt to try what you suggest, maybe we'd get a lucky
break in this case.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 15:15                                 ` Ihor Radchenko
@ 2022-06-08 16:16                                   ` Eli Zaretskii
  2022-06-09  9:15                                     ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-08 16:16 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 08 Jun 2022 23:15:16 +0800
> 
> >> There are 3 main purposes of remappings in Org:
> >
> > I'm sure there are good reasons for that.  My point is that such
> > remappings effectively force the user to re-learn the commands he/she
> > is very familiar with.  So it's a non-trivial burden.
> 
> Not really. The remappings usually intend to re-implement the usual
> expected Emacs behavior inside Org. It's just that it is not always
> possible using the built-in functions. Hence, we implement a layer on
> top.
> 
> There should not be anything to learn with regard to remapped commands
> given that Org defaults are not changed.

Are you sure this is always true?  There are several dozens of
remapped commands; did you audit all of them?

And anyway, even if what you say is 110% true, how am I as a user to
know that up front?  I'm used to read the documentation of every
command I don't already know by heart, so when faced with such massive
remapping, I have quite some reading to do before I can feel myself at
ease.

And please note that, unlike Alan, I _do_ use Org, just not very
often, at least these days.  So what I'm sating doesn't come from the
POV of an anti-Org user.  I _want_ Org to be easier and less demanding
to use.

> > The difference is that we had years or decades to get used to the
> > Emacs defaults, and once Org is turned on in a buffer, one has a lot
> > of new stuff to get used to.  Unless Org is used constantly, you will
> > forget most of those changes till the next time, so this re-learning
> > experience will be repeated every time.
> 
> Isn't it the same for any other major mode?

No, not IME.  Show me another general-purpose editing mode that
defines so many key bindings.  The only modes that get close are those
where text is read-only, so normal editing is impossible anyway.  And
even those leave alone basic movement commands, like C-S-<up> which
Alan mentioned.

> > It isn't a catastrophe, of course, but we should recognize this as an
> > issue, especially if many of the bindings aren't needed.
> 
> I am not sure what you mean by aren't needed.

Ask Tim Cross: the claim that most of the 230 bindings I counted
aren't needed comes from him.

> There is no doubt that you do not need most of the bindings just to
> navigate Org files or do basic editing. You do not need to learn those
> other bindings either.

Then why bind them by default? why not wait until I actually need that
functionality?  If that's hard or impossible to detect automatically,
let the user say so.  For example, I could envision a minor mode that
enables Org-Babel and binds the corresponding commands to keys.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-05 10:26             ` Tassilo Horn
                                 ` (4 preceding siblings ...)
  2022-06-05 11:59               ` Akib Azmain Turja
@ 2022-06-08 17:21               ` Yoni Rabkin
  5 siblings, 0 replies; 376+ messages in thread
From: Yoni Rabkin @ 2022-06-08 17:21 UTC (permalink / raw)
  To: Tassilo Horn
  Cc: Po Lu, Michael Albinus, Alan Mackenzie, Stefan Kangas,
	Akib Azmain Turja, emacs-devel

Tassilo Horn <tsdh@gnu.org> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>>>>> I think we should actively encourage using it for most types of
>>>>> documentation, certainly for README type files.
>>>>
>>>> I disagree entirely with you.  Org mode is highly complicated,
>>>> obscure (I've never managed to get a feel for what it does), and
>>>> difficult to learn (I've never managed it).  A text file is far
>>>> easier to read for those not familiar with org.  We're talking about
>>>> READMEs here.  They're typically 20 to 100 lines long.  A text file
>>>> is ideal for these.
>>>
>>> 1+. README is a text file which is consulted in order to get
>>> information about the related software. There's no rule to visit it
>>> by Emacs only.
>
> People nowadays write README.org and README.md files instead of
> plain-text READMEs because the packages' repositories are hosted on some
> forge (sr.ht, github, gitlab, gitea, whatever) which supports rendering
> those formats in a nice way.  So the choice of format is natural for
> package authors.  And they frequently contain markup like headings,
> bullet lists, bold, italics, and code snippets which simply look much
> better than their text-only alternative (even in their
> editing/non-rendered versions).

I personally don't think that org is nice, or natural. Those are
subjective statements. Relying on the fact that lots of people use org
and formatted READMEs on github and such is not a justification either;
lots and lots of people use windows...

Making Emacs look like whatever happens to be popular shouldn't be done
at the expense of existing users. Org is popular, yes, but Emacs
shouldn't be a popularity contest.

The fact that there are people, like me, asking to be able to read a
text file as a text file should be respected. I would be fine with being
able to:

(setq packages-org-to-text t)

Or similar, and never have to to be confronted with ellipses and foreign
(to me) key-bindings when I'm trying to read a few lines of text.

Even better, if I had a simple:

(org-disable-org-mode t)

From then on I won't have to worry about being forced to read plain text
through a reincarnated implementation of "hypertext" from the 80s.

First do no harm.

-- 
   "Cut your own wood and it will warm you twice"



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 15:44                         ` Ihor Radchenko
@ 2022-06-08 17:37                           ` Stefan Monnier
  2022-06-09  3:11                             ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-08 17:37 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Eli Zaretskii, luangruo, michael.albinus, acm, stefan, tsdh,
	akib, emacs-devel

> Moreover, I know myself that the actual memory allocation is done down
> in org-perist-read when running (read (current-buffer)). I got memory
> profiler wrong significant amount of time to learn not to trust it too
> much.

Indeed, the memory profiler is not very reliable.  Some of that is
a conscious implementation choice (e.g. we take samples when a whole new
16KB block is allocated to be divided into several smaller objects
rather than for every object allocation).  Other problems are things for
which I don't have an explanation (and haven't spent the time to try
and track them down).

I've still found it useful to help figure out in which part of the code
most of the allocation takes place, but you need to supplement it
without your own knowledge of what the code does  :-(


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 21:51                             ` Convert README.org to plain text README while installing package Tim Cross
@ 2022-06-08 19:26                               ` chad
  0 siblings, 0 replies; 376+ messages in thread
From: chad @ 2022-06-08 19:26 UTC (permalink / raw)
  To: Tim Cross; +Cc: Alan Mackenzie, EMACS development team

[-- Attachment #1: Type: text/plain, Size: 3239 bytes --]

On Tue, Jun 7, 2022 at 7:04 PM Tim Cross <theophilusx@gmail.com> wrote:

> So going back to my original post, which was simply saying that
> implying you would get 784 additional key bindings just because you open
> an org file was misleading because you simply don't get that many
> additional key bindings when you open an org file - in fact, you get
> only about 1/3 of that many.
>

I'll add to this statement: the vast majority of those keybindings aren't
relevant when viewing a read-only org file, which is what we're talking
about here. In fact, I used Org (very lightly) for a few years with exactly
these two extra keybindings: Control-Return and Meta-Return. I also used
Tab to expand/compact headings, so perhaps three, depending on your usage
(I was coming from outline-mode, so this wasn't new.) This is a read/write
usage, not a read-only usage.

Everything else is basically the same. There are a few places where Org
changes the meaning of keys to accommodate Org features, but most of these
are DWIM-ish adaptations, or they're guarded by an Org warning that Org
uses the keys for something else (for example, shift-selection). A few
years in, I discovered that Org would automatically reflow the plain-text
tables I sometimes used, and I learned another use for the Tab key; as near
as I can tell, this feature had been present for months if not years
without ever bothering me.

These days, most of my work involves creative writing and light number
crunching, not coding; as part of that, I use dozens of text files over
several active topics. I keep these in a simple hierarchy of plain Org
files. These load in emacs instantaneously. I search through them using
standard text-searching tools (mostly find and grep, sometimes trying out
things like fd, ack, ripgrep, and deadgrep). I once started reading the Org
manual (several years ago) and bounced off pretty quickly; I haven't been
back in years (although I hear that the manual has improved quite a lot
since then, and it's on my overly-long todo list for "someday".)

I say all this not to try to convince anyone to try Org; in fact, if you're
not already using outline-mode a bunch, then you're probably only
interested in Org if there's a specific use that you'd like for emacs
(agenda, note-keeping, literate programming, etc). That said, there are
people, especially elisp package authors, who use it and like it and will
continue to use it to write package documentation. In the (few) packages
that I personally load, most have a README.org, a few have a README.md, and
none have a plain README.

Put another way, this isn't about encouraging authors to use Org or not;
the authors have chosen already. For people who don't really want to try to
"learn Org": I hear and understand. I've been there myself; time is usually
rarer than interesting ideas to explore. I'm writing this in the hope that
it encourages people to not be afraid of looking at .org files even if you
don't have the time, interest, or inclination to "learn Org", because while
the ocean is indeed (seemingly?) quite deep, the shallow end is quite
comfortable. In the meantime, we can all hope that the arrival of the
"modular Org" is sooner rather than later.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 3819 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-07 11:21                           ` Eli Zaretskii
  2022-06-08 13:12                             ` Ihor Radchenko
@ 2022-06-08 19:34                             ` Tim Cross
  2022-06-09  5:18                               ` Eli Zaretskii
  2022-06-09 19:48                               ` Alan Mackenzie
  1 sibling, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-08 19:34 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: acm, emacs-devel


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: acm@muc.de, emacs-devel@gnu.org
>> Date: Tue, 07 Jun 2022 16:14:52 +1000
>> 
>> > I can tell you what I see with my very simple 1570-line Org file,
>> > which just has some todo items organized in 4-level hierarchy:
>> >
>> >   . ~100 keys that look "general Org" to me
>> >   . ~30 keys that Org remaps to its s own commands, like 'yank' and
>> >     'backward-sentence'
>> >   . ~40 keys bound to org-babel-SOMETHING -- no idea why these are
>> >     bound by default, but they are there
>> >   . ~60 more keys that I'm not sure whether they are "basic" or not,
>> >     but they are all bound by default, just because I visited an Org
>> >     file 
>> >
>> > So "just" 230 key bindings.  And this is without any minor/add-on Org
>> > mode/feature enabled, at least according to "C-h m".
>> >
>> > Do you see something different?  Are you still saying that it's not a
>> > lot, or that it's "based on ignorance and paranoia"?  If so, please
>> > point out where I'm ignorant and/or paranoiac, because I'd really like
>> > to know.
>> 
>> I think Alan's response has pretty much confirmed what I was saying. Alan has
>> made it fairly clear that his real underlying concern is about a stealthy
>> strategy to get org mode more intrinsically tied into Emacs and in the end,
>> making it so critical to normal operation that everyone will be forced to learn
>> org mode regardless. I don't necessarily agree with such an argument, but think
>> having a debate around that would be more up-front and in some ways honest
>> compared to what appear to be somewhat contrived alternative arguments about
>> excessive key bindings and added complexity. My characterisation of this all
>> being fear and paranoia may have been overstating things, but then again I'm not
>> sure. There certainly does seem to be considerable fear involved. I don't know
>> about paranoia, but I have not seen any evil plan to have org mode assimilate
>> Emacs as if it was the Borg and I've seen no discussions on the org list about
>> ways to get org more ingrained in Emacs or make Emacs more dependent on org.
>> 
>> With respect to ignorance, Alan made it quite clear he has not spent
>> the time or effort to learn org mode, so he is ignorant about its
>> operation and lacks any real understanding of its operation. He has made
>> assertions about the number of key bindings which are wildly over
>> inflated and about required levels of complexity which simply would not
>> be necessary with the org mode rendering of a read only buffer. In
>> short, he has made a lot of assumptions about both the key bindings and
>> complexity based on only a fairly cursory scan of the manual and very
>> limited experience. 
>> 
>> All that appears to have occurred here is that someone noted that packages often
>> had a readme.org file and wondered if this should be translated into just a
>> plain readme file. Someone else suggested we could have package descriptions
>> render the readme.org file using org and not require export into plain readme
>> file and mentioned some possible advantages this could have wrt navigation,
>> rendering of tables, links and images or inclusion of source code snippets,
>> examples etc.
>> 
>> >From this point, some somewhat extreme claims and arguments have been thrown in.
>> One was Alan's claim that org would introduce 785 new key bindings, many of
>> which not bounmd to C-c and which would presumably either pollute the key space
>> or conflict with existing bindings. All I have done is point out this claim was
>> inaccurate and overstated due to an overly simplistic analysis of what is in the
>> key index of the org manual. 
>> 
>> it should be noted that if we were to add 'native' rendering of readme.org
>> files, this would likely be done with an org minor mode rather than a full blown
>> org mode as very little of the overall org functionality would be required in
>> this situation. All that would be required is org rendering support and org
>> support for navigation (folding/unfolding, following links and possibly
>> list/table navigation). All the advanced org functionality relating to babel,
>> noweb, todo and time management, agendas, etc have no relevance in this
>> scenario. LIkewise, if this functionality was provided via a minor mode, it
>> would also be possible for people to disable that mode and be left with just the
>> readme.org file, which is just plain text and quite readable.
>> 
>> Alan's claim was 785 additional key bindings, many of which not bound to C-c. 
>
> The above is completely irrelevant to my questions.
>
>> Running Emacs wiht -q and opeing up an org file, I find the following 
>> 
>> - 236 org related key bindings - far less than 785 Of these, a number are
>> - actually remapping of existing bindings, so not new ones. This leaves 208.
>
> ??? How is remapping "not new"?  It takes some very common Emacs
> commands and redirects them to different commands.  E.g., 'open-line'
> now does something quite different.  This means the user should either
> go learn what the Org commands do, or be prepared to be surprised.

This are not new key bindings, they are redefinitions of existing key
bindings to make them work in an org mode context in a wayu which is
consistent with user expectations. They are not new in the sense of them
being a key binding which did not exist prior to loading org mode. 


>
>> I said that for a read only buffer, many of the org key bindings are not
>> relevant as they relate to features which are not pertinent to a read only org
>> buffer. Any bindings relating to babel, todo management, time management,
>> agendas etc have no relevance when reading a readme.org file.
>
> Then why does Org define them in that case?
>

because standard org mode is not typically used in read only buffers.
How org is defined now and how it would be defined/loaded when it comes
to a specialised use case as discussed here are completely different. As
I said in previous posts, I would expect that you would define an org
minor mode which would only include the key bindings which would be
appropriate for the use case of viewing a read-only org file. 

>> Really, the only
>> key bindings of any relevance are navigation related bindings (including
>> opening/closing folds, following links, etc). Removing all unnecessary org
>> bindings leave us with just 77 new bindings. Of these remaining bindings, quite
>> a few could also be removed, but I've left them in as I only wanted to remove
>> those which would clearly be of no benefit in a read only buffer rendering of a
>> readme.org file. In an org minor mode setup for rendering of readme.org (and
>> possibly other read-only rendering of org files), I estimate at least another 30
>> bindings could be removed because they have no real benefit or are of only
>> marginal benefit. Creating an org minor mode for this purpose which only
>> introduced 20 or so new bindings would easily be possible. The key point is that
>> claims of 785 new bindings arfe incorrect and misleading. Even being very
>> conservative, we are really talking about 10% of the number being incorrectly claimed.  
>> if we introduced a minor mode for this, we are talking likely less than 20 or so.
>
> I'm sorry, but IMO this hand-waves away a real problem by ignoring the
> problematic UX of suddenly having more than 200 keybindings defined
> just because I visited a small file.  I hoped you will present some
> serious analysis of the issue, and perhaps even show me where I'm
> wrong in my conclusions.  Sadly, it sounds like all you wanted to do
> is to prove that you are right, the facts notwithstanding.

Sorry, but it seems you are the one trying to drag this off into a
different argument about whether org mode is too large or defines too
much or loads too much. As stated multiple times in my previous posts in
this thread, should org mode be used in the context being discussed, you
would define a new org minor mode whic only included the functionality
and key bindings relevant for this use case. You would not just use a
full org mode for this situation.

>
> You accuse Alan in extremism, but your own argumentation is another
> example of a similar extremism -- just in the opposite direction.
>

I have never said that 200 key bindings wasn't a lot. All I've said is
that 200 key bindings is a LOT less than the 784 being claimed. I went
further to state that in a specialised read-only org mode, you would be
able to reduce that number to around 50 or possibly even fewer. I also
reject what was stated about the org mode key bindings polluting the
global key space. I find no evidence of such pollution and in fact
believe org mode is very good at not doing so. I find very few (if any)
examples of org doing global key binding and in fact have to define new
personal bindings when I want global access to org functionality (such
as org capture or the org agenda).  

> This doesn't facilitate useful discussions of what I think is a real
> problem.  Any unbiased observer should agree that adding 200+ key
> bindings when a small file is visited is a problem that needs to be
> solved.  Org is too large for such simple jobs, and should be
> refactored to avoid loading everything under the sun when a README is
> visited.  And we should work towards solving it, not sweep it under
> the carpet.
>

which is exactly why I was saying that should you want org rendering of
small read only files, you would create an org minor mode which only
loaded the functionality and key bindings that would be relevant in that
context. It is only you who are assuming that a full org mode would be
used in this context. 

Nobody has tried to sweep anything under the carpet. Right back in
my very 1st post to this thread, I stated that should org be used to
render content in things like a help buffer or package description
whhich would presumably be a read only presentation, you would want an
org minor mode which only loads the functionality, including appropriate
key bindings which made sense in that context. This sentiment was also
mirrored by others in the thread. . 

Questions about general org mode and whether it defines too many key
bindings or too many features when loaded is a completely separate
question. While there could be some important discussion which needs to
happen in that context, it has little to do with what was being
discussed here and even less relevance to what I was saying, whihc was
that claiming opening an org file would define 784 new key bindings and
would pollute the key space was an exaggeration and over statement of
what happens when you open an org file. Even your own example confirmed
this.   

As to whether defining 200 key bindings is too many - well I don't think
you can possibly say either way without significantly more analysis. It
also isn't at all given that adding 200 new key bindings in a new mode
is of itself problematic. There could be very good reasons to add that
many bindings. Besides, I'm not sure it even is a problem. Emacs has
lots of key bindings - the vast majority of which I never use. I don't
find this a problem and I'm not convinced just citing absolute binding
numbers in itself is evidence of a problem. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 17:37                           ` Stefan Monnier
@ 2022-06-09  3:11                             ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-09  3:11 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Eli Zaretskii, luangruo, michael.albinus, acm, stefan, tsdh,
	akib, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I've still found it useful to help figure out in which part of the code
> most of the allocation takes place, but you need to supplement it
> without your own knowledge of what the code does  :-(

Unfortunately, the profiler (both cpu and mem) is particularly useless
when I try to measure loading time.

For example, my last profiler run on Emacs-29 + native-comp yielded the
following:

Elapsed time: 0.505677s (0.295615s in 23 GCs)

Memory:

     32,661,616  98% - command-execute
     32,378,659  97%  - funcall-interactively
     28,537,695  86%   - eval-last-sexp
     28,537,695  86%    - elisp--eval-last-sexp
     28,470,377  86%     - progn
     28,470,377  86%      - let
     28,469,646  86%       - progn
     26,927,800  81%        - org-mode
     18,457,233  55%         - org-load-modules-maybe
     18,113,241  54%          + require
      7,294,107  22%         - byte-code
      4,198,028  12%          + require

CPU:

         251  79% - command-execute
         230  72%  - funcall-interactively
         225  70%   - eval-last-sexp
         225  70%    - elisp--eval-last-sexp
         225  70%     - progn
         225  70%      - let
         225  70%       - progn
         218  68%        - org-mode
         158  49%         - org-load-modules-maybe
         148  46%          - require
         148  46%           - byte-code
         129  40%            - require
         129  40%             - byte-code
         119  37%              - require
         119  37%               - byte-code

Sure, loading appears to take a lot of time, but loading of which file?
The profiler gives no answer whatsoever.

I went as far as stripping all the requires from org.el and removing all
but defun/defvar/defcustom statements. Still, the loading time was
almost 0.3 sec with no easy way to tell which part is creating the
bottleneck.

In summary, I see no other way of speeding up Org loading than splitting
org.el and other big files into modules and hoping for the best.

A practical interim solution could be doing (require 'org) in init.el
Given that Org mode is compiled and loaded on Emacs startup, the loading
generally takes a 0.3-0.5 seconds, which is much better compared to 5.1
seconds reported by Eli.

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 19:34                             ` Convert README.org to plain text README while installing package Tim Cross
@ 2022-06-09  5:18                               ` Eli Zaretskii
  2022-06-09 19:48                               ` Alan Mackenzie
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-09  5:18 UTC (permalink / raw)
  To: Tim Cross; +Cc: acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: acm@muc.de, emacs-devel@gnu.org
> Date: Thu, 09 Jun 2022 05:34:09 +1000
> 
> > ??? How is remapping "not new"?  It takes some very common Emacs
> > commands and redirects them to different commands.  E.g., 'open-line'
> > now does something quite different.  This means the user should either
> > go learn what the Org commands do, or be prepared to be surprised.
> 
> This are not new key bindings, they are redefinitions of existing key
> bindings to make them work in an org mode context in a wayu which is
> consistent with user expectations.

That's exactly my point: suddenly commands that I'm familiar with are
working in ways that are different, even if slightly different.  A
conscientious Emacs user will want/need to study the differences,
before using those remapped commands.

> They are not new in the sense of them being a key binding which did
> not exist prior to loading org mode.

This completely misses the point I explain above.  The _commands_
behave differently, and that is all that counts.

> >> Any bindings relating to babel, todo management, time management,
> >> agendas etc have no relevance when reading a readme.org file.
> >
> > Then why does Org define them in that case?
> 
> because standard org mode is not typically used in read only buffers.

Which is one more argument towards more modularity in Org, so that
only the relevant bindings and commands are defined for each use case.

> As to whether defining 200 key bindings is too many - well I don't think
> you can possibly say either way without significantly more analysis.

Which is why I suggested you do this kind of analysis.

> It also isn't at all given that adding 200 new key bindings in a new
> mode is of itself problematic. There could be very good reasons to
> add that many bindings.

You have just explained that many of them are not relevant, even for
editing simple enough Org files.  For example, everything related to
org-babel -- why do these bindings have to be available by default?
Many Org files have no code blocks at all.

> Besides, I'm not sure it even is a problem. Emacs has lots of key
> bindings - the vast majority of which I never use. I don't find this
> a problem and I'm not convinced just citing absolute binding numbers
> in itself is evidence of a problem.

Well, I _am_ convinced, looking at this from the POV of the Emacs
maintainer.  I also think almost everyone agrees that too many
unneeded keybindings are a problem in general -- witness the removal
of some veteran global bindings in recent Emacs version, which is
evidence that we consider even half a dozen less-then-useful bindings
not a good thing.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 16:16                                   ` Eli Zaretskii
@ 2022-06-09  9:15                                     ` Ihor Radchenko
  2022-06-09  9:38                                       ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-09  9:15 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> Not really. The remappings usually intend to re-implement the usual
>> expected Emacs behavior inside Org. It's just that it is not always
>> possible using the built-in functions. Hence, we implement a layer on
>> top.
>> 
>> There should not be anything to learn with regard to remapped commands
>> given that Org defaults are not changed.
>
> Are you sure this is always true?  There are several dozens of
> remapped commands; did you audit all of them?

I have audited a couple of them and relied on a good faith to other Org
devs + absence of bug reports relevant to the raised concern.
Rather then re-checking the remapped functions, I'd rather just fix any
concrete bug reports about misbehavior, if such reports arise.

> And anyway, even if what you say is 110% true, how am I as a user to
> know that up front?  I'm used to read the documentation of every
> command I don't already know by heart, so when faced with such massive
> remapping, I have quite some reading to do before I can feel myself at
> ease.
>
> And please note that, unlike Alan, I _do_ use Org, just not very
> often, at least these days.  So what I'm sating doesn't come from the
> POV of an anti-Org user.  I _want_ Org to be easier and less demanding
> to use.

While I understand the possible fear of unexpected or unnatural
behavior of the remapped command, I'd like to point out that command
behavior can span over quite a significant range even without remapping.
Just the fact that some major mode does not use remapping, does not mean
that you are guaranteed to get the normal command behavior.

Some major modes rely on post-command-hook instead of remapping, which
not only modifies the original command behavior in unspecified ways, but
is also not directly visible from the mode bindings. Or consider changes
in filter-buffer-substring-function and how it can modify killing text.

So, I'd say that your concern is not really justified. There is
generally no reliable way to tell if an arbitrary major mode modifies
standard command behavior just by looking at its key bindings. You have
to rely on faith and report bugs if you stumble upon them.

>> > The difference is that we had years or decades to get used to the
>> > Emacs defaults, and once Org is turned on in a buffer, one has a lot
>> > of new stuff to get used to.  Unless Org is used constantly, you will
>> > forget most of those changes till the next time, so this re-learning
>> > experience will be repeated every time.
>> 
>> Isn't it the same for any other major mode?
>
> No, not IME.  Show me another general-purpose editing mode that
> defines so many key bindings.  The only modes that get close are those
> where text is read-only, so normal editing is impossible anyway.

I am not sure what you mean by general-purpose. Auctex binds a decent
number of key-bindings. VHDL mode binds even more (I just looked into
one of the largest mode built-in prog modes). markdown-mode binds over
100 keys not counting prefix maps.

>   And even those leave alone basic movement commands, like C-S-<up>
> which Alan mentioned.

C-S-<up> is not a basic movement command. By default, it is activating
region. Without shift-select-mode, it is not bound to anything. Org does
provide support for shift-select-mode, but we never got overwhelming
requests to enable this support by default.

C-<up>/<down>, which is the actual basic movement command is not
overridden by Org.

AFAIK, the only shadowed command is M-<right>/<left> with its other
C-<right>/<left> binding being untouched. However, I am not even sure
what Org can do about this. Arrow bindings are one of the most core and
frequently used commands. Removing them will dissatisfy existing users.

Note that Emacs dedicates C-c + letter for user bindings. Anything else
is not guaranteed to be not shadowed by major/minor modes. If users
absolutely want they personal bindings to be preferred, there is always
override-global-map.

>> > It isn't a catastrophe, of course, but we should recognize this as an
>> > issue, especially if many of the bindings aren't needed.
>> 
>> I am not sure what you mean by aren't needed.
>
> Ask Tim Cross: the claim that most of the 230 bindings I counted
> aren't needed comes from him.

He was referring to reading Org files without editing.
You can argue the same for reading pretty much any other major-mode
files. Most bindings will not be needed.

>> There is no doubt that you do not need most of the bindings just to
>> navigate Org files or do basic editing. You do not need to learn those
>> other bindings either.
>
> Then why bind them by default? why not wait until I actually need that
> functionality?  If that's hard or impossible to detect automatically,
> let the user say so.  For example, I could envision a minor mode that
> enables Org-Babel and binds the corresponding commands to keys.

Org-babel bindings are needed as soon as you create a source block
inside the Org file. Are you proposing to track changes in buffer and
autoload babel as soon as a source block appears inside the buffer? I'd
say that conventional approach is binding the relevant commands and
letting Emacs to do autoloading once those commands are called.

Would it help if we hide the babel-related stuff under dedicated prefix
map?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-09  9:15                                     ` Ihor Radchenko
@ 2022-06-09  9:38                                       ` Eli Zaretskii
  2022-06-11  3:52                                         ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-09  9:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 09 Jun 2022 17:15:05 +0800
> 
> > Are you sure this is always true?  There are several dozens of
> > remapped commands; did you audit all of them?
> 
> I have audited a couple of them and relied on a good faith to other Org
> devs + absence of bug reports relevant to the raised concern.
> Rather then re-checking the remapped functions, I'd rather just fix any
> concrete bug reports about misbehavior, if such reports arise.

Didn't this discussion raise such "bug reports"?  Why not consider
what several people here said as such a bug report?  The difference is
that these reports come from people who use Org rarely, so the
concerns are different.  But if the Org developers want to have Org
adopted more widely, I think they should listen to these concerns, not
reject them.

> While I understand the possible fear of unexpected or unnatural
> behavior of the remapped command, I'd like to point out that command
> behavior can span over quite a significant range even without remapping.
> Just the fact that some major mode does not use remapping, does not mean
> that you are guaranteed to get the normal command behavior.

Most major modes don't remap global and frequently-used commands.
Those that do are problematic, and should be fixed rather than be used
as examples of what's "kosher".

> Some major modes rely on post-command-hook instead of remapping, which
> not only modifies the original command behavior in unspecified ways, but
> is also not directly visible from the mode bindings. Or consider changes
> in filter-buffer-substring-function and how it can modify killing text.

And that is in your opinion a Good Thing?  I don't think it is, and
therefore don't consider such problematic behavior a good example for
development.

> So, I'd say that your concern is not really justified.

??? Just because "others do that"?

> There is generally no reliable way to tell if an arbitrary major
> mode modifies standard command behavior just by looking at its key
> bindings. You have to rely on faith and report bugs if you stumble
> upon them.

I don't see how this solves the concerns.  We should still try to
minimize such problematic behaviors as much as possible.

> > No, not IME.  Show me another general-purpose editing mode that
> > defines so many key bindings.  The only modes that get close are those
> > where text is read-only, so normal editing is impossible anyway.
> 
> I am not sure what you mean by general-purpose.

Modes for editing mostly human-readable text.

> Auctex binds a decent number of key-bindings. VHDL mode binds even
> more (I just looked into one of the largest mode built-in prog
> modes). markdown-mode binds over 100 keys not counting prefix maps.

Two of these aren't part of Emacs, and VHDL is an example of
programming language, where text is not really for human consumption.

> >   And even those leave alone basic movement commands, like C-S-<up>
> > which Alan mentioned.
> 
> C-S-<up> is not a basic movement command.

Of course, it is: it moves by paragraphs.

> By default, it is activating
> region. Without shift-select-mode, it is not bound to anything.

But shift-select-mode is on by default.

> > Then why bind them by default? why not wait until I actually need that
> > functionality?  If that's hard or impossible to detect automatically,
> > let the user say so.  For example, I could envision a minor mode that
> > enables Org-Babel and binds the corresponding commands to keys.
> 
> Org-babel bindings are needed as soon as you create a source block
> inside the Org file. Are you proposing to track changes in buffer and
> autoload babel as soon as a source block appears inside the buffer?

That's one way.  Another one is to have a minor mode that users could
turn on when they need it (or turn it on by default in their init
files if they need it a lot).

> Would it help if we hide the babel-related stuff under dedicated prefix
> map?

Probably.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08 19:34                             ` Convert README.org to plain text README while installing package Tim Cross
  2022-06-09  5:18                               ` Eli Zaretskii
@ 2022-06-09 19:48                               ` Alan Mackenzie
  2022-06-11  4:09                                 ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-09 19:48 UTC (permalink / raw)
  To: Tim Cross; +Cc: Eli Zaretskii, emacs-devel

Hello, Tim.

On Thu, Jun 09, 2022 at 05:34:09 +1000, Tim Cross wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

[ .... ]

> >> Alan's claim was 785 additional key bindings, many of which not
> >> bound to C-c. 

It wasn't a "claim" it was a statement of fact.  These 784 bindings are
DOCUMENTED in org.info.  You attempted to explain it away by saying, at
first vaguely, that not all of them "come into play".  Only later did you
make that meaningful by saying that not all of them get loaded at any
time.

Many of these binding are not bound to C-c C-<letter> or other major mode
sequences.

> > The above is completely irrelevant to my questions.

> >> Running Emacs wiht -q and opeing up an org file, I find the following 

> >> - 236 org related key bindings - far less than 785.

236 and 784 are both good approximations to infinity when it comes to key
binding numbers.  Even 236 is barely manageable for anybody wanting to
learn the mode in the way I would.

[ .... ]

> > Then why does Org define them [lots of key bindings] in that case?

> because standard org mode is not typically used in read only buffers.
> How org is defined now and how it would be defined/loaded when it comes
> to a specialised use case as discussed here are completely different.
> As I said in previous posts, I would expect that you would define an
> org minor mode which would only include the key bindings which would be
> appropriate for the use case of viewing a read-only org file. 

I think the use case is that of a small org file rather than a read-only
one.  For some value of "small".

[ .... ]

> > I'm sorry, but IMO this hand-waves away a real problem by ignoring the
> > problematic UX of suddenly having more than 200 keybindings defined
> > just because I visited a small file.  I hoped you will present some
> > serious analysis of the issue, and perhaps even show me where I'm
> > wrong in my conclusions.  Sadly, it sounds like all you wanted to do
> > is to prove that you are right, the facts notwithstanding.

> Sorry, but it seems you are the one trying to drag this off into a
> different argument about whether org mode is too large or defines too
> much or loads too much.

That has been the main topic of this discussion for several rounds in the
thread.  I've found your responses very frustrating, feeling like you've
been broad brushing precise topics with vagueness, and like I'm dealing
with a politician rather than a hacker.  In a typical thread in this
mailing list, pin-point accuracy is rare, but participants attempt to
answer what the other person means rather than the precise literal
meaning of what was said (sometimes getting it wrong).  I don't feel this
has been happening in this part of this thread.

> As stated multiple times in my previous posts in this thread, ....

And overbearing with it.  Why?

> .... should org mode be used in the context being discussed, you would
> define a new org minor mode whic only included the functionality and
> key bindings relevant for this use case. You would not just use a full
> org mode for this situation.

The context being discussed is, I think, "small" org files rather than
read-only ones, whatever small means here.  I think we're all agreed that
full org mode is inappropriate for these.

> > You accuse Alan in extremism, but your own argumentation is another
> > example of a similar extremism -- just in the opposite direction.

> I have never said that 200 key bindings wasn't a lot. All I've said is
> that 200 key bindings is a LOT less than the 784 being claimed.

It's not a LOT less, both being unmanageably large.

> I went further to state that in a specialised read-only org mode, you
> would be able to reduce that number to around 50 or possibly even
> fewer. I also reject what was stated about the org mode key bindings
> polluting the global key space. I find no evidence of such pollution
> ....

I find plenty of evidence in org.info.

> and in fact believe org mode is very good at not doing so.

Go into org.info and count up all the "new" (as you have defined it
above) bindings which don't begin C-c C-<letter> or other major mode
sequences.  There are more of these than the total number of bindings a
typical mode has.

Examples:

* C-c /:                                 Sparse Trees.
* M-DOWN:                                Structure Editing.


> I find very few (if any) examples of org doing global key binding and
> in fact have to define new personal bindings when I want global access
> to org functionality (such as org capture or the org agenda).  

> > This doesn't facilitate useful discussions of what I think is a real
> > problem.  Any unbiased observer should agree that adding 200+ key
> > bindings when a small file is visited is a problem that needs to be
> > solved.  Org is too large for such simple jobs, and should be
> > refactored to avoid loading everything under the sun when a README is
> > visited.  And we should work towards solving it, not sweep it under
> > the carpet.

> which is exactly why I was saying that should you want org rendering of
> small read only files, you would create an org minor mode which only
> loaded the functionality and key bindings that would be relevant in that
> context. It is only you who are assuming that a full org mode would be
> used in this context. 

I think Eli has experienced this.  He is not "assuming" it at all.

> Nobody has tried to sweep anything under the carpet. Right back in
> my very 1st post to this thread, I stated that should org be used to
> render content in things like a help buffer or package description
> whhich would presumably be a read only presentation, you would want an
> org minor mode which only loads the functionality, including appropriate
> key bindings which made sense in that context. This sentiment was also
> mirrored by others in the thread. . 

> Questions about general org mode and whether it defines too many key
> bindings or too many features when loaded is a completely separate
> question.

Not at all.  Without org mode's massive numbers of bindings (several
hundred), the question of refactoring it wouldn't even have been an
issue.

> While there could be some important discussion which needs to happen in
> that context, it has little to do with what was being discussed here
> and even less relevance to what I was saying, which was that claiming
> opening an org file would define 784 new key bindings ....

That was not claimed.  The thing remarked upon was that there are 784 key
bindings in org mode.

[ .... ]

> As to whether defining 200 key bindings is too many - well I don't think
> you can possibly say either way without significantly more analysis. It
> also isn't at all given that adding 200 new key bindings in a new mode
> is of itself problematic.

Problems with it have been identified in this thread.

> There could be very good reasons to add that many bindings. Besides,
> I'm not sure it even is a problem. Emacs has lots of key bindings - the
> vast majority of which I never use. I don't find this a problem and I'm
> not convinced just citing absolute binding numbers in itself is
> evidence of a problem. 

Maybe not, but the problems are there nonetheless.

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-07 22:10                               ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Tim Cross
  2022-06-08  6:06                                 ` Org mode and Emacs Visuwesh
@ 2022-06-09 22:31                                 ` Richard Stallman
  2022-06-09 23:10                                   ` Tim Cross
                                                     ` (2 more replies)
  1 sibling, 3 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-09 22:31 UTC (permalink / raw)
  To: Tim Cross; +Cc: monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we are going to redesign Org mode syntax,
can we please add room for extensibility
so that an extension of Org mode syntax
could have all the features of Texinfo?

If the new syntax can express all the distinctions that Texinfo
can already express, that would make it possible to switch to it
for our documentation sources.

This needs to include conversion into some variant of TeX syntax so
that we can generate high-quality printed manuals through TeX.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-09 22:31                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
@ 2022-06-09 23:10                                   ` Tim Cross
  2022-06-10  5:59                                     ` Eli Zaretskii
  2022-06-12  0:42                                     ` Richard Stallman
  2022-06-10 13:56                                   ` Ihor Radchenko
  2022-06-10 19:32                                   ` Akib Azmain Turja
  2 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-09 23:10 UTC (permalink / raw)
  To: rms; +Cc: monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?
>
> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.


Nobody has said that org mode syntax was getting redesigned. What I said
was that we were working to clarify the existing syntax and remove
ambiguities and provide a standard parser. 

Making org mode syntax equivalent to texinfo syntax seems like a
mistake to me. The original idea was to have a light weight syntax which
is easyh to learn, not create a clone of texinfo. Besides, I suspect it
would be very difficult to maintain backwards compatibility. 

I"m also not convinced that org mode would be the right solution for
Emacs documentation. That is a completely different conversation and I'm
not convinced that org mode is a better answer wrt documentation than
texinfo. While there are certainly some areas where texinfo is showing
its age, it still works well for providing Emacs documentation within
the editor. There may be some role for org mode to augment what is
provided by tgexinfo, but again, that is a whole different conversation.
Note also that the previous thread was about handling of existing org
files within Emacs package ecosystem, not about replacing org
documentation with org mode. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-09 23:10                                   ` Tim Cross
@ 2022-06-10  5:59                                     ` Eli Zaretskii
  2022-06-10  6:20                                       ` Ihor Radchenko
  2022-06-12  0:42                                     ` Richard Stallman
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-10  5:59 UTC (permalink / raw)
  To: Tim Cross; +Cc: rms, monnier, acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 09:10:50 +1000
> 
> I"m also not convinced that org mode would be the right solution for
> Emacs documentation.

Then why Org's own manual is written in Org instead of Texinfo?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-10  5:59                                     ` Eli Zaretskii
@ 2022-06-10  6:20                                       ` Ihor Radchenko
  2022-06-10  6:44                                         ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-10  6:20 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Tim Cross, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I"m also not convinced that org mode would be the right solution for
>> Emacs documentation.
>
> Then why Org's own manual is written in Org instead of Texinfo?

I do not know if Org mode would be a good solution for Emacs
documentation. However, compared to texinfo, our manual is lacking index
entries in all the exported versions but texinfo.

The indexes in pdf version of the manual are actually generated by
org -> texinfo -> tex -> pdf.

AFAIK.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-10  6:20                                       ` Ihor Radchenko
@ 2022-06-10  6:44                                         ` Eli Zaretskii
  2022-06-11  4:49                                           ` Tim Cross
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-10  6:44 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 10 Jun 2022 14:20:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I"m also not convinced that org mode would be the right solution for
> >> Emacs documentation.
> >
> > Then why Org's own manual is written in Org instead of Texinfo?
> 
> I do not know if Org mode would be a good solution for Emacs
> documentation. However, compared to texinfo, our manual is lacking index
> entries in all the exported versions but texinfo.

Lack of an index makes for a less useful manual, IME.

Anyway, is the lack of the index the _only_ missing feature from what
the Org markup provides as compared to Texinfo?  Tim's message sounded
like there are many more omissions.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-09 22:31                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  2022-06-09 23:10                                   ` Tim Cross
@ 2022-06-10 13:56                                   ` Ihor Radchenko
  2022-06-10 19:32                                   ` Akib Azmain Turja
  2 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-10 13:56 UTC (permalink / raw)
  To: rms; +Cc: Tim Cross, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?

Adding to Tim's statement, there is currently no plan to redesign Org
_syntax_. What is going on now is a slow process of consolidating our
parser to be more consistent + clarifying the synatax description.
See https://orgmode.org/worg/dev/org-syntax.html

There are still some places in the parser and in the syntax description
that a vague or contradict each other.

As for Texinfo features, Org syntax is extensible enough to cover all of
them. At least, from quick skimming through the texinfo manual TOC.
Org does not have dedicated syntax for all the features in "6.1
Indicating Definitions, Commands, etc.", but they can be represented as
special link types. Also, Org does not provide index generation for all
the export backends, which has nothing to do with syntax. It's just not
implemented in our generic export library. Only dedicated Texinfo
exporter supprots the index generation.

> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.

Org is able to export to TeX and many other formats. However, I do not
know what features are required in the context of manuals.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-09 22:31                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  2022-06-09 23:10                                   ` Tim Cross
  2022-06-10 13:56                                   ` Ihor Radchenko
@ 2022-06-10 19:32                                   ` Akib Azmain Turja
  2 siblings, 0 replies; 376+ messages in thread
From: Akib Azmain Turja @ 2022-06-10 19:32 UTC (permalink / raw)
  To: Richard Stallman, Tim Cross; +Cc: monnier, acm, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> If we are going to redesign Org mode syntax,
> can we please add room for extensibility
> so that an extension of Org mode syntax
> could have all the features of Texinfo?

AFAIK, no, Org syntax is not being changed.  It seems like Org syntax is
already much extensible.

>
> If the new syntax can express all the distinctions that Texinfo
> can already express, that would make it possible to switch to it
> for our documentation sources.
>
> This needs to include conversion into some variant of TeX syntax so
> that we can generate high-quality printed manuals through TeX.

Converting all Texinfo to Org would require significant effort, which I
think should be spend on new packages or existing packages like GCC,
Guix, Emacs (and perhaps Hurd to complete the GNU system, LOL).

Also switching to Org would probably make vi (no, I'm don't use it)
users angry.  (No flamewar please.)

I think we should better make texinfo-mode better.

>
> -- 
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>

-- 
Akib Azmain Turja

This message is signed by me with my GnuPG key.  It's fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-09  9:38                                       ` Eli Zaretskii
@ 2022-06-11  3:52                                         ` Ihor Radchenko
  2022-06-11  6:52                                           ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-11  3:52 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Thu, 09 Jun 2022 17:15:05 +0800
>> 
>> > Are you sure this is always true?  There are several dozens of
>> > remapped commands; did you audit all of them?
>> 
>> I have audited a couple of them and relied on a good faith to other Org
>> devs + absence of bug reports relevant to the raised concern.
>> Rather then re-checking the remapped functions, I'd rather just fix any
>> concrete bug reports about misbehavior, if such reports arise.
>
> Didn't this discussion raise such "bug reports"?  Why not consider
> what several people here said as such a bug report?  The difference is
> that these reports come from people who use Org rarely, so the
> concerns are different.  But if the Org developers want to have Org
> adopted more widely, I think they should listen to these concerns, not
> reject them.

To clarify, it is metally difficult to process bug reports raised in
such discussions because they are often entangled with quite negative
attitude and much more broad claims.

What I have summarised so far from the discussions:

1. Org mode shadows default C-S-<up>/C-S-<down> bindings.

   This can be solved in two ways:
   a. Change org-support-shift-select to t by default + fix the fact
      that org-support-shift-select is actually ignored by
      C-S-<up>/<down>.
   b. Disable Org mode arrow bindings by default and move them to a
      dedicated minor-mode / setting.
      (This will make existing Org users angry)

   Note that I do not see C-c / and C-c C-s as a bug. The first one is
   not recommended for major-mode but allowed. C-c C-s is dedicated to
   major modes.

2. Org mode binds too many keys

   The solution can be disabling those keys and moving them to dedicated
   minor modes / settings.

   After thinking, I do not like the idea of enabling some keys
   depending on the presence absense of tables/source blocks inside the
   Org file. This will not solve the issue in the files actually
   containing all those and may surprise users.

3. Some of the context-dependent Org command names are confusing as they
   do not reveal what the command does.

   The solution is to rename those commands to something more
   meaningful. Totally doable.
      
4. Org remaps some of the default commands.
   I can see how it can be confusing, but do not see it as a critical
   issue. See my response below.

5. Org mode is too slow to load.
   I do not see an easy way to resolve this apart from general
   refactoring and modularizing. Profiling is not very helpful in this
   specific case, unfortunately. At least, my profiling skills are not
   good enough.

>> While I understand the possible fear of unexpected or unnatural
>> behavior of the remapped command, I'd like to point out that command
>> behavior can span over quite a significant range even without remapping.
>> Just the fact that some major mode does not use remapping, does not mean
>> that you are guaranteed to get the normal command behavior.
>
> Most major modes don't remap global and frequently-used commands.
> Those that do are problematic, and should be fixed rather than be used
> as examples of what's "kosher".

Let's take one example: org-forward-paragraph that is remapping
forward-paragraph.

The docstring is:

Move forward by a paragraph, or equivalent, unit.
...
The function moves point between two structural
elements (paragraphs, tables, lists, etc.).

It also provides the following special moves for convenience:

  - on a table or a property drawer, move to its beginning;
  - on comment, example, export, source and verse blocks, stop
    at blank lines;
  - skip consecutive clocks, diary S-exps, and keywords.

In Org mode, there is no single definition of a paragraph. The
paragraphs or equivalent syntax elements are context-depended and we
have to use parser to know where to move.

The default forward-paragraph can only be controlled by paragraph-start
and paragraph-separate regexps. Both are not sufficient to cater Org
mode. Regexps do not cut it for Org mode.

Hence, the built-in version of forward-paragraph is not usable in Org
mode. If we did not remap it, forward-paragraph would not really move by
paragraph, as it is understood by human. It would be more confusing than
org version of the command.

If you know a better way to resolve the described limitation, please let
me know.

Similar limitations motivated
<remap> <backward-paragraph>	org-backward-paragraph
<remap> <backward-sentence>	org-backward-sentence
<remap> <comment-dwim>		org-comment-dwim
<remap> <forward-paragraph>	org-forward-paragraph
<remap> <forward-sentence>	org-forward-sentence
<remap> <move-beginning-of-line> org-beginning-of-line
<remap> <move-end-of-line>	org-end-of-line
<remap> <delete-indentation>	org-delete-indentation

The following functions have a natural fallback to default behavior and
might be moved to minor-modes enabled by default. However, their default
behavior is context dependent and motivated by the lack of flexibility
of the built-in equivalents. Similar to the above functions.

<remap> <open-line>		org-open-line
<remap> <move-beginning-of-line> org-beginning-of-line
<remap> <move-end-of-line>	org-end-of-line

The following functions might be implemented using built-in Emacs
functionality:

<remap> <transpose-words>	org-transpose-words
could use find-word-boundary-function-table

<remap> <fill-paragraph>	org-fill-paragraph
appears to be redundant as we do set fill-paragraph-function


<remap> <yank>			org-yank
may be instead implemented using yank-handler property (it takes care
about adjusting level of the inserted headlines), but not completely. We
cannot control properties of text from outside of Org mode and thus the
functionality cannot be fully implemented using built-in Emacs features

The following commands are providing convenience self-alignment
functionality and protect from accidental editing of invisible text
(AFAIK, this issue is not addressed by Emacs):
<remap> <kill-line>		org-kill-line
<remap> <delete-backward-char>	org-delete-backward-char
<remap> <delete-char>		org-delete-char
<remap> <self-insert-command>	org-self-insert-command

I am not sure how to achieve equivalent functionality using built-in
Emacs customisations.

The following functions are convenience building for prople coming from
outline mode. I do not see any problem here.

<remap> <outline-backward-same-level> org-backward-heading-same-level
<remap> <outline-demote>	org-demote-subtree
<remap> <outline-forward-same-level> org-forward-heading-same-level
<remap> <outline-insert-heading> org-ctrl-c-ret
<remap> <outline-mark-subtree>	org-mark-subtree
<remap> <outline-next-visible-heading> org-next-visible-heading
<remap> <outline-previous-visible-heading> org-previous-visible-heading
<remap> <outline-promote>	org-promote-subtree
<remap> <outline-show-branches> org-kill-note-or-show-branches
<remap> <outline-show-children> org-fold-show-children
<remap> <outline-show-subtree>	org-fold-show-subtree

>> Some major modes rely on post-command-hook instead of remapping, which
>> not only modifies the original command behavior in unspecified ways, but
>> is also not directly visible from the mode bindings. Or consider changes
>> in filter-buffer-substring-function and how it can modify killing text.
>
> And that is in your opinion a Good Thing?  I don't think it is, and
> therefore don't consider such problematic behavior a good example for
> development.

Maybe. But then we need a more "appropriate" way to implement the
functionality Org provides. See the above.

>> So, I'd say that your concern is not really justified.
>
> ??? Just because "others do that"?

>> There is generally no reliable way to tell if an arbitrary major
>> mode modifies standard command behavior just by looking at its key
>> bindings. You have to rely on faith and report bugs if you stumble
>> upon them.
>
> I don't see how this solves the concerns.  We should still try to
> minimize such problematic behaviors as much as possible.

I'd like you to focus on the last example with
filter-buffer-substring-function or on the ability for major mode to
customise fill-paragraph-function or paragraph-separate, etc. Those
changes can alter the default commands drastically. Yet, they will not
be noticeable by users from looking at the M-x describe-bindings.

I do not say that major modes should change the default behavior in
unexpected ways. Rather the opposite - major modes should adjust the
default commands to behave more expectedly within the major mode
context. However, remapping is by no means an indication that a major
mode is going to change things unexpectedly. Its neither sufficient nor
required condition.

>> > No, not IME.  Show me another general-purpose editing mode that
>> > defines so many key bindings.  The only modes that get close are those
>> > where text is read-only, so normal editing is impossible anyway.
>> ...
>> Auctex binds a decent number of key-bindings. VHDL mode binds even
>> more (I just looked into one of the largest mode built-in prog
>> modes). markdown-mode binds over 100 keys not counting prefix maps.
>
> Two of these aren't part of Emacs, and VHDL is an example of
> programming language, where text is not really for human consumption.

I get your point on VHDL. However, I am not sure why you reject the
non-built-in examples. Are you implying that text modes outside Emacs
core are worse than built-in? At least Auctex is largely popular among
users and yet I have never seen as many complaints about Auctex
complexity as I do for Org (I may be biased).

Auctex and markdown-mode deal with similar editing tasks with Org mode.
The fact that they also use large number of key bindings is an
indication that they are really needed for editing complex documents.

>> >   And even those leave alone basic movement commands, like C-S-<up>
>> > which Alan mentioned.
>> 
>> C-S-<up> is not a basic movement command.
>
> Of course, it is: it moves by paragraphs.

Nope. It is unbound command working because of key translation:

C-<down> (translated from C-S-<down>) runs the command
forward-paragraph (found in global-map), which is an interactive
native compiled Lisp function in ‘paragraphs.el’.

>> By default, it is activating
>> region. Without shift-select-mode, it is not bound to anything.
>
> But shift-select-mode is on by default.

Fair point.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-09 19:48                               ` Alan Mackenzie
@ 2022-06-11  4:09                                 ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-11  4:09 UTC (permalink / raw)
  To: Alan Mackenzie; +Cc: Tim Cross, Eli Zaretskii, emacs-devel

Alan Mackenzie <acm@muc.de> writes:

>> >> Alan's claim was 785 additional key bindings, many of which not
>> >> bound to C-c. 
>
> It wasn't a "claim" it was a statement of fact.  These 784 bindings are
> DOCUMENTED in org.info.  You attempted to explain it away by saying, at
> first vaguely, that not all of them "come into play".  Only later did you
> make that meaningful by saying that not all of them get loaded at any
> time.
>
> Many of these binding are not bound to C-c C-<letter> or other major mode
> sequences.

Please note that not only C-c C-<letter> are dedicated to major modes.
Also, C-c <non-letter> (though not all C-c <non-letter> are
recommended).

I vent through all the bindings we define in the manual and the only
bindings that do not use reserved major-mode bindings space are:
C-'
C-,
<control character/s>-<arrow>
<control character/s>-<RET>

Out of the above bindings only C-S-<up>/<down> override the default
bindings. But only when shift-select-mode is enabled (which had not been
the default when these Org bindings were introduced).

Other bindings might cause unexpected behavior when user has personal
bindings. However, please note that Emacs recommends binding
C-c <letter> for user space. Anything else is a "gray" zone.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-10  6:44                                         ` Eli Zaretskii
@ 2022-06-11  4:49                                           ` Tim Cross
  2022-06-11  6:58                                             ` Eli Zaretskii
  2022-06-12  0:42                                             ` Richard Stallman
  0 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-11  4:49 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Ihor Radchenko, rms, monnier, acm, emacs-devel


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
>> Date: Fri, 10 Jun 2022 14:20:37 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I"m also not convinced that org mode would be the right solution for
>> >> Emacs documentation.
>> >
>> > Then why Org's own manual is written in Org instead of Texinfo?
>> 
>> I do not know if Org mode would be a good solution for Emacs
>> documentation. However, compared to texinfo, our manual is lacking index
>> entries in all the exported versions but texinfo.
>
> Lack of an index makes for a less useful manual, IME.
>
> Anyway, is the lack of the index the _only_ missing feature from what
> the Org markup provides as compared to Texinfo?  Tim's message sounded
> like there are many more omissions.

My statement is simply waht it says. I am not convinced org mode would
be the right solution for Emacs documentation. This is for many reasons,
some relating to org, some relating to the fact I'm not convinced there
is a need to replace texinfo, especially given that in some areas, I
think texinfo is more feature rich and consistent and some relating to the feeling I have
that the result is hard to justify given the amount of work it would
require. At the end of the day, proposing org mode as a replacement for
texinfo feels too much like a solution looking for a problem and one
that doesn't appear to have clear enough advantages to justify the work
needed to make such a transition, especially given work being done in
the texinfo area to improve its support for new formats, like epub. 

Org mode is a fantastic tool and I use it every day. There are ways org
mode could be used to augment the Emacs environment, but I don't think
it is an all or nothing situation. I'm also not convinced that embedding
org into Emacs more and more is necessarily a good thing. Tighter
coupling comes with both pros and cons.  



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-08  6:50                     ` Tim Cross
  2022-06-08  7:25                       ` Ihor Radchenko
  2022-06-08 11:39                       ` Jean Louis
@ 2022-06-11  6:40                       ` Jean Louis
  2022-06-11 11:16                         ` Protesilaos Stavrou
  2022-06-12  2:23                         ` Tim Cross
  2 siblings, 2 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-11  6:40 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

* Tim Cross <theophilusx@gmail.com> [2022-06-08 09:57]:
> Please give me an example of org mode 'not make space where it would be
> otherwise required'. Can you provide a single example of org mode
> syntax which is not readable in any text editor. There are quite a few
> projects on Github/Gitlab which have readme.org files - can you point to
> one which cannot be read with a plain text editor? 

Hello Tim,

my response time is not so fast.

I am now reviewing package descriptions:

- adoc-mode -- it uses Markdown to display descriptions, this is
  similar case to using Org. However, it is pretty human readable. In
  any case, both Markdown or Org formats should be converted to plain
  text. 

- ample-theme -- uses Org for package description, I consider this
  description not human readable. It is Emacs package theme. It is not
  even related to Org mode, but user has to look into following:

> * ample-theme.el
>   - A Light and Dark pair of themes for Emacs.
>   - Full colored gui and 256 color support only for now
> ** Installation
>     If MELPA is set up: 
>     #+BEGIN_SRC emacs-lisp
>       (package-install 'ample-theme)
      
>       (load-theme 'ample t)
>     #+END_SRC
> ** If you get ugly colors in terminal:
> #+BEGIN_SRC shell-script
>   # throw this in your ~/.bash_profile
>   export TERM="xterm-256color"
> #+END_SRC
> ** Ample-light-theme vs Ample-theme
>     [[http://i.imgur.com/86VLSV9.png]]

In the above Org type of a description, author does not have a sense
even to separate headings with the blank space. 

And how does "#+BEGIN_SRC emacs-lisp" help user to install the theme?
Can we understand that user simply wants to install the theme and that
"#+BEGIN_SRC emacs-lisp" is there not in right place, it is
confusing. Author expects of the user to know what is Org mode in
addition to installation of a theme, that is not proportional. Users
installing themes may not need to know nothing about Org.

Recommended for everybody to read:

Usability 101: Introduction to Usability
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

Usability Testing 101
https://www.nngroup.com/articles/usability-testing-101/

More reviews:

- package corfu, totally not human readable as it begins with:

> #+title: corfu.el - Completion Overlay Region FUnction
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: corfu.texi
> #+texinfo_dir_category: Emacs
> #+texinfo_dir_title: Corfu: (corfu).
> #+texinfo_dir_desc: Completion Overlay Region FUnction

> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://github.com/minad/corfu/blob/screenshots/emacs.svg?raw=true"/></a>
> #+html: <a href="http://elpa.gnu.org/packages/corfu.html"><img alt="GNU ELPA" src="https://elpa.gnu.org/packages/corfu.svg"/></a>
> #+html: <a href="http://elpa.gnu.org/devel/corfu.html"><img
> #alt="GNU-devel ELPA"
> #src="https://elpa.gnu.org/devel/corfu.svg"/></a

- package crdt -- pretty confusing when reading that Org, something
  like this:

> =M-x crdt-list-sessions= lists all sessions.

it expects user to know that "=" is markup. It cannot help to full
understanding. Users of CRDT need not be Org users. Even many Org
users do not know full Org markup.

- package ftable

> #+TITLE: ftable.el

> This package provides some convenient commands for filling a table, i.e., adjusting the layout of the table so it can fit in n columns.

> [[./ftable.gif]]

How is that tag "#+TITLE: ftable.el" helping user? We cannot assume
that users will know about the meaning of "#+" and why shall package
description again repeat the title of the package, that is already in
the header of the package. 

And how am I supposed to know what is "[[./ftable.gif]]" and where it
is? I have no access to it. It communicates zero information to
user. It confuses user.

Usability would be greatly improved if there would be simple link to
the image, as the package description is anyway showing links
correctly. 

- package leaf -- this one completely expects user to know about Org
  markup. It has terrible readibility score.

- package logos -- until user comes to read anything about
  description, must go through this garbage:

> #+title: logos.el: simple focus mode for Emacs with page breaks or outlines
> #+author: Protesilaos Stavrou
> #+email: info@protesilaos.com
> #+language: en
> #+options: ':t toc:nil author:t email:t num:t
> #+startup: content

> #+macro: stable-version 0.3.0
> #+macro: release-date 2022-03-30
> #+macro: development-version 0.4.0-dev
> #+macro: file @@texinfo:@file{@@$1@@texinfo:}@@
> #+macro: space @@texinfo:@: @@
> #+macro: kbd @@texinfo:@kbd{@@$1@@texinfo:}@@

> #+export_file_name: logos.texi

> #+texinfo_filename: logos.info
> #+texinfo_dir_category: Emacs misc features
> #+texinfo_dir_title: Logos: (logos)
> #+texinfo_dir_desc: Simple focus mode with page breaks or outlines
> #+texinfo_header: @set MAINTAINERSITE @uref{https://protesilaos.com,maintainer webpage}
> #+texinfo_header: @set MAINTAINER Protesilaos Stavrou
> #+texinfo_header: @set MAINTAINEREMAIL @email{info@protesilaos.com}
> #+texinfo_header: @set MAINTAINERCONTACT @uref{mailto:info@protesilaos.com,contact the maintainer}

> #+texinfo: @insertcopying

- package org-modern -- author did not check if those links
  work. Example communication by author:

> The screenshots shows [[file:example.org][example.org]] with
> =org-modern-mode= turned on and off

Links appear there, however, the file example.org is not clickable, it
could easily be made properly clickable.

- package osm -- not readable due to Org settings in package
  description. Those settings do not belong to package description.

- package pdf-tools -- same thing, user is supposed to read
  description. Things like below are confusing:

> :PROPERTIES:
> :CREATED:  [2021-12-29 Wed 18:34]
> :ID:       5a884389-6aec-498a-90d5-f37168809b4f
> :EXPORT_FILE_NAME: index
> :END:

Those tags are by no means relevant to package description. User wants
PDF tools, and what is user supposed to mean about the ID hashes and
what else...

- package posframe -- readability is minimized due to usage of heavier
  Org markup

- package svg-lib -- user cannot click on provided screenshots in Org
  markup. Why do that? Provide correct links.

- package system-packages -- Org markup degrades human readability

I have reviewed just some of packages installed on my side. There may
be many others. Readability and usability is greatest issue in
computing and helping user with understanding.

One shall compare Org package descriptions to so many other packages
without Org and find out how Org greatly degrades readability and thus
understanding on what that package is about and how to use it. 

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-11  3:52                                         ` Ihor Radchenko
@ 2022-06-11  6:52                                           ` Eli Zaretskii
  2022-06-12  8:40                                             ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-11  6:52 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 11:52:05 +0800
> 
> > Didn't this discussion raise such "bug reports"?  Why not consider
> > what several people here said as such a bug report?  The difference is
> > that these reports come from people who use Org rarely, so the
> > concerns are different.  But if the Org developers want to have Org
> > adopted more widely, I think they should listen to these concerns, not
> > reject them.
> 
> To clarify, it is metally difficult to process bug reports raised in
> such discussions because they are often entangled with quite negative
> attitude and much more broad claims.
> 
> What I have summarised so far from the discussions:
> 
> 1. Org mode shadows default C-S-<up>/C-S-<down> bindings.
> 
>    This can be solved in two ways:
>    a. Change org-support-shift-select to t by default + fix the fact
>       that org-support-shift-select is actually ignored by
>       C-S-<up>/<down>.
>    b. Disable Org mode arrow bindings by default and move them to a
>       dedicated minor-mode / setting.
>       (This will make existing Org users angry)
> 
>    Note that I do not see C-c / and C-c C-s as a bug. The first one is
>    not recommended for major-mode but allowed. C-c C-s is dedicated to
>    major modes.
> 
> 2. Org mode binds too many keys
> 
>    The solution can be disabling those keys and moving them to dedicated
>    minor modes / settings.
> 
>    After thinking, I do not like the idea of enabling some keys
>    depending on the presence absense of tables/source blocks inside the
>    Org file. This will not solve the issue in the files actually
>    containing all those and may surprise users.
> 
> 3. Some of the context-dependent Org command names are confusing as they
>    do not reveal what the command does.
> 
>    The solution is to rename those commands to something more
>    meaningful. Totally doable.
>       
> 4. Org remaps some of the default commands.
>    I can see how it can be confusing, but do not see it as a critical
>    issue. See my response below.
> 
> 5. Org mode is too slow to load.
>    I do not see an easy way to resolve this apart from general
>    refactoring and modularizing. Profiling is not very helpful in this
>    specific case, unfortunately. At least, my profiling skills are not
>    good enough.

Thank you, I think this is a very good summary.  I hope some of these
issues could be worked on and improved in the future.

> > Most major modes don't remap global and frequently-used commands.
> > Those that do are problematic, and should be fixed rather than be used
> > as examples of what's "kosher".
> 
> Let's take one example: org-forward-paragraph that is remapping
> forward-paragraph.
> 
> The docstring is:
> 
> Move forward by a paragraph, or equivalent, unit.
> ...
> The function moves point between two structural
> elements (paragraphs, tables, lists, etc.).
> 
> It also provides the following special moves for convenience:
> 
>   - on a table or a property drawer, move to its beginning;
>   - on comment, example, export, source and verse blocks, stop
>     at blank lines;
>   - skip consecutive clocks, diary S-exps, and keywords.
> 
> In Org mode, there is no single definition of a paragraph. The
> paragraphs or equivalent syntax elements are context-depended and we
> have to use parser to know where to move.
> 
> The default forward-paragraph can only be controlled by paragraph-start
> and paragraph-separate regexps. Both are not sufficient to cater Org
> mode. Regexps do not cut it for Org mode.
> 
> Hence, the built-in version of forward-paragraph is not usable in Org
> mode. If we did not remap it, forward-paragraph would not really move by
> paragraph, as it is understood by human. It would be more confusing than
> org version of the command.

For a serious discussion of this example, I'd need more detail about
the aspects you mentioned above (it is not trivial to deduce them from
looking at the top levels of org-forward-paragraph).  Specifically:

  . how does paragraph definition depend on context in Org, and what
    are the possible definitions in the existing contexts?
  . why don't paragraph-start and paragraph-separate fit Org, and can
    we come up with a small number of additional variables that would
    augment these two so that the built-in forward-paragraph could be
    extended to cover Org?

> If you know a better way to resolve the described limitation, please let
> me know.

The better way, IMO, is this:

  . try to find a way of extending the built-in forward-paragraph to
    cover the Org use cases, by adding variables in addition to the 2
    existing ones (I don't yet understand why "regexps do not cut it
    for Org mode -- it sounds like a very strong assertion)
  . if the above doesn't work, make forward-paragraph work through
    forward-paragraph-function, so that Org could define its own
    function
  . in any case, the add-on convenience features of
    org-forward-paragraph should be provided as options that the user
    can control

My main concern is that forward-paragraph is used when editing the
purely textual parts of an Org document, and when used there, casual
users of Org will expect it to behave as the command behaves in plain
text.  Any deviation from the behavior of the built-in command will
confuse such users when they edit the plain-text parts, and should
therefore be avoided.

> The following functions have a natural fallback to default behavior and
> might be moved to minor-modes enabled by default. However, their default
> behavior is context dependent and motivated by the lack of flexibility
> of the built-in equivalents. Similar to the above functions.
> 
> <remap> <open-line>		org-open-line
> <remap> <move-beginning-of-line> org-beginning-of-line
> <remap> <move-end-of-line>	org-end-of-line

Each of these (and other examples you provided) should require a
separate serious discussion.  Let's take open-line as an example.  It
is basically the built-in open-line, unless org-special-ctrl-o is
non-nil.  A trivial extension of the built-in command could have
prevented the need to remap.  (And why does the support for tables in
org-open-line need to be turned on outside of orgtbl-mode?)

> <remap> <yank>			org-yank
> may be instead implemented using yank-handler property (it takes care
> about adjusting level of the inserted headlines), but not completely. We
> cannot control properties of text from outside of Org mode and thus the
> functionality cannot be fully implemented using built-in Emacs features

IMO, org-yank is sufficiently different to justify not taking over the
built-in command.  Again, consider a user who edits the plain-text
portions of an Org document and expects the "usual" behavior of C-y.

> >> Some major modes rely on post-command-hook instead of remapping, which
> >> not only modifies the original command behavior in unspecified ways, but
> >> is also not directly visible from the mode bindings. Or consider changes
> >> in filter-buffer-substring-function and how it can modify killing text.
> >
> > And that is in your opinion a Good Thing?  I don't think it is, and
> > therefore don't consider such problematic behavior a good example for
> > development.
> 
> Maybe. But then we need a more "appropriate" way to implement the
> functionality Org provides. See the above.

Each of these commands should be discussed separately, and the best
solution found and implemented.  There's no single answer to that
question for all of these remapped commands.  But remapping should be
avoided as much as possible; see below.

> > I don't see how this solves the concerns.  We should still try to
> > minimize such problematic behaviors as much as possible.
> 
> I'd like you to focus on the last example with
> filter-buffer-substring-function or on the ability for major mode to
> customise fill-paragraph-function or paragraph-separate, etc. Those
> changes can alter the default commands drastically. Yet, they will not
> be noticeable by users from looking at the M-x describe-bindings.

When fill-paragraph-function and other means of customizing a command
to the particular mode make sense, they will not need to be noticed,
because they will do what the users expect.  Moreover, customizing the
behavior of commands through those variables has a couple of important
advantages:

  . it doesn't use remapping, so it doesn't stick out and bother
    conscientious Emacs users, who are used to study unfamiliar
    commands before using them
  . they don't require separate documentation, apart of saying that
    paragraph-start and paragraph-separate can be modified by modes as
    appropriate -- all the rest of the command's documentation is
    still intact and easily grasped in the context of any mode (this
    is also one more reason why "niceties" like special behavior
    inside tables should be separately controlled)

> I do not say that major modes should change the default behavior in
> unexpected ways. Rather the opposite - major modes should adjust the
> default commands to behave more expectedly within the major mode
> context. However, remapping is by no means an indication that a major
> mode is going to change things unexpectedly. Its neither sufficient nor
> required condition.

My point is that there are better ways of customizing the behavior of
a built-in command.  Org is now an integral part of Emacs, so it
should implement these customizations in ways that are available to a
core package; remapping is basically a tool used by unbundled packages
that cannot affect the core in any way.

> >> Auctex binds a decent number of key-bindings. VHDL mode binds even
> >> more (I just looked into one of the largest mode built-in prog
> >> modes). markdown-mode binds over 100 keys not counting prefix maps.
> >
> > Two of these aren't part of Emacs, and VHDL is an example of
> > programming language, where text is not really for human consumption.
> 
> I get your point on VHDL. However, I am not sure why you reject the
> non-built-in examples. Are you implying that text modes outside Emacs
> core are worse than built-in?

No, I'm saying that, sadly, we have no real control on what the
developers of unbundled packages decide to do.  Thus, what you see
there is the evidence of our lack of control more than anything else.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  4:49                                           ` Tim Cross
@ 2022-06-11  6:58                                             ` Eli Zaretskii
  2022-06-11  7:59                                               ` Tim Cross
                                                                 ` (2 more replies)
  2022-06-12  0:42                                             ` Richard Stallman
  1 sibling, 3 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-11  6:58 UTC (permalink / raw)
  To: Tim Cross; +Cc: yantar92, rms, monnier, acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: Ihor Radchenko <yantar92@gmail.com>, rms@gnu.org,
>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 14:49:21 +1000
> 
> >> >> I"m also not convinced that org mode would be the right solution for
> >> >> Emacs documentation.
> >> >
> >> > Then why Org's own manual is written in Org instead of Texinfo?
> >> 
> >> I do not know if Org mode would be a good solution for Emacs
> >> documentation. However, compared to texinfo, our manual is lacking index
> >> entries in all the exported versions but texinfo.
> >
> > Lack of an index makes for a less useful manual, IME.
> >
> > Anyway, is the lack of the index the _only_ missing feature from what
> > the Org markup provides as compared to Texinfo?  Tim's message sounded
> > like there are many more omissions.
> 
> My statement is simply waht it says. I am not convinced org mode would
> be the right solution for Emacs documentation. This is for many reasons,
> some relating to org, some relating to the fact I'm not convinced there
> is a need to replace texinfo, especially given that in some areas, I
> think texinfo is more feature rich and consistent and some relating to the feeling I have
> that the result is hard to justify given the amount of work it would
> require.

Well, it is hard to say anything intelligent without knowing those
"many reasons".

And if your POV is shared by the Org developers, then I ask again: why
is the Org manual, which weighs in at 23,000 lines, is being written
in Org?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  6:58                                             ` Eli Zaretskii
@ 2022-06-11  7:59                                               ` Tim Cross
  2022-06-11  8:29                                                 ` Eli Zaretskii
  2022-06-12  9:05                                               ` Ihor Radchenko
  2022-06-12 22:38                                               ` Richard Stallman
  2 siblings, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-11  7:59 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: yantar92, rms, monnier, acm, emacs-devel


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tim Cross <theophilusx@gmail.com>
>> Cc: Ihor Radchenko <yantar92@gmail.com>, rms@gnu.org,
>>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
>> Date: Sat, 11 Jun 2022 14:49:21 +1000
>> 
>> >> >> I"m also not convinced that org mode would be the right solution for
>> >> >> Emacs documentation.
>> >> >
>> >> > Then why Org's own manual is written in Org instead of Texinfo?
>> >> 
>> >> I do not know if Org mode would be a good solution for Emacs
>> >> documentation. However, compared to texinfo, our manual is lacking index
>> >> entries in all the exported versions but texinfo.
>> >
>> > Lack of an index makes for a less useful manual, IME.
>> >
>> > Anyway, is the lack of the index the _only_ missing feature from what
>> > the Org markup provides as compared to Texinfo?  Tim's message sounded
>> > like there are many more omissions.
>> 
>> My statement is simply waht it says. I am not convinced org mode would
>> be the right solution for Emacs documentation. This is for many reasons,
>> some relating to org, some relating to the fact I'm not convinced there
>> is a need to replace texinfo, especially given that in some areas, I
>> think texinfo is more feature rich and consistent and some relating to the feeling I have
>> that the result is hard to justify given the amount of work it would
>> require.
>
> Well, it is hard to say anything intelligent without knowing those
> "many reasons".
>
> And if your POV is shared by the Org developers, then I ask again: why
> is the Org manual, which weighs in at 23,000 lines, is being written
> in Org?

I have never claimed my POV was that of the org developers, I've never
claimed to be an org developer or to speak in any way for org
developers. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  7:59                                               ` Tim Cross
@ 2022-06-11  8:29                                                 ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-11  8:29 UTC (permalink / raw)
  To: Tim Cross; +Cc: yantar92, rms, monnier, acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: yantar92@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca, acm@muc.de,
>  emacs-devel@gnu.org
> Date: Sat, 11 Jun 2022 17:59:46 +1000
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> My statement is simply waht it says. I am not convinced org mode would
> >> be the right solution for Emacs documentation. This is for many reasons,
> >> some relating to org, some relating to the fact I'm not convinced there
> >> is a need to replace texinfo, especially given that in some areas, I
> >> think texinfo is more feature rich and consistent and some relating to the feeling I have
> >> that the result is hard to justify given the amount of work it would
> >> require.
> >
> > Well, it is hard to say anything intelligent without knowing those
> > "many reasons".
> >
> > And if your POV is shared by the Org developers, then I ask again: why
> > is the Org manual, which weighs in at 23,000 lines, is being written
> > in Org?
> 
> I have never claimed my POV was that of the org developers, I've never
> claimed to be an org developer or to speak in any way for org
> developers. 

I didn't say you have.  The above question was aimed at the Org
developers, as should be obvious from the context; apologies if that
wasn't clear enough.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-11  6:40                       ` Jean Louis
@ 2022-06-11 11:16                         ` Protesilaos Stavrou
  2022-06-12 11:32                           ` Jean Louis
  2022-06-12  2:23                         ` Tim Cross
  1 sibling, 1 reply; 376+ messages in thread
From: Protesilaos Stavrou @ 2022-06-11 11:16 UTC (permalink / raw)
  To: Jean Louis, Tim Cross; +Cc: emacs-devel

> From: Jean Louis <bugs@gnu.support>
> Date: Sat, 11 Jun 2022 09:40:44 +0300
>
> [... 109 lines elided]
>
> - package logos -- until user comes to read anything about
>   description, must go through this garbage:
>
> [... 68 lines elided]

Hello Jean,

What you describe as "garbage" package authors consider essential to
providing high quality documentation in a variety of formats.  The
README.org is converted into an Info manual and has a corresponding HTML
page both on GNU ELPA and the author's (me) website:

- <https://elpa.gnu.org/packages/logos.html>
- <https://protesilaos.com/emacs/logos>
- Evaluate this to read the Info manual: (info "(logos) Top")

Nobody forces you to read the Org source and there are good alternatives
on offer.

All the best,
Protesilaos (or simply "Prot")

-- 
Protesilaos Stavrou
https://protesilaos.com



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-09 23:10                                   ` Tim Cross
  2022-06-10  5:59                                     ` Eli Zaretskii
@ 2022-06-12  0:42                                     ` Richard Stallman
  2022-06-12  1:39                                       ` Tim Cross
                                                         ` (2 more replies)
  1 sibling, 3 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12  0:42 UTC (permalink / raw)
  To: Tim Cross; +Cc: monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Making org mode syntax equivalent to texinfo syntax seems like a
  > mistake to me.

If this succeeds, it would be an important advance for the GNU system.
We would replace Texinfo with a much cleaner system, easier to use and
more maintainable.  Not solely for Emacs, but for ALL our
documentation!

                   The original idea was to have a light weight syntax which
  > is easyh to learn, not create a clone of texinfo. Besides, I suspect it
  > would be very difficult to maintain backwards compatibility. 

These are real concerns, but they are not real certainties.  If we look
for solutions, we may find good ones.

I hope someone will give it a try.

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

These constructs make it possible to carry out our documentation style
constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

There is nothing fundamentally hard about supporting these
distinctions.  Can someone please examine which ones Org supports and
which ones not, and propose exensions for the ones not yet supported?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  4:49                                           ` Tim Cross
  2022-06-11  6:58                                             ` Eli Zaretskii
@ 2022-06-12  0:42                                             ` Richard Stallman
  2022-06-12  1:27                                               ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Richard Stallman @ 2022-06-12  0:42 UTC (permalink / raw)
  To: Tim Cross; +Cc: eliz, yantar92, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

One of Texinfo's crucial features is a wide range of semantic markup
constructs, each of which can generate different output depending on
the output format.  For instance, Texinfo has @var, @emph and @dfn,
all of which generate italics in printed output, but they differ in
what they generate for other output formats.  There are probably 15
other such constructs.

Does Org format have the ability to make all these distinctions?
If not, I suspect that the Org mode documentation isn't following
all our style conventions for documentation.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  0:42                                             ` Richard Stallman
@ 2022-06-12  1:27                                               ` Ihor Radchenko
  2022-06-12  5:45                                                 ` Eli Zaretskii
  2022-06-12 22:38                                                 ` Richard Stallman
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  1:27 UTC (permalink / raw)
  To: rms; +Cc: Tim Cross, eliz, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> One of Texinfo's crucial features is a wide range of semantic markup
> constructs, each of which can generate different output depending on
> the output format.  For instance, Texinfo has @var, @emph and @dfn,
> all of which generate italics in printed output, but they differ in
> what they generate for other output formats.  There are probably 15
> other such constructs.

Could you please point to the place where I can read about the different
generated output?

> Does Org format have the ability to make all these distinctions?
> If not, I suspect that the Org mode documentation isn't following
> all our style conventions for documentation.

Org manual does not AFAIK, other than various index entries.
However, I am not sure what exactly you are referring to.
Could you please provide a link describing the conventions you are
referring to?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  0:42                                     ` Richard Stallman
@ 2022-06-12  1:39                                       ` Tim Cross
  2022-06-12  2:40                                         ` Org mode and Emacs T.V Raman
  2022-06-12  6:02                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  2022-06-12  1:45                                       ` Org mode and Emacs Po Lu
  2022-06-12  4:53                                       ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Christopher Dimech
  2 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-12  1:39 UTC (permalink / raw)
  To: rms; +Cc: monnier, acm, emacs-devel


Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Making org mode syntax equivalent to texinfo syntax seems like a
>   > mistake to me.
>
> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!
>

One key reason I worry about going down that road is that I suspect it
would complicate org's syntax. Two key benefits of org mode is that the
basic syntax is simple and it maps reasonably consistently acorss
different output formats. However, this flexibility does come at a cost.
To provide consistency across export formats, the basic formatting
'concepts' need to be somewhat 'generalised', which means at times you
will loose some of the more advanced or sophisticated formatting power
of some export back-ends. 

There are times, primarily where you need specialised or specific
formatting in a particular format, where you need to drop down to that
low level formatting 'language'. This is the org 'escape hatch', which
provides a way to better leverage off the specific capabilities of a
partricular back-end. For exmaple, you can embed latex in your document
which will be added to your formated output when you generate a PDF/PS
or *.tex output file. However, that formatting often won't work when you
use the same source file to generate an HTML or Markdown or ODT version
because there isn't a clear way to translate the latex to that format.

As pointed out elswhere in this thread, org could support missing
texinfo syntax using texinfo specific blocks. However, that isn't a
great experience from an authoring perspective. It is fine if you only
need to do it occasionally, but if you have to constantly add such
blocks in order to get really well formatted texinfo output, it will
become frustrating. Org also supports a powerful custom link format
which could be used to add missing syntax elements, but I'm unsure on
the usability of such an approach once you have a few such cusotm links.

If we want to avoid this situation, the syntax would need to be added to
the basic org syntax. However, that will also require all back-ends
being able to interpret that syntax in some 'sane' manner, which would
likley be considerable work and in some situations, could be extremely
difficult to do consistently with good results. It would also add to the
overall amount of syntax, potentially making it more complex and harder
to learn. 

Org mode is a great mode, especially for general documentation and
information management, todo and time management and to some extent, for
literate programming and/or 'executable' documents, such as lab books or
even some devops type applicaitons. Being able to have a single source
document which can be used to generate 'reasonable' versions in
different formats. You can even spend a bit of effort to customise
things so that you can get some pretty consistent advanced formatting in
some specific export formats. However, I often find when you need the
advanced formatting power of a specific back-end, your often better just
writing the document in that back-end as it tends to take less effort
and results in cleaner source. 

The other side of th coin is the on-going development of texinfo. I have
not written enough texinfo to really understand what the issues are
which drive the desire to replace it with something else, such as org. I
know there are some criticisms regarding output formats, but I also know
there is on-going work to improve that situation. Is the right strategy
to work on org mode so that it can replace texinfo or work on texinfo to
address limitations (or both?)?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  0:42                                     ` Richard Stallman
  2022-06-12  1:39                                       ` Tim Cross
@ 2022-06-12  1:45                                       ` Po Lu
  2022-06-12  2:15                                         ` Ihor Radchenko
  2022-06-12  4:53                                       ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Christopher Dimech
  2 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-12  1:45 UTC (permalink / raw)
  To: Richard Stallman; +Cc: Tim Cross, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!

Org mode is not easier to use than Texinfo... at least, I didn't figure
out how to use it.

IIRC it's also much slower to create similarly sized .info files from
Org source than it is with makeinfo, and it also requires an entire
manual to be contained in a single file.

Texinfo also comes with the added benefit of not requiring Emacs to edit
or to translate into other formats.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  1:45                                       ` Org mode and Emacs Po Lu
@ 2022-06-12  2:15                                         ` Ihor Radchenko
  2022-06-12  2:36                                           ` David Masterson
                                                             ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  2:15 UTC (permalink / raw)
  To: Po Lu; +Cc: Richard Stallman, Tim Cross, monnier, acm, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

> IIRC it's also much slower to create similarly sized .info files from
> Org source than it is with makeinfo

Could you please quantify much slower?
Do you have concrete example of a .org file which can demonstrate the
problem?

Also, I think that I have to clarify about texinfo export from Org.
Org does not currently generate .info files directly. .org file is first
transformed into .texi and then uses makeinfo to translate the generated
.texi into .info.

If there is an interest in Org being used instead of texinfo, it should
be based on the extensibility of Org rather than just making Org
replicate what texinfo does. However, I am not sure if all parties in
this discussion have a clear idea about concrete advantages of Org
compared with texinfo. I am not faimiliar with texinfo. Others may not
be as faimilar with Org.

I believe that if we want to be serious about using Org for
documentation more widely, we should first identify which particular
aspects of Org are beneficial for documentation and which particular
aspects are available in other documentation generators and must be then
provided by Org (if not yet available).

> ... , and it also requires an entire
> manual to be contained in a single file.

This is not true. Org mode supports #+include: directives to incorporate
multiple smaller files into one larger file.

> Texinfo also comes with the added benefit of not requiring Emacs to edit
> or to translate into other formats.

You can edit Org files outside Emacs. Say, in vim.
The point about exporting to other formats is valid.

This last point also raises a question. Can Elisp interpreter and
libraries be factored out of Emacs to create a way to execute Elisp
programs without installing all the interactive parts of Emacs?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-11  6:40                       ` Jean Louis
  2022-06-11 11:16                         ` Protesilaos Stavrou
@ 2022-06-12  2:23                         ` Tim Cross
  2022-06-12 11:41                           ` Jean Louis
  1 sibling, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-12  2:23 UTC (permalink / raw)
  To: Jean Louis; +Cc: emacs-devel


Jean Louis <bugs@gnu.support> writes:

> * Tim Cross <theophilusx@gmail.com> [2022-06-08 09:57]:
>> Please give me an example of org mode 'not make space where it would be
>> otherwise required'. Can you provide a single example of org mode
>> syntax which is not readable in any text editor. There are quite a few
>> projects on Github/Gitlab which have readme.org files - can you point to
>> one which cannot be read with a plain text editor? 
>
> Hello Tim,
>
> my response time is not so fast.
>
> I am now reviewing package descriptions:
>
> Recommended for everybody to read:
>
> Usability 101: Introduction to Usability
> https://www.nngroup.com/articles/usability-101-introduction-to-usability/
>
> Usability Testing 101
> https://www.nngroup.com/articles/usability-testing-101/
>

I gues sour differences here are that you have conflated issues of
readabiity and usability into the definition of plain text. Few of the
criticisms you have raised IMO are specific to org mode or due to org
modes syntax. Many of them are due to the individual author's layout
and not the result of org mode syntax. You can also write documents
which are very difficult to read using just plain ASCII. They are both
plain text.

I also think you are over-stating the difficulty of reading the raw org
file contents. While I would agree they are asier to read when correctly
formatted, they are not impossible to understand without it. For example,
to say something like 

> package corfu, totally not human readable as it begins with:

> #+title: corfu.el - Completion Overlay Region FUnction
> #+author: Daniel Mendler
> #+language: en
> #+export_file_name: corfu.texi
> #+texinfo_dir_category: Emacs
> #+texinfo_dir_title: Corfu: (corfu).
> #+texinfo_dir_desc: Completion Overlay Region FUnction

is I think totally over stating the situation. I showed the above to
someone who is not a technical person and only uses computers for email,
web and office work and they were able to understand it perfectly well.
They didn't know what the texinfo lines meant, but they understood what
the title was, who the author was, the language and even the
texinfo_dir_desc being a description. I think most people are far more
capable at reading such information than you assert. They might not
understand all of it, but then again, I often read documents where I
don't understand all of it. The point is, there is a big difference
betrween being able to read contents and understand them. 

As a comparison, consider some RFCs which you mentioned previously. I've
read RFCs which perfectly comply with the formatting you referenced that
were (to me) totally unintelligible. Sometimes this is because the
writing is poor or the concepts are too alien or it is simply too
complicated for my limited understanding. However, this is still a plain
text document.  

For me, a file is plain text if you can edit and save it using just a
plain text editor and the file will still be valid. I also consider HTML
files to be plain text, (but far less readable than org files). On the
other hand, *.docx are not plain text and cannot be edited with a plain
text editor, saved and still be valid. Likewise, other formats which
use non-printable control characters are not plain text.

I will try to be clearer when I say something is plain text. Readability
and useability are separate from file formats - they are a higher, but
related, concern. Plain text does not imply anything about readability
or usability other than a plain text file can be opened and physically
read using a plain text editor or a utility like 'cat'. The ability of
the individual to interpret that information does not affect whether it
is plain text. If, for example, you write lots of HTML, an HTML file is
easily understood when displayed with cat. 

Of course, appropriate formatting within a plain text document can
affect how easily the document can be read and I think it is good to try
and format the document to be as easy to read as possible. However, for
org mode, this is really under the control of the author. Org data files
can be formatted in a way which makes them easier to read and there are
various Emacs settings which can be used which affects this (wrap,
visual line mode, adding blank lines at the end of sections etc). The
point is, many of your criticisms are not due to org syntax or org
enforced forrmatting, which is one reason I disagree with assertions
that org files are not plain text.

I'm sure this is apoint we will never agree on, so happy to leave it there.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:15                                         ` Ihor Radchenko
@ 2022-06-12  2:36                                           ` David Masterson
  2022-06-12  3:06                                             ` Ihor Radchenko
  2022-06-12  3:28                                             ` Tim Cross
  2022-06-12  2:50                                           ` Po Lu
  2022-06-12  6:57                                           ` Eli Zaretskii
  2 siblings, 2 replies; 376+ messages in thread
From: David Masterson @ 2022-06-12  2:36 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Po Lu, Richard Stallman, Tim Cross, monnier, acm, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>> or to translate into other formats.
>
> You can edit Org files outside Emacs. Say, in vim.
> The point about exporting to other formats is valid.
>
> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Isn't Tim Cross(?) working on something like that -- ie. a parser for
the Org language.  Once we have a solid parser, we can build a standard
(set of?) backend(s) for much of Orgmode.

-- 
David Masterson



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  1:39                                       ` Tim Cross
@ 2022-06-12  2:40                                         ` T.V Raman
  2022-06-12  6:02                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  1 sibling, 0 replies; 376+ messages in thread
From: T.V Raman @ 2022-06-12  2:40 UTC (permalink / raw)
  To: Tim Cross; +Cc: rms, monnier, acm, emacs-devel

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 383 bytes --]

Note that org's export to texinfo is quite good; as an example, I wrote
the "Emacspeak Turning Twenty" article in org-mode and later exported it
to texinfo among other formats. Org's export to TeX,LaTeX and PDF is
also very good; likely superior to its export to Texinfo.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:15                                         ` Ihor Radchenko
  2022-06-12  2:36                                           ` David Masterson
@ 2022-06-12  2:50                                           ` Po Lu
  2022-06-12  3:54                                             ` chad
  2022-06-12  6:57                                           ` Eli Zaretskii
  2 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-12  2:50 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Richard Stallman, Tim Cross, monnier, acm, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> Could you please quantify much slower?

Making the Org manual takes about 27 seconds on my system:

  GEN      org.texi
  GEN      ../../info/org.info

real	0m27.210s
user	0m25.567s
sys	0m0.783s

Which is longer than it takes for the entire (much more massive) Emacs
Lisp reference manual:

  GEN      ../../info/elisp.info

real	0m24.561s
user	0m23.660s
sys	0m0.515s

And generating the .info file from the Org manual texinfo source takes
only:

  GEN      ../../info/org.info

real	0m5.381s
user	0m5.109s
sys	0m0.141s

So it's clear that the org to texinfo export takes most of the time.

> I believe that if we want to be serious about using Org for
> documentation more widely, we should first identify which particular
> aspects of Org are beneficial for documentation and which particular
> aspects are available in other documentation generators and must be then
> provided by Org (if not yet available).

I don't see why we should be serious about using Org for our
documentation, when most people already know texinfo and are quite happy
with it.

> This is not true. Org mode supports #+include: directives to incorporate
> multiple smaller files into one larger file.

That's nice, but why isn't the Org manual itself structured that way?

> You can edit Org files outside Emacs. Say, in vim.

But does it work very well?

> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Probably not without a lot of work.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:36                                           ` David Masterson
@ 2022-06-12  3:06                                             ` Ihor Radchenko
  2022-06-12  3:39                                               ` David Masterson
  2022-06-12  3:28                                             ` Tim Cross
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  3:06 UTC (permalink / raw)
  To: David Masterson
  Cc: Po Lu, Richard Stallman, Tim Cross, monnier, acm, emacs-devel

David Masterson <dsmasterson@gmail.com> writes:

>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

Org already have a parser. Written in Elisp. Export is built on top of
the parser.

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:36                                           ` David Masterson
  2022-06-12  3:06                                             ` Ihor Radchenko
@ 2022-06-12  3:28                                             ` Tim Cross
  1 sibling, 0 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-12  3:28 UTC (permalink / raw)
  To: David Masterson
  Cc: Ihor Radchenko, Po Lu, Richard Stallman, monnier, acm, emacs-devel


David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>>> or to translate into other formats.
>>
>> You can edit Org files outside Emacs. Say, in vim.
>> The point about exporting to other formats is valid.
>>
>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

No. Ihor is the one working on the parser. 

I think Ihor's question about isolating the Elisp interpreter is about
options to make org mode work outside of Emacs. There are frequent
questions to the org list about making org available to other
editors/environments. However, the big problem is that much of the
really powerful featurs of org mode are intrinsically tied to elisp
functionality - for exmaple all the babel and backend export support.
While it is possible to write a parser in any language which would
enable basic org markup/formatting, that does not solve the problem of
executable blocks, babel/noweb and interaction with back end exporters
etc. 

This would be one example of where something like an elisp LSP module
would be useful. However, the idea of an elisp LSP module was
discouraged by a couple of people on this list over concerns that such a
module would enable non-free platforms to take advantage of elisp - a
fear which I think is overstated and which I think one person referred
to as "Jumping at shadows" 

The overall conclusion was that such a module would be considerable
amount of work. Some suggested you could use emacsclient to create
something which could be used in such a manner. However, I suspect that
if you go to the effort of installing and configuring emacs, you would
just use emacs. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  3:06                                             ` Ihor Radchenko
@ 2022-06-12  3:39                                               ` David Masterson
  2022-06-12  4:43                                                 ` Tim Cross
  0 siblings, 1 reply; 376+ messages in thread
From: David Masterson @ 2022-06-12  3:39 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Po Lu, Richard Stallman, Tim Cross, monnier, acm, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This last point also raises a question. Can Elisp interpreter and
>>> libraries be factored out of Emacs to create a way to execute Elisp
>>> programs without installing all the interactive parts of Emacs?
>>
>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>> the Org language.  Once we have a solid parser, we can build a standard
>> (set of?) backend(s) for much of Orgmode.
>
> Org already have a parser. Written in Elisp. Export is built on top of
> the parser.

But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
long run, it would be better to define a "parseable" language as the
standard basis for Org.  From that, people can develop (parts of) Org on
other platforms (Vim, Beorg, Orgzly) and test/prove that they are
compatible with version X of the language.  I think Organice was doing
this, but I haven't looked at it deeply.

Oh, but I see your point about "export". By backend, I was assuming a
true parser would generate a standard "internal" language which could be
fed into simpler backends to actually do the work.  The front-end parser
and back-ends could be translated/rebuilt as needed on new platorms
(iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
the idea.

-- 
David Masterson



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:50                                           ` Po Lu
@ 2022-06-12  3:54                                             ` chad
  2022-06-12  5:04                                               ` Po Lu
  2022-06-12  6:21                                               ` Eli Zaretskii
  0 siblings, 2 replies; 376+ messages in thread
From: chad @ 2022-06-12  3:54 UTC (permalink / raw)
  To: Po Lu
  Cc: Ihor Radchenko, Richard Stallman, Tim Cross, Stefan Monnier,
	Alan Mackenzie, EMACS development team

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Sat, Jun 11, 2022 at 10:52 PM Po Lu <luangruo@yahoo.com> wrote:

> I don't see why we should be serious about using Org for our
> documentation, when most people already know texinfo and are quite happy
> with it.
>

I think a reasonable examination of the emacs-devel archives as well as the
common practices of most of the people publishing emacs lisp packages today
would lead to a very different conclusion. There are several threads about
maintenance concerns around makeinfo/texinfo, and many discussions about
replacing texinfo with, for example, HTML.There are periodic threads where
people claim that they won't try to add their project to GNU because the
burden of learning and using texinfo is too high, although those have died
down in volume since it became more practical to translate other formats to
texinfo.

None of this is to say that Org is the "right" format for Emacs
documentation. I just don't think it's correct to say "most people already
know texinfo and are quite happy with it".

~Chad

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  3:39                                               ` David Masterson
@ 2022-06-12  4:43                                                 ` Tim Cross
  2022-06-12  5:08                                                   ` Po Lu
  2022-06-12  5:53                                                   ` David Masterson
  0 siblings, 2 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-12  4:43 UTC (permalink / raw)
  To: David Masterson
  Cc: Ihor Radchenko, Po Lu, Richard Stallman, monnier, acm, emacs-devel


David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>>> This last point also raises a question. Can Elisp interpreter and
>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>> programs without installing all the interactive parts of Emacs?
>>>
>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>> the Org language.  Once we have a solid parser, we can build a standard
>>> (set of?) backend(s) for much of Orgmode.
>>
>> Org already have a parser. Written in Elisp. Export is built on top of
>> the parser.
>
> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
> long run, it would be better to define a "parseable" language as the
> standard basis for Org.  From that, people can develop (parts of) Org on
> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
> compatible with version X of the language.  I think Organice was doing
> this, but I haven't looked at it deeply.
>

This is all part of the aims and process. However, the first step is to
develop a robust elisp based parser. This helps to ensure the org syntax
is consistent and helps identifies ambiguities which need to be fixed as
well as provides a reference implementation which developers can use to
develop parsers in other languages.

> Oh, but I see your point about "export". By backend, I was assuming a
> true parser would generate a standard "internal" language which could be
> fed into simpler backends to actually do the work.  The front-end parser
> and back-ends could be translated/rebuilt as needed on new platorms
> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
> the idea.

Personally, I think it is unlikely we will ever see an org mode for
other platforms which is equivalent to Emacs. However, the work which is
being done to create a clean parser in elisp and use that as the basis
for an API to manipulate org data, generate font-locking and formatting
and provide a clean API for exporters to use will make it much easier
for people to develop external org support at varying levels. 

Those doing the main work, like Ihor, are very aware of the desire to
facilitate external implementations. It is one reason that a fairly
conservative attitude to change is adopted by the project and often, one
consideration is what imapct a change would have on external org
compatible projects. Sometimes, this is challenging as on one level, we
want ot advance and improve org mode, but on the other, we want it to be
as stable as possible to reduce adverse impact on external projects. So
while the main target is and will always be Emacs, an eye is kept on
efforts of other external projects and an effort is made to support them
when possible and within Emacs and FSF principals/guidelines. 

So far, the key contributors have been doing a very good job. 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  0:42                                     ` Richard Stallman
  2022-06-12  1:39                                       ` Tim Cross
  2022-06-12  1:45                                       ` Org mode and Emacs Po Lu
@ 2022-06-12  4:53                                       ` Christopher Dimech
  2 siblings, 0 replies; 376+ messages in thread
From: Christopher Dimech @ 2022-06-12  4:53 UTC (permalink / raw)
  To: rms; +Cc: Tim Cross, monnier, acm, emacs-devel


> Sent: Sunday, June 12, 2022 at 12:42 PM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Tim Cross" <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > Making org mode syntax equivalent to texinfo syntax seems like a
>   > mistake to me.
>
> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!
>
>                    The original idea was to have a light weight syntax which
>   > is easyh to learn, not create a clone of texinfo. Besides, I suspect it
>   > would be very difficult to maintain backwards compatibility.
>
> These are real concerns, but they are not real certainties.  If we look
> for solutions, we may find good ones.
>
> I hope someone will give it a try.

An important aspect moving forward is to permit the use of latex
rather than tex.

> One of Texinfo's crucial features is a wide range of semantic markup
> constructs, each of which can generate different output depending on
> the output format.  For instance, Texinfo has @var, @emph and @dfn,
> all of which generate italics in printed output, but they differ in
> what they generate for other output formats.  There are probably 15
> other such constructs.
>
> These constructs make it possible to carry out our documentation style
> constructs.
>
> Does Org format have the ability to make all these distinctions?
> If not, I suspect that the Org mode documentation isn't following
> all our style conventions for documentation.
>
> There is nothing fundamentally hard about supporting these
> distinctions.  Can someone please examine which ones Org supports and
> which ones not, and propose exensions for the ones not yet supported?
>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  3:54                                             ` chad
@ 2022-06-12  5:04                                               ` Po Lu
  2022-06-12  7:02                                                 ` Ihor Radchenko
                                                                   ` (2 more replies)
  2022-06-12  6:21                                               ` Eli Zaretskii
  1 sibling, 3 replies; 376+ messages in thread
From: Po Lu @ 2022-06-12  5:04 UTC (permalink / raw)
  To: chad
  Cc: Ihor Radchenko, Richard Stallman, Tim Cross, Stefan Monnier,
	Alan Mackenzie, EMACS development team

chad <yandros@gmail.com> writes:

> I think a reasonable examination of the emacs-devel archives as well
> as the common practices of most of the people publishing emacs lisp
> packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo,
> and many discussions about replacing texinfo with, for example,
> HTML.

Published Emacs Lisp packages in the wild typically come with no
documentation at all, aside from doc strings and a single README file.

Org mode happens to be popular for the latter, but I don't know why.
The learning curve is high and the results seem to be unjustified.  An
easier format such as HTML or Markdown can be used instead.

> There are periodic threads where people claim that they won't try to
> add their project to GNU because the burden of learning and using
> texinfo is too high, although those have died down in volume since it
> became more practical to translate other formats to texinfo.

Why would Org mode be any different?  Unlike HTML, it is completely
specific to Emacs, which actually makes it worse than Texinfo, because
Texinfo is at least widely used throughout the entire GNU project.

And anyway, I make the following offer: if someone doesn't want to
contribute documentation (or code) because he doesn't know Texinfo, he
can write the documentation in plain text, and I will translate it to
Texinfo for him.  IIRC the GDB developers made a similar offer, and for
the many years that it existed nobody ever took advantage of it, which
shows that Texinfo is not a serious barrier to contributors.

Further more, using Org mode for documentation will make Emacs lose many
people who are actually writing documentation, right now.  At least I
have no interest in ever learning Org mode, and there seem to be no
volunteers who will translate Texinfo (or plain text) to Org mode for me
if we begin to use it for documentation in the future.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  4:43                                                 ` Tim Cross
@ 2022-06-12  5:08                                                   ` Po Lu
  2022-06-12  5:20                                                     ` Ihor Radchenko
  2022-06-12  5:27                                                     ` Tim Cross
  2022-06-12  5:53                                                   ` David Masterson
  1 sibling, 2 replies; 376+ messages in thread
From: Po Lu @ 2022-06-12  5:08 UTC (permalink / raw)
  To: Tim Cross
  Cc: David Masterson, Ihor Radchenko, Richard Stallman, monnier, acm,
	emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.

> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to

It is not the goal of Emacs to support tyrant devices such as the iPhone
running proprietary operating systems such as iOS.

If people are going to refactor Org mode, I hope they will not do it
with support for iPhones in mind.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:08                                                   ` Po Lu
@ 2022-06-12  5:20                                                     ` Ihor Radchenko
  2022-06-12  5:27                                                     ` Tim Cross
  1 sibling, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  5:20 UTC (permalink / raw)
  To: Po Lu
  Cc: Tim Cross, David Masterson, Richard Stallman, monnier, acm, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>
> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

I am not sure why you only picked up iPhones from the list suggested by
David. The aim is not supporting iPhones. The aim is supporting
other GNU and Free Software projects that are making use of Org, which
implies conservative approach to breaking changes, especially in syntax.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:08                                                   ` Po Lu
  2022-06-12  5:20                                                     ` Ihor Radchenko
@ 2022-06-12  5:27                                                     ` Tim Cross
  1 sibling, 0 replies; 376+ messages in thread
From: Tim Cross @ 2022-06-12  5:27 UTC (permalink / raw)
  To: Po Lu
  Cc: David Masterson, Ihor Radchenko, Richard Stallman, monnier, acm,
	emacs-devel


Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>>> long run, it would be better to define a "parseable" language as the
>>> standard basis for Org.  From that, people can develop (parts of) Org on
>>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>>> compatible with version X of the language.  I think Organice was doing
>>> this, but I haven't looked at it deeply.
>
>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>

> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

There has never been suggestion that what we are doing is to facilitate
development on non-free platforms. However, if you develop good clean
code and clear algorithms which others are able to use as a reference to
understand the syntax and semantics of a system, you cannot control how
they will use that information. In fact, it can be argued that making
such information readily available and accessible is a large part of the
freedoms being promoted by the FSF.

This is completely different from, for example, providing an interface
which can be used by a non-free platform. 

Besides, if org is as bad and difficult to learn as you have indicated
in other posts, nobody will bother implementing it on those platforms
you seem so threatened by anyway!




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  1:27                                               ` Ihor Radchenko
@ 2022-06-12  5:45                                                 ` Eli Zaretskii
  2022-06-12 22:38                                                 ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  5:45 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: rms, theophilusx, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  eliz@gnu.org,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 09:27:57 +0800
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > One of Texinfo's crucial features is a wide range of semantic markup
> > constructs, each of which can generate different output depending on
> > the output format.  For instance, Texinfo has @var, @emph and @dfn,
> > all of which generate italics in printed output, but they differ in
> > what they generate for other output formats.  There are probably 15
> > other such constructs.
> 
> Could you please point to the place where I can read about the different
> generated output?

In the Texinfo manual.  It describes, for each markup, what that
produces in every major output format.

> > Does Org format have the ability to make all these distinctions?
> > If not, I suspect that the Org mode documentation isn't following
> > all our style conventions for documentation.
> 
> Org manual does not AFAIK, other than various index entries.

That is strange to hear.  I was under the impression that Org can
export to various format, and each such output format has its own
export support.  So at least in principle, the distinction Richard is
talking about should be not only possible, but quite easy to
implement, if not already implemented.

> However, I am not sure what exactly you are referring to.
> Could you please provide a link describing the conventions you are
> referring to?

Most of them are in the Texinfo manual, and some are in the GNU Coding
Standards manual.

To take one example, @dfn produces italics in HTML and in printed
formats (PDF etc.), and quoted text in Info.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  4:43                                                 ` Tim Cross
  2022-06-12  5:08                                                   ` Po Lu
@ 2022-06-12  5:53                                                   ` David Masterson
  2022-06-12  6:56                                                     ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: David Masterson @ 2022-06-12  5:53 UTC (permalink / raw)
  To: Tim Cross
  Cc: Ihor Radchenko, Po Lu, Richard Stallman, monnier, acm, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>>> This last point also raises a question. Can Elisp interpreter and
>>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>>> programs without installing all the interactive parts of Emacs?
>>>>
>>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>>> the Org language.  Once we have a solid parser, we can build a standard
>>>> (set of?) backend(s) for much of Orgmode.
>>>
>>> Org already have a parser. Written in Elisp. Export is built on top of
>>> the parser.
>>
>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.
>>
>
> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to
> develop parsers in other languages.

Semantic/Bovine ??

Agree

>> Oh, but I see your point about "export". By backend, I was assuming a
>> true parser would generate a standard "internal" language which could be
>> fed into simpler backends to actually do the work.  The front-end parser
>> and back-ends could be translated/rebuilt as needed on new platorms
>> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
>> the idea.
>
> Personally, I think it is unlikely we will ever see an org mode for
> other platforms which is equivalent to Emacs. However, the work which is
> being done to create a clean parser in elisp and use that as the basis
> for an API to manipulate org data, generate font-locking and formatting
> and provide a clean API for exporters to use will make it much easier
> for people to develop external org support at varying levels.

Agree

> Those doing the main work, like Ihor, are very aware of the desire to
> facilitate external implementations. It is one reason that a fairly
> conservative attitude to change is adopted by the project and often, one
> consideration is what imapct a change would have on external org
> compatible projects. Sometimes, this is challenging as on one level, we
> want ot advance and improve org mode, but on the other, we want it to be
> as stable as possible to reduce adverse impact on external projects. So
> while the main target is and will always be Emacs, an eye is kept on
> efforts of other external projects and an effort is made to support them
> when possible and within Emacs and FSF principals/guidelines. 
>
> So far, the key contributors have been doing a very good job. 

That's good.

-- 
David Masterson



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  1:39                                       ` Tim Cross
  2022-06-12  2:40                                         ` Org mode and Emacs T.V Raman
@ 2022-06-12  6:02                                         ` Eli Zaretskii
  2022-06-12 22:38                                           ` Richard Stallman
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  6:02 UTC (permalink / raw)
  To: Tim Cross; +Cc: rms, monnier, acm, emacs-devel

> From: Tim Cross <theophilusx@gmail.com>
> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 11:39:07 +1000
> 
> The other side of th coin is the on-going development of texinfo. I have
> not written enough texinfo to really understand what the issues are
> which drive the desire to replace it with something else, such as org. I
> know there are some criticisms regarding output formats, but I also know
> there is on-going work to improve that situation. Is the right strategy
> to work on org mode so that it can replace texinfo or work on texinfo to
> address limitations (or both?)?

The main problem with Texinfo that I'm aware of is its reliance on TeX
for the printed output formats (PDF, PS, etc.).  Many restrictions in
Texinfo come from the fact that something is impossible to implement
in TeX.

Another issue that frequently comes up is the HTML format it produces:
it currently doesn't support many features expected from good on-line
documentation, such as multiple frames etc.

A better place to discuss the shortcomings of Texinfo is on the
Texinfo mailing list, where we can talk to the few people who are
actively developing Texinfo and aware of the main issues.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  3:54                                             ` chad
  2022-06-12  5:04                                               ` Po Lu
@ 2022-06-12  6:21                                               ` Eli Zaretskii
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  6:21 UTC (permalink / raw)
  To: chad; +Cc: luangruo, yantar92, rms, theophilusx, monnier, acm, emacs-devel

> From: chad <yandros@gmail.com>
> Date: Sat, 11 Jun 2022 23:54:39 -0400
> Cc: Ihor Radchenko <yantar92@gmail.com>, Richard Stallman <rms@gnu.org>,
>  Tim Cross <theophilusx@gmail.com>, 
>  Stefan Monnier <monnier@iro.umontreal.ca>, Alan Mackenzie <acm@muc.de>, 
>  EMACS development team <emacs-devel@gnu.org>
> 
>  I don't see why we should be serious about using Org for our
>  documentation, when most people already know texinfo and are quite happy
>  with it.
> 
> I think a reasonable examination of the emacs-devel archives as well as the common practices of most of
> the people publishing emacs lisp packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo, and many discussions about
> replacing texinfo with, for example, HTML.There are periodic threads where people claim that they won't try
> to add their project to GNU because the burden of learning and using texinfo is too high, although those have
> died down in volume since it became more practical to translate other formats to texinfo.

The real issue behind these claims is that developers seldom like
writing good documentation.  So learning to use a sophisticated markup
system such as Texinfo is overkill for them.  On top of that, if the
package is not an Emacs package, there's a problem that Texinfo
editing is not really well supported outside of Emacs, which is
another reason for complaining by people who don't use Emacs for
developing their package.

Good HTML is no easier to write than good Texinfo, but the difference
is that there are tools out there other than Emacs which can be used
to create HTML, especially if the manual is relatively simple (as many
Free Software manuals are).

Bottom line: the complaints are real, but I'm not sure they help in
this discussion, because there's no known alternative to Texinfo for
creating good documentation of the kind that we are used to in Emacs.
GCC folks are switching to Sphinx, but that has its own problems, and
without good support for it in Emacs (and no, rst.el is not enough, as
it doesn't provide enough markup commands, AFAICT) it isn't (yet) a
viable alternative.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:53                                                   ` David Masterson
@ 2022-06-12  6:56                                                     ` Ihor Radchenko
  2022-06-12 18:29                                                       ` David Masterson
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  6:56 UTC (permalink / raw)
  To: David Masterson
  Cc: Tim Cross, Po Lu, Richard Stallman, monnier, acm, emacs-devel

David Masterson <dsmasterson@gmail.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>> develop parsers in other languages.
>
> Semantic/Bovine ??

Org is not context-free.
Also, Org maintaners previously rejected the idea of implementing Org
parser not in Elisp. Mainly because it would limit the ability to
maintain and contribute to Org - one would need to learn another
programming language to alter anything in Org syntax.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  2:15                                         ` Ihor Radchenko
  2022-06-12  2:36                                           ` David Masterson
  2022-06-12  2:50                                           ` Po Lu
@ 2022-06-12  6:57                                           ` Eli Zaretskii
  2 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  6:57 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: luangruo, rms, theophilusx, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Richard Stallman <rms@gnu.org>,  Tim Cross <theophilusx@gmail.com>,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 10:15:59 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > IIRC it's also much slower to create similarly sized .info files from
> > Org source than it is with makeinfo
> 
> Could you please quantify much slower?
> Do you have concrete example of a .org file which can demonstrate the
> problem?

With an unoptimized build of Emacs 29, generation of org.texi from
org.org takes 2.5 minutes here.  An optimized build of Emacs 28 does
that in 30 sec on one system and 1 minute on another.  Generation of
org.info then takes between 3 and 4 seconds.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:04                                               ` Po Lu
@ 2022-06-12  7:02                                                 ` Ihor Radchenko
  2022-06-12 22:38                                                 ` Richard Stallman
  2022-06-12 22:38                                                 ` Richard Stallman
  2 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  7:02 UTC (permalink / raw)
  To: Po Lu
  Cc: chad, Richard Stallman, Tim Cross, Stefan Monnier,
	Alan Mackenzie, EMACS development team

Po Lu <luangruo@yahoo.com> writes:

> Published Emacs Lisp packages in the wild typically come with no
> documentation at all, aside from doc strings and a single README file.
>
> Org mode happens to be popular for the latter, but I don't know why.
> The learning curve is high and the results seem to be unjustified.  An
> easier format such as HTML or Markdown can be used instead.

Which indicates that others may not agree with your assertion about the
learning curve.
For simple README
https://gitlab.com/publicvoit/orgdown/-/blob/master/doc/Orgdown1-Syntax-Examples.org
is more than enough.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-11  6:52                                           ` Eli Zaretskii
@ 2022-06-12  8:40                                             ` Ihor Radchenko
  2022-06-12  8:56                                               ` Eli Zaretskii
                                                                 ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  8:40 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> Thank you, I think this is a very good summary.  I hope some of these
> issues could be worked on and improved in the future.

It is in my todo-list. I may find some time in future to work on it.

> For a serious discussion of this example, I'd need more detail about
> the aspects you mentioned above (it is not trivial to deduce them from
> looking at the top levels of org-forward-paragraph).  Specifically:
>
>   . how does paragraph definition depend on context in Org, and what
>     are the possible definitions in the existing contexts?
>   . why don't paragraph-start and paragraph-separate fit Org, and can
>     we come up with a small number of additional variables that would
>     augment these two so that the built-in forward-paragraph could be
>     extended to cover Org?

I do not want to start a serious discussion on this just yet as I do not
plant to work on this specific area in near future, however I would like
to answer some of your questions in order to provide some insight for
Emacs developers.

To answer your question about paragraph definition, we need to clarify
what "paragraph" means.

(*)
Plain text documents, in their simplest forms, usually consist of a
series of sections containing a series of paragraphs each. With
paragraphs being a series of sentences. The paragraphs are usually
separated by blank lines or by indentation.

Org mode extends the above simple structure by defining additional
syntactic elements, which are equivalent to paragraphs. Apart from
paragraphs, there can be tables, lists, other specially treated blocks
of text (source code blocks, comments, verse blocks, drawers, etc).
These elements may or may not be separated with blank lines. For
example,
| This table    | is considered as a new "paragraph"-like element |
| Not separated | from the above paragraph by blank lines         |
#+begin_src emacs-lisp
(message "And here we have yet another paragraph-like element")

;; This element may contain empty lines inside, yet these empty lines
;; are not indicating "paragraph"-like element boundary.

(message "They may, in emacs-lisp specifically, from some point of view;

but consider other possible programming languages with arbitrary syntax;
or even elisp, with the above blank line inside string.")
#+end_src

When a user calls forward-paragraph on a simple paragraph as defined in
(*), built-in forward-paragraph will behave narutally. However, things
become tricky when we have various paragraph-like elements. How can one
configure forward-paragraph to forward classic paragraph, tables, source
blocks, and all other possible elements?

To make things more complex, Org mode syntax is not context-free.
#+end_src
in here does not belong to a source block because it lacks #+begin_src
opener.
Also,
:DRAWER-LIKE-THIS:
Cannot intersect with other source-block-looking constructs like
#+begin_src emacs-lisp
(message "I am not a source block")
:END:
The
#+end_src
here is also not a part of source block because it cannot intersect
drawer boundary.

>> If you know a better way to resolve the described limitation, please let
>> me know.
>
> The better way, IMO, is this:
>
>   . try to find a way of extending the built-in forward-paragraph to
>     cover the Org use cases, by adding variables in addition to the 2
>     existing ones (I don't yet understand why "regexps do not cut it
>     for Org mode -- it sounds like a very strong assertion)

For regexps, I hope that the above example illustrated the problem
clearly. Also, note that a number of people attempted to develop
BNF/EBNF parsers for Org mode, but failed because of issues tike the
above. Org is not context-free and cannot be parsed using LR parsers,
let alone simple regexps.

For more insight, you may also consult org-element-paragraph-parser
code, which has to deal with the problems illustrated above. (note that
even this function alone is not sufficient to find paragraphs; it also
relies on other parser code).

>   . if the above doesn't work, make forward-paragraph work through
>     forward-paragraph-function, so that Org could define its own
>     function

Introducing the required flexibility into Emacs core is certainly an
option. Though it is somewhat difficult one because Org also has to
support older versions of Emacs. We are somewhat limited in using newly
introduced built-in features.

>   . in any case, the add-on convenience features of
>     org-forward-paragraph should be provided as options that the user
>     can control

They are not necessarily add-ons. First of all, org-forward-paragraph is
dealing with the above complications. Using forward-paragraph directly
would behave unexpectedly on various Org elements.

> My main concern is that forward-paragraph is used when editing the
> purely textual parts of an Org document, and when used there, casual
> users of Org will expect it to behave as the command behaves in plain
> text.  Any deviation from the behavior of the built-in command will
> confuse such users when they edit the plain-text parts, and should
> therefore be avoided.

There is no difference between forward-paragraph and
org-forward-paragraph on purely texture parts. The difference is only on
the Org-specific constructs, where forward-paragraph would behave
unexpectedly.

>> The following functions have a natural fallback to default behavior and
>> might be moved to minor-modes enabled by default. However, their default
>> behavior is context dependent and motivated by the lack of flexibility
>> of the built-in equivalents. Similar to the above functions.
>> 
>> <remap> <open-line>		org-open-line
>> <remap> <move-beginning-of-line> org-beginning-of-line
>> <remap> <move-end-of-line>	org-end-of-line
>
> Each of these (and other examples you provided) should require a
> separate serious discussion.  Let's take open-line as an example.  It
> is basically the built-in open-line, unless org-special-ctrl-o is
> non-nil.  A trivial extension of the built-in command could have
> prevented the need to remap.  (And why does the support for tables in
> org-open-line need to be turned on outside of orgtbl-mode?)

You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
provided for users who want to edit tables "like in Org mode", but in
_other_ major modes. Inside Org mode, tables are always supported.

Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
on Org tables can produce incorrect formatting. For example, calling
open-line on the following table

| this | is        | table |
| with | <point> 2 | lines |

will produce

| this | is | table |
| with | 
 2 | lines |

while org-open-line will produce

| this | is | table |
|      |    |       |
| with |  2 | lines |

The default behavior is not satisfactory and somewhat unexpected.

>> <remap> <yank>			org-yank
>> may be instead implemented using yank-handler property (it takes care
>> about adjusting level of the inserted headlines), but not completely. We
>> cannot control properties of text from outside of Org mode and thus the
>> functionality cannot be fully implemented using built-in Emacs features
>
> IMO, org-yank is sufficiently different to justify not taking over the
> built-in command.  Again, consider a user who edits the plain-text
> portions of an Org document and expects the "usual" behavior of C-y.

I personally tend to agree here.

> When fill-paragraph-function and other means of customizing a command
> to the particular mode make sense, they will not need to be noticed,
> because they will do what the users expect.  Moreover, customizing the
> behavior of commands through those variables has a couple of important
> advantages:
>
>   . it doesn't use remapping, so it doesn't stick out and bother
>     conscientious Emacs users, who are used to study unfamiliar
>     commands before using them
>   . they don't require separate documentation, apart of saying that
>     paragraph-start and paragraph-separate can be modified by modes as
>     appropriate -- all the rest of the command's documentation is
>     still intact and easily grasped in the context of any mode (this
>     is also one more reason why "niceties" like special behavior
>     inside tables should be separately controlled)

I get your point. As long as default customisations are sufficient to
achive the expected and reasonable behavior, they should indeed be
preferred.

>> I get your point on VHDL. However, I am not sure why you reject the
>> non-built-in examples. Are you implying that text modes outside Emacs
>> core are worse than built-in?
>
> No, I'm saying that, sadly, we have no real control on what the
> developers of unbundled packages decide to do.  Thus, what you see
> there is the evidence of our lack of control more than anything else.

I do not get your point here. I was referring to the markdown-mode and
Auctex to illustrate the _need_ to have numerous key bindings in order
to edit complex Org documents. It is not just something introduced into
Org out of the blue. Other people solved similar problems and did not
come up with significantly less key bindings. If you say that usual
40-50 major mode bindings are sufficient, I am not convinced.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  8:40                                             ` Ihor Radchenko
@ 2022-06-12  8:56                                               ` Eli Zaretskii
  2022-06-12  9:46                                                 ` Ihor Radchenko
  2022-06-12 13:53                                               ` Kévin Le Gouguec
  2022-06-12 20:07                                               ` Common keybindings idea for multiple markups Jean Louis
  2 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  8:56 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 16:40:31 +0800
> 
> > For a serious discussion of this example, I'd need more detail about
> > the aspects you mentioned above (it is not trivial to deduce them from
> > looking at the top levels of org-forward-paragraph).  Specifically:
> >
> >   . how does paragraph definition depend on context in Org, and what
> >     are the possible definitions in the existing contexts?
> >   . why don't paragraph-start and paragraph-separate fit Org, and can
> >     we come up with a small number of additional variables that would
> >     augment these two so that the built-in forward-paragraph could be
> >     extended to cover Org?
> 
> I do not want to start a serious discussion on this just yet as I do not
> plant to work on this specific area in near future, however I would like
> to answer some of your questions in order to provide some insight for
> Emacs developers.

Thanks, but if we are not going to continue this discussion until we
come to some conclusions, I see no reason to keep talking about it.  I
do understand better the difficulties now (thanks!), but I'm not yet
convinced that the existing solution is the best one.

> > My main concern is that forward-paragraph is used when editing the
> > purely textual parts of an Org document, and when used there, casual
> > users of Org will expect it to behave as the command behaves in plain
> > text.  Any deviation from the behavior of the built-in command will
> > confuse such users when they edit the plain-text parts, and should
> > therefore be avoided.
> 
> There is no difference between forward-paragraph and
> org-forward-paragraph on purely texture parts. The difference is only on
> the Org-specific constructs, where forward-paragraph would behave
> unexpectedly.

How does Org make sure that the different behavior doesn't happen
without the user's intent?  Isn't it possible that Org perceives
something as a sign of an "Org-specific construct" when the user
didn't mean that?  When that happens, what can the user do to avoid
the unintended (from that user's POV) Org-specific behavior?

These situations are where user control is necessary, especially for
casual, non-seasoned users of Org.

> >> <remap> <open-line>		org-open-line
> >> <remap> <move-beginning-of-line> org-beginning-of-line
> >> <remap> <move-end-of-line>	org-end-of-line
> >
> > Each of these (and other examples you provided) should require a
> > separate serious discussion.  Let's take open-line as an example.  It
> > is basically the built-in open-line, unless org-special-ctrl-o is
> > non-nil.  A trivial extension of the built-in command could have
> > prevented the need to remap.  (And why does the support for tables in
> > org-open-line need to be turned on outside of orgtbl-mode?)
> 
> You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
> provided for users who want to edit tables "like in Org mode", but in
> _other_ major modes. Inside Org mode, tables are always supported.

What if the user doesn't intend to use Org tables?  How can such a
user turn off this automatic support, which interprets some patterns
of buffer text as an indication of a table, and turns on the special
behavior reserved for Org tables?

> Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
> on Org tables can produce incorrect formatting. For example, calling
> open-line on the following table
> 
> | this | is        | table |
> | with | <point> 2 | lines |
> 
> will produce
> 
> | this | is | table |
> | with | 
>  2 | lines |
> 
> while org-open-line will produce
> 
> | this | is | table |
> |      |    |       |
> | with |  2 | lines |
> 
> The default behavior is not satisfactory and somewhat unexpected.

You assume that users always expect what Org does in that case.  This
is not a given, if we want to make Org more attractive for editing
text that is "less Org-specific", so to speak, like README files, NEWS
files, etc.

> > No, I'm saying that, sadly, we have no real control on what the
> > developers of unbundled packages decide to do.  Thus, what you see
> > there is the evidence of our lack of control more than anything else.
> 
> I do not get your point here. I was referring to the markdown-mode and
> Auctex to illustrate the _need_ to have numerous key bindings in order
> to edit complex Org documents. It is not just something introduced into
> Org out of the blue. Other people solved similar problems and did not
> come up with significantly less key bindings. If you say that usual
> 40-50 major mode bindings are sufficient, I am not convinced.

I'm not arguing with the need, I'm arguing with the particular
implementation that caters to that need.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  6:58                                             ` Eli Zaretskii
  2022-06-11  7:59                                               ` Tim Cross
@ 2022-06-12  9:05                                               ` Ihor Radchenko
  2022-06-12  9:18                                                 ` Eli Zaretskii
  2022-06-13 22:37                                                 ` Richard Stallman
  2022-06-12 22:38                                               ` Richard Stallman
  2 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  9:05 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Tim Cross, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> And if your POV is shared by the Org developers, then I ask again: why
> is the Org manual, which weighs in at 23,000 lines, is being written
> in Org?

The original reason is simply

https://list.orgmode.org/AANLkTin=gUscE4znd4pYY85X9d2wLkC+JADVcDN8AeoJ@mail.gmail.com/
>>> I was just wondering... Is the manual written in texinfo markup, or is there
>>> some obscure .org file behind the manual still?
>>> 
>>> If it really is written in texinfo, is this not a shortcoming? Org mode is
>>> capable of generating html and pdf etc. Why not use it for the manual then
>>> to set the example and show its powers!?

and then

https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
>>> It's a real pleasure to know that Org can now eat its own dog food (and
>>> to see Carsten's memorable phrase on the Org mode list again). Many
>>> thanks for picking up this orphaned project. I look forward to the day
>>> it serves as the source for org.texi.

So, now we have our manual written in Org mode and we never had reasons
to come back to texi.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  9:05                                               ` Ihor Radchenko
@ 2022-06-12  9:18                                                 ` Eli Zaretskii
  2022-06-12 10:04                                                   ` Ihor Radchenko
  2022-06-13 22:37                                                 ` Richard Stallman
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  9:18 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 17:05:02 +0800
> 
> https://list.orgmode.org/m2fu8942pr.fsf@tsdye.com/
> >>> It's a real pleasure to know that Org can now eat its own dog food (and
> >>> to see Carsten's memorable phrase on the Org mode list again). Many
> >>> thanks for picking up this orphaned project. I look forward to the day
> >>> it serves as the source for org.texi.
> 
> So, now we have our manual written in Org mode and we never had reasons
> to come back to texi.

And I get to wait for 2 minutes for the build to finish each time
org.org is touched.  Right.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  8:56                                               ` Eli Zaretskii
@ 2022-06-12  9:46                                                 ` Ihor Radchenko
  2022-06-12  9:59                                                   ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12  9:46 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I do not want to start a serious discussion on this just yet as I do not
>> plant to work on this specific area in near future, however I would like
>> to answer some of your questions in order to provide some insight for
>> Emacs developers.
>
> Thanks, but if we are not going to continue this discussion until we
> come to some conclusions, I see no reason to keep talking about it.  I
> do understand better the difficulties now (thanks!), but I'm not yet
> convinced that the existing solution is the best one.

There are some interim conclusions for now.
I agree that the approach Org takes on remapping may be improved.
I will keep in mind to reach to emacs-devel about possibility to extend
built-in Emacs functionality if it is required to get rid of the Org
remappings.

>> There is no difference between forward-paragraph and
>> org-forward-paragraph on purely texture parts. The difference is only on
>> the Org-specific constructs, where forward-paragraph would behave
>> unexpectedly.
>
> How does Org make sure that the different behavior doesn't happen
> without the user's intent?  Isn't it possible that Org perceives
> something as a sign of an "Org-specific construct" when the user
> didn't mean that?  When that happens, what can the user do to avoid
> the unintended (from that user's POV) Org-specific behavior?
>
> These situations are where user control is necessary, especially for
> casual, non-seasoned users of Org.
...
>> You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
>> provided for users who want to edit tables "like in Org mode", but in
>> _other_ major modes. Inside Org mode, tables are always supported.
>
> What if the user doesn't intend to use Org tables?  How can such a
> user turn off this automatic support, which interprets some patterns
> of buffer text as an indication of a table, and turns on the special
> behavior reserved for Org tables?

Let me be clear here. Org syntax is fixed as much as possible. One
cannot just tell Org to ignore tables in Org buffers. However, users can
alter behavior of specific Org commands if they wish to. Using available
customization and other usual Elisp tools.

One would not expect, say, tex-mode behave correctly if the user
arbitrarily changes buffer syntax table. Similarly, tables are the core
part of Org syntax.

>> Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
>> on Org tables can produce incorrect formatting. For example, calling
>> open-line on the following table
>> 
>> | this | is        | table |
>> | with | <point> 2 | lines |
>>  ...
>> The default behavior is not satisfactory and somewhat unexpected.
>
> You assume that users always expect what Org does in that case.  This
> is not a given, if we want to make Org more attractive for editing
> text that is "less Org-specific", so to speak, like README files, NEWS
> files, etc.

We do not assume that users always expect specific behavior. That's wy
org-special-ctrl-o is customization. It's default value has been agreed
upon and has not been questioned by many. The current behavior is what
most of Org users expect. The default open-line behavior is not.
Otherwise, nobody would bother implementing org-open-line.

>> > No, I'm saying that, sadly, we have no real control on what the
>> > developers of unbundled packages decide to do.  Thus, what you see
>> > there is the evidence of our lack of control more than anything else.
>> 
>> I do not get your point here. I was referring to the markdown-mode and
>> Auctex to illustrate the _need_ to have numerous key bindings in order
>> to edit complex Org documents. It is not just something introduced into
>> Org out of the blue. Other people solved similar problems and did not
>> come up with significantly less key bindings. If you say that usual
>> 40-50 major mode bindings are sufficient, I am not convinced.
>
> I'm not arguing with the need, I'm arguing with the particular
> implementation that caters to that need.

Sorry, but I do not understand what you mean here, except that you are
dissatisfied with the existing implementation. AFAIU, you objected the
number of Org bindings. That many of them are not needed. I think my
reply was targeting your objection. I did not recognize any kind of
reference to implementation specifics.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  9:46                                                 ` Ihor Radchenko
@ 2022-06-12  9:59                                                   ` Eli Zaretskii
  2022-06-15  5:13                                                     ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12  9:59 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 17:46:04 +0800
> 
> > Thanks, but if we are not going to continue this discussion until we
> > come to some conclusions, I see no reason to keep talking about it.  I
> > do understand better the difficulties now (thanks!), but I'm not yet
> > convinced that the existing solution is the best one.
> 
> There are some interim conclusions for now.
> I agree that the approach Org takes on remapping may be improved.
> I will keep in mind to reach to emacs-devel about possibility to extend
> built-in Emacs functionality if it is required to get rid of the Org
> remappings.

Thanks, I think this will indeed help in the future.

> > What if the user doesn't intend to use Org tables?  How can such a
> > user turn off this automatic support, which interprets some patterns
> > of buffer text as an indication of a table, and turns on the special
> > behavior reserved for Org tables?
> 
> Let me be clear here. Org syntax is fixed as much as possible. One
> cannot just tell Org to ignore tables in Org buffers. However, users can
> alter behavior of specific Org commands if they wish to. Using available
> customization and other usual Elisp tools.
> 
> One would not expect, say, tex-mode behave correctly if the user
> arbitrarily changes buffer syntax table. Similarly, tables are the core
> part of Org syntax.

The syntax of TeX is dictated by an external tool.  By contrast, the
syntax of Org is determined by Org itself.

Is an Org table always preceded by some directive that makes it
impossible to interpret innocent text as a table?  If so, perhaps the
problems that were bothering me don't actually exist.

But if Org interprets some text patterns as meaning that there's a
table here, that could be contrary to user expectations.

> >> | this | is        | table |
> >> | with | <point> 2 | lines |
> >>  ...
> >> The default behavior is not satisfactory and somewhat unexpected.
> >
> > You assume that users always expect what Org does in that case.  This
> > is not a given, if we want to make Org more attractive for editing
> > text that is "less Org-specific", so to speak, like README files, NEWS
> > files, etc.
> 
> We do not assume that users always expect specific behavior. That's wy
> org-special-ctrl-o is customization.

Relying on a user option for something that can need frequent
adjustment is not the best UX.  defcustom is only a good solution when
it expresses a more-or-less constant user preference that change only
very rarely.  If I may need to change the value before invoking a
command, that's inconvenient.

> It's default value has been agreed upon and has not been questioned
> by many.

Maybe because most Org users are frequent Org users and always know
what they want well enough?  As mentioned, I have other kind of users
in mind.

> >> I do not get your point here. I was referring to the markdown-mode and
> >> Auctex to illustrate the _need_ to have numerous key bindings in order
> >> to edit complex Org documents. It is not just something introduced into
> >> Org out of the blue. Other people solved similar problems and did not
> >> come up with significantly less key bindings. If you say that usual
> >> 40-50 major mode bindings are sufficient, I am not convinced.
> >
> > I'm not arguing with the need, I'm arguing with the particular
> > implementation that caters to that need.
> 
> Sorry, but I do not understand what you mean here, except that you are
> dissatisfied with the existing implementation. AFAIU, you objected the
> number of Org bindings. That many of them are not needed. I think my
> reply was targeting your objection. I did not recognize any kind of
> reference to implementation specifics.

Adding keybindings is a solution to a problem.  I'm saying that
alternatives to that solution were not seriously explored fro those
unbundled packages.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  9:18                                                 ` Eli Zaretskii
@ 2022-06-12 10:04                                                   ` Ihor Radchenko
  2022-06-12 10:15                                                     ` Eli Zaretskii
  2022-06-12 19:25                                                     ` Jean Louis
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12 10:04 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> So, now we have our manual written in Org mode and we never had reasons
>> to come back to texi.
>
> And I get to wait for 2 minutes for the build to finish each time
> org.org is touched.  Right.

This is not very polite. The change did not aim to make your life
harder.

You are free to open discussion in Org ML about turning back to texi
format, but it will, for example, make things harder for me and other
people who are not familiar with texinfo format. I am not sure if build
time justifies extra maintenance burden.

I just pushed several improvements to ox.el. They reduce manual
generation time 2x on my system (using main branch). Feel free to try it
on your side. AFAIU, the effect should be more noticeable on slower
systems.

It may also help if you try to profile org-make-manuals from
mk/org-fixup.el and share the results.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 10:04                                                   ` Ihor Radchenko
@ 2022-06-12 10:15                                                     ` Eli Zaretskii
  2022-06-12 10:38                                                       ` Ihor Radchenko
  2022-06-12 19:25                                                     ` Jean Louis
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12 10:15 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:04:56 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So, now we have our manual written in Org mode and we never had reasons
> >> to come back to texi.
> >
> > And I get to wait for 2 minutes for the build to finish each time
> > org.org is touched.  Right.
> 
> This is not very polite. The change did not aim to make your life
> harder.

It is an expression of frustration I feel each time the build needs to
rebuild the Org manual.  I wrote it to contrast that frustration with
what I perceived as satisfaction with this solution expressed in your
description of the history of this.

I really wonder how come no one on the Org list paid any attention to
the 10-fold to 40-fold slowdown in the time it takes to build the
manual, as result of that change.  But that's water under the bridge.

> You are free to open discussion in Org ML about turning back to texi
> format, but it will, for example, make things harder for me and other
> people who are not familiar with texinfo format. I am not sure if build
> time justifies extra maintenance burden.

I don't see any chance I will convince Org developers to go back to
Texinfo at this point.

> I just pushed several improvements to ox.el. They reduce manual
> generation time 2x on my system (using main branch). Feel free to try it
> on your side. AFAIU, the effect should be more noticeable on slower
> systems.

Thanks, I hope to see this soon in the Emacs repository.

> It may also help if you try to profile org-make-manuals from
> mk/org-fixup.el and share the results.

If profiling can help, wouldn't it be simpler to invoke the same
commands from an interactive Emacs session, then show the profile?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 10:15                                                     ` Eli Zaretskii
@ 2022-06-12 10:38                                                       ` Ihor Radchenko
  2022-06-12 14:36                                                         ` Eli Zaretskii
  2022-06-12 14:58                                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-12 10:38 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> It is an expression of frustration I feel each time the build needs to
> rebuild the Org manual.  I wrote it to contrast that frustration with
> what I perceived as satisfaction with this solution expressed in your
> description of the history of this.

That's not my satisfaction. I quoted the emails.
In any case, while I understand your frustration, you managed to spread
it to me at least. It did not make parsing this thread any easier.

> I really wonder how come no one on the Org list paid any attention to
> the 10-fold to 40-fold slowdown in the time it takes to build the
> manual, as result of that change.  But that's water under the bridge.

We rarely have bugs related to manual builds. I recall two in many
years.

Usually documentation is built automatically on ELPA and by our
publishing scripts on orgmode.org.

>> I just pushed several improvements to ox.el. They reduce manual
>> generation time 2x on my system (using main branch). Feel free to try it
>> on your side. AFAIU, the effect should be more noticeable on slower
>> systems.
>
> Thanks, I hope to see this soon in the Emacs repository.

Not soon. Unless you want major changes for Emacs 28.2. We restricted
stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

>> It may also help if you try to profile org-make-manuals from
>> mk/org-fixup.el and share the results.
>
> If profiling can help, wouldn't it be simpler to invoke the same
> commands from an interactive Emacs session, then show the profile?

This is exactly what I meant. To run org-make-manuals from interactive
Emacs session.

org-make-manuals takes about 20 seconds on my system (for combined Org
export and texinfo invocation). Your system is clearly different and
might have different bottlenecks.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-11 11:16                         ` Protesilaos Stavrou
@ 2022-06-12 11:32                           ` Jean Louis
  0 siblings, 0 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-12 11:32 UTC (permalink / raw)
  To: Protesilaos Stavrou; +Cc: Tim Cross, emacs-devel

* Protesilaos Stavrou <info@protesilaos.com> [2022-06-11 14:17]:
> What you describe as "garbage" package authors consider essential to
> providing high quality documentation in a variety of formats.  The
> README.org is converted into an Info manual and has a corresponding HTML
> page both on GNU ELPA and the author's (me) website:

I do not doubt good intentions. I use the word "garbage" from a
viewpoint of user who does not know Org mode and who is interested to
understand package description.

Markup impairs the understanding of package descriptions. Markup does
not belong in package description.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  2:23                         ` Tim Cross
@ 2022-06-12 11:41                           ` Jean Louis
  2022-06-12 21:58                             ` [External] : " Drew Adams
  2022-06-13 22:37                             ` Richard Stallman
  0 siblings, 2 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-12 11:41 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-devel

* Tim Cross <theophilusx@gmail.com> [2022-06-12 06:15]:
> I gues sour differences here are that you have conflated issues of
> readabiity and usability into the definition of plain text.

I did not. I speak of understanding of package descriptions. When user
clicks on a package within Emacs that user expects package
description, not a markup.

It is totally irrelevant for user if markup and tags, properties are
plain text in the context of editors. 

What matters is if user is helped to understand the package
description. Or maybe user is obstructed to understand it.

> Few of the criticisms you have raised IMO are specific to org mode
> or due to org modes syntax.

They are not specific, it just happens that I had in my package list
several Org type package descriptions. If too complex Markdown markup
is used it will also obstruct user to understand package description.

> Many of them are due to the individual author's layout and not the
> result of org mode syntax. You can also write documents which are
> very difficult to read using just plain ASCII. They are both plain
> text.

I do not think that "plain text" is relevant. HTML markup is also
plain text, it is irrelevant in the context of a user reading package
descriptions. User is not supposed to read HTML tags either, right?

> I also think you are over-stating the difficulty of reading the raw org
> file contents. While I would agree they are asier to read when correctly
> formatted, they are not impossible to understand without it.

You may stay in your viewpoint of knowledgable Emacs user, in that
sense there is also no need for package descriptions, then we can
simply tell users to read sources (irony). One has to mentally become
a user who does not know Org, the one who simply wants to find out
what package is about. Only so one can understand it.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  8:40                                             ` Ihor Radchenko
  2022-06-12  8:56                                               ` Eli Zaretskii
@ 2022-06-12 13:53                                               ` Kévin Le Gouguec
  2022-06-12 20:07                                               ` Common keybindings idea for multiple markups Jean Louis
  2 siblings, 0 replies; 376+ messages in thread
From: Kévin Le Gouguec @ 2022-06-12 13:53 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, acm, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

>>   . if the above doesn't work, make forward-paragraph work through
>>     forward-paragraph-function, so that Org could define its own
>>     function
>
> Introducing the required flexibility into Emacs core is certainly an
> option. Though it is somewhat difficult one because Org also has to
> support older versions of Emacs. We are somewhat limited in using newly
> introduced built-in features.

A tiny note on that one very specific issue: it is possible to expose
bleeding-edge core Emacs features to older Emacsen using ELPA's ":core"
package type.

Eglot, for example, makes use of improvements in eldoc.el, flymake.el,
project.el, xref.el that live exclusively on emacs's master branch, and
have not made it into a formal release yet.

As long as those core libraries are protected from changes which make
them reliant on newer features that cannot be released on ELPA[1], they
allow their dependents to keep targeting older Emacs releases.  E.g. the
master branch's project.el still aims to support Emacs 26.1.

So to some extent, "the technology exists" to allow Org to add hooks to
core Emacs features, and use those hooks through GNU ELPA, *and* support
older Emacsen.


[1] See e.g. 2022-02-03 "Don't use `string-search` in soap-client
    (bug#53744)" (1b0342628e).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 10:38                                                       ` Ihor Radchenko
@ 2022-06-12 14:36                                                         ` Eli Zaretskii
  2022-06-12 15:31                                                           ` Org mode and Emacs Colin Baxter
  2022-06-15  5:19                                                           ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Ihor Radchenko
  2022-06-12 14:58                                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  1 sibling, 2 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12 14:36 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:45 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I really wonder how come no one on the Org list paid any attention to
> > the 10-fold to 40-fold slowdown in the time it takes to build the
> > manual, as result of that change.  But that's water under the bridge.
> 
> We rarely have bugs related to manual builds. I recall two in many
> years.
> 
> Usually documentation is built automatically on ELPA and by our
> publishing scripts on orgmode.org.

So basically no one builds Org, including the manual, on their own
system?  Even not the Org developers?

> >> I just pushed several improvements to ox.el. They reduce manual
> >> generation time 2x on my system (using main branch). Feel free to try it
> >> on your side. AFAIU, the effect should be more noticeable on slower
> >> systems.
> >
> > Thanks, I hope to see this soon in the Emacs repository.
> 
> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.

This is not needed for the emacs-28 branch, so I meant master.

> >> It may also help if you try to profile org-make-manuals from
> >> mk/org-fixup.el and share the results.
> >
> > If profiling can help, wouldn't it be simpler to invoke the same
> > commands from an interactive Emacs session, then show the profile?
> 
> This is exactly what I meant. To run org-make-manuals from interactive
> Emacs session.

Then why would I need org-fixup.el?

> org-make-manuals takes about 20 seconds on my system (for combined Org
> export and texinfo invocation). Your system is clearly different and
> might have different bottlenecks.

Your Emacs is probably built with optimizations, whereas mine isn't.
The optimized version build org.info in about 30 sec, as I said, which
is not very different from your timing.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 10:38                                                       ` Ihor Radchenko
  2022-06-12 14:36                                                         ` Eli Zaretskii
@ 2022-06-12 14:58                                                         ` Eli Zaretskii
  2022-06-15  5:48                                                           ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-12 14:58 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:45 +0800
> 
> >> It may also help if you try to profile org-make-manuals from
> >> mk/org-fixup.el and share the results.
> >
> > If profiling can help, wouldn't it be simpler to invoke the same
> > commands from an interactive Emacs session, then show the profile?
> 
> This is exactly what I meant. To run org-make-manuals from interactive
> Emacs session.

I've run org-texinfo-export-to-texinfo instead.  Is that OK?

Here's the profile, collected after loading ox-texinfo.el (NOT .elc):

          35  54% - ...
          31  48%    Automatic GC
           4   6%  - minibuffer-complete
           4   6%   - completion-in-region
           4   6%    - completion--in-region
           4   6%     - #<compiled -0x1b56346374fd962>
           4   6%      - apply
           4   6%       - #<compiled 0x1145271b5beccb90>
           4   6%        - completion--in-region-1
           4   6%         - completion--do-completion
           4   6%          - completion-try-completion
           4   6%           - completion--nth-completion
           4   6%            - completion--some
           4   6%             - #<compiled -0x1c897b8c342611de>
           4   6%              - completion-basic-try-completion
           4   6%               - try-completion
           4   6%                - #<compiled -0x1144c96014867ed9>
           4   6%                   complete-with-action
          29  45% - command-execute
          29  45%  - call-interactively
          25  39%   - funcall-interactively
          25  39%    - execute-extended-command
          25  39%     - command-execute
          25  39%      - call-interactively
          25  39%       - funcall-interactively
          25  39%        - org-texinfo-export-to-texinfo
          25  39%         - let
          23  35%          - org-export-to-file
          23  35%           - org-export-as
          10  15%            - org-macro-initialize-templates
          10  15%             - org-macro--collect-macros
           5   7%                org-macro--find-keyword-value
           3   4%              - org-collect-keywords
           3   4%               - org--collect-keywords-1
           1   1%                - org--collect-keywords-1
           1   1%                 - org-element-at-point
           1   1%                  - org-element--parse-to
           1   1%                     org-element--current-element
           2   3%              - org-macro--find-date
           2   3%                 org-macro--find-keyword-value
           6   9%              org-export--delete-comment-trees
           4   6%              org-export-expand-include-keyword
           3   4%            - org-macro-replace-all
           3   4%             - org-macro-expand
           3   4%              - apply
           3   4%               - #<lambda 0xf2f300dd4>
           3   4%                - save-current-buffer
           3   4%                 - set-buffer
           3   4%                  - find-file-noselect
           2   3%                   - find-file-noselect-1
           1   1%                    - after-find-file
           1   1%                     - normal-mode
           1   1%                      - set-auto-mode
           1   1%                       - set-buffer-major-mode
           1   1%                        - fundamental-mode
           1   1%                         - run-mode-hooks
           1   1%                          - hack-local-variables
           1   1%                           - hack-dir-local-variables
           1   1%                            - hack-dir-local--get-variables
           1   1%                             - dir-locals-find-file
           1   1%                              - locate-dominating-file
           1   1%                                 dir-locals--all-files
           1   1%                     find-buffer-visiting
           2   3%            org-export-output-file-name
           4   6%   - byte-code
           4   6%    - read-extended-command
           4   6%     - completing-read
           4   6%      - completing-read-default
           2   3%         read-from-minibuffer



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12 14:36                                                         ` Eli Zaretskii
@ 2022-06-12 15:31                                                           ` Colin Baxter
  2022-06-15  5:19                                                           ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Ihor Radchenko
  1 sibling, 0 replies; 376+ messages in thread
From: Colin Baxter @ 2022-06-12 15:31 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Ihor Radchenko, theophilusx, rms, monnier, acm, emacs-devel

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Ihor Radchenko <yantar92@gmail.com> Cc:
    >> theophilusx@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca,
    >> acm@muc.de, emacs-devel@gnu.org Date: Sun, 12 Jun 2022 18:38:45
    >> +0800
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > I really wonder how come no one on the Org list paid any
    >> attention to > the 10-fold to 40-fold slowdown in the time it
    >> takes to build the > manual, as result of that change.  But
    >> that's water under the bridge.
    >> 
    >> We rarely have bugs related to manual builds. I recall two in
    >> many years.
    >> 
    >> Usually documentation is built automatically on ELPA and by our
    >> publishing scripts on orgmode.org.

    > So basically no one builds Org, including the manual, on their own
    > system?  Even not the Org developers?

I build org-mode from git and have noticed the slowness in building the
documentation.

Weekly building from git is my way of avoiding having to deal
immediately with a myriad of changes in past Versions of org-mode.

Best wishes.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  6:56                                                     ` Ihor Radchenko
@ 2022-06-12 18:29                                                       ` David Masterson
  2022-06-14  5:09                                                         ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: David Masterson @ 2022-06-12 18:29 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Tim Cross, Po Lu, Richard Stallman, monnier, acm, emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This is all part of the aims and process. However, the first step is to
>>> develop a robust elisp based parser. This helps to ensure the org syntax
>>> is consistent and helps identifies ambiguities which need to be fixed as
>>> well as provides a reference implementation which developers can use to
>>> develop parsers in other languages.
>>
>> Semantic/Bovine ??
>
> Org is not context-free.

But could it be moved in that direction? (ie. Organice)

> Also, Org maintaners previously rejected the idea of implementing Org
> parser not in Elisp. Mainly because it would limit the ability to
> maintain and contribute to Org - one would need to learn another
> programming language to alter anything in Org syntax.

Hmmm. That would make it difficult to keep the language "parseable" by a
different parser.  Elisp would not provide the checks for (say) keeping
the language context-free.

-- 
David Masterson



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 10:04                                                   ` Ihor Radchenko
  2022-06-12 10:15                                                     ` Eli Zaretskii
@ 2022-06-12 19:25                                                     ` Jean Louis
  2022-06-13 12:14                                                       ` Eli Zaretskii
  1 sibling, 1 reply; 376+ messages in thread
From: Jean Louis @ 2022-06-12 19:25 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

* Ihor Radchenko <yantar92@gmail.com> [2022-06-12 21:00]:
> You are free to open discussion in Org ML about turning back to texi
> format, but it will, for example, make things harder for me and other
> people who are not familiar with texinfo format. I am not sure if build
> time justifies extra maintenance burden.

If Org exports to texi, export that file, and how I understand it
emacs build process was using maybe Org file to convert it to texi.

Why not export texi internally in Org project, then let Emacs build
process use texi file only.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Common keybindings idea for multiple markups
  2022-06-12  8:40                                             ` Ihor Radchenko
  2022-06-12  8:56                                               ` Eli Zaretskii
  2022-06-12 13:53                                               ` Kévin Le Gouguec
@ 2022-06-12 20:07                                               ` Jean Louis
  2 siblings, 0 replies; 376+ messages in thread
From: Jean Louis @ 2022-06-12 20:07 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, acm, emacs-devel

* Ihor Radchenko <yantar92@gmail.com> [2022-06-12 20:52]:
> I was referring to the markdown-mode and
> Auctex to illustrate the _need_ to have numerous key bindings in order
> to edit complex Org documents. It is not just something introduced into
> Org out of the blue. Other people solved similar problems and did not
> come up with significantly less key bindings.

There are many markups and using various of them is very beneficial
due to various outputs and qualities. Example is Asciidoc and
Asciidoctor implementation 

AsciiDoc Writer’s Guide | Asciidoctor
https://asciidoctor.org/docs/asciidoc-writers-guide/#

and if I may say, there may be reasons why is Asciidoctor so much
financially supported:

Asciidoctor - Open Collective
https://opencollective.com/asciidoctor

or txt2tags:

txt2tags
https://txt2tags.org/

and many others like reStructuredText and Wiki and what else, too
many.

The way to go to minimize at least some key bindings is to find the
union on what is common to each of those markups.

Common markup tags are headers, italic, bold, underlined, monospaced,
lists, links, tables, separators, etc. etc.

Once it is found what is common, then Emacs and markup modes may
define how those markups look like.

Then we can use functions similar to facemenu-set-bold to get that
bold markup in a way how a markup mode specifies it.

Then if key binding is for example "C-c b" for "bold" that bold would
appear different in HTML mode, rather like <strong>word</strong> but
in Markdown it would appear as **strong** and in Org as *strong* and
so on. Markup mode would define how those common markups are to be
applied, while Emacs could have its standardized key bindings. That
would make life easier for us who use multiple markups.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/



^ permalink raw reply	[flat|nested] 376+ messages in thread

* RE: [External] : Re: Convert README.org to plain text README while installing package
  2022-06-12 11:41                           ` Jean Louis
@ 2022-06-12 21:58                             ` Drew Adams
  2022-06-13 22:37                             ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Drew Adams @ 2022-06-12 21:58 UTC (permalink / raw)
  To: Jean Louis, Tim Cross; +Cc: emacs-devel

> I did not. I speak of understanding of package descriptions. When user
> clicks on a package within Emacs that user expects package
> description, not a markup.

Dunno (I'm not really following this thread), but
maybe this example is relevant in discussing the
difference in effect between viewing/reading really
_plain_ plain-text and viewing/reading plain-text
that has some mark-up, labels, instructions, or
whatever you might call it mixed in with the text:


Consider the difference between viewing/reading a
Commentary section in a Lisp file and viewing/reading
it in the buffer that `M-x finder-commentary' creates.

This difference is about as minimal as you can get.
And yet `finder-commentary' is useful, just to give
you a friendlier, simpler, more searchable, and more
readable presentation.

All that command does is show you a buffer with just
the Commentary text - no code, comment chars removed,
etc., and with the `...'-quoted names of known
libraries converted to links to those libraries.

It's very rudimentary, but it shows well the benefit
of removing even just a few, slight distractions to
give users a really _plain_ plain-text description,
in the interest of reading it as doc.

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-11  6:58                                             ` Eli Zaretskii
  2022-06-11  7:59                                               ` Tim Cross
  2022-06-12  9:05                                               ` Ihor Radchenko
@ 2022-06-12 22:38                                               ` Richard Stallman
  2022-06-13  4:38                                                 ` Org mode and Emacs Werner LEMBERG
  2022-06-13 16:28                                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Christopher Dimech
  2 siblings, 2 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, yantar92, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > some relating to org, some relating to the fact I'm not convinced there
  > > is a need to replace texinfo, especially given that in some areas, I
  > > think texinfo is more feature rich and consistent and some relating to the feeling I have
  > > that the result is hard to justify given the amount of work it would
  > > require.

Why replace Texinfo?  Because it has some tricky syntax which it
inherits directly from TeX.  And some syntax which is cumbersome,
again imposed directly by TeX.

Nowadays nobody can maintain the TeX-based part which is used to
generate printed manuals.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  1:27                                               ` Ihor Radchenko
  2022-06-12  5:45                                                 ` Eli Zaretskii
@ 2022-06-12 22:38                                                 ` Richard Stallman
  2022-06-12 22:56                                                   ` Tim Cross
  2022-06-14 13:18                                                   ` Ihor Radchenko
  1 sibling, 2 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, eliz, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   For instance, Texinfo has @var, @emph and @dfn,
  > > all of which generate italics in printed output, but they differ in
  > > what they generate for other output formats.  There are probably 15
  > > other such constructs.

  > Could you please point to the place where I can read about the different
  > generated output?

They are in the Texinfo manual.  Look at node "Indicating", which is
section 7.1 in the version I have in Info.

(In the Info version of the file, the title of the node for @code is
actually formatted wrong.  It has single quotes around `@code' in the
title itself.  Texinfo does have some problems that need fixing, and
it is hard to fix anything in it.)

  > One key reason I worry about going down that road is that I suspect it
  > would complicate org's syntax. Two key benefits of org mode is that the
  > basic syntax is simple and it maps reasonably consistently acorss
  > different output formats. However, this flexibility does come at a cost.
  > To provide consistency across export formats, the basic formatting
  > 'concepts' need to be somewhat 'generalised', which means at times you
  > will loose some of the more advanced or sophisticated formatting power
  > of some export back-ends. 

I suspect we are slightly miscommunicating, because Texinfo already
generates several different formats of output, and each markup construct
is carefully defined about how it should appear in each output format.

So I'm sure it is possible to define additional markup constucts and
make each one do, in each output format, what Texinfo would (or does)
do with it.

The only hard part is finding syntax for them.

  > As pointed out elswhere in this thread, org could support missing
  > texinfo syntax using texinfo specific blocks. However, that isn't a
  > great experience from an authoring perspective. It is fine if you only
  > need to do it occasionally, but if you have to constantly add such
  > blocks in order to get really well formatted texinfo output,

This is rather vague.  Why do you thik this would be difficult to use
Can we find a solution to make it easy to use?

We don't have to be limited to whichever escape syntaxes Org already
satisfies.  This use is _important_.  For this, it is worth making new
ones.




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  6:02                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-12 22:38                                           ` Richard Stallman
  2022-06-13  2:29                                             ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The main problem with Texinfo that I'm aware of is its reliance on TeX
  > for the printed output formats (PDF, PS, etc.).  Many restrictions in
  > Texinfo come from the fact that something is impossible to implement
  > in TeX.

I am not sure which kind of "something" this refers to.  Formatting
capabilities?  Input syntax?

Since we want to take advantage of TeX to make beautiful printed
manuals, we will be limited to TeX's formatting capabilities.  But I think
they are more powerful than anything else.

The limitations of TeX's parser are things we could avoid, if we didn't
have TeX directly read the texinfo source code.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
       [not found]                                 ` <jwvr13z452j.fsf-monnier+emacs@gnu.org>
  2022-06-08 15:08                                   ` Org mode and Emacs Ihor Radchenko
@ 2022-06-12 22:38                                   ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: yantar92, acm, theophilusx, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since we cannot start Org from scratch, factoring out individual modules
  > > is taking a lot of time and having the hostile attitudes expressed in
  > > some of the emails in this thread is not exactly encouraging.

  > I know it's a lot of work and I'm glad you (plural) are taking it on,

I feel the same.  To move some of the features out of Org, so that
they are available in all Emacs modes, will make enhance their
availabiliy to all Emacs users.

Also, it could eventually make Org itself simpler -- and less daunting
to try out and learn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:04                                               ` Po Lu
  2022-06-12  7:02                                                 ` Ihor Radchenko
@ 2022-06-12 22:38                                                 ` Richard Stallman
  2022-06-12 22:38                                                 ` Richard Stallman
  2 siblings, 0 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Po Lu; +Cc: yandros, yantar92, theophilusx, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And anyway, I make the following offer: if someone doesn't want to
  > contribute documentation (or code) because he doesn't know Texinfo, he
  > can write the documentation in plain text, and I will translate it to
  > Texinfo for him.

Thank you for making this offer.  I hope that many contributors will
take you up on it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12  5:04                                               ` Po Lu
  2022-06-12  7:02                                                 ` Ihor Radchenko
  2022-06-12 22:38                                                 ` Richard Stallman
@ 2022-06-12 22:38                                                 ` Richard Stallman
  2 siblings, 0 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-12 22:38 UTC (permalink / raw)
  To: Po Lu; +Cc: yandros, yantar92, theophilusx, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Published Emacs Lisp packages in the wild typically come with no
  > documentation at all, aside from doc strings and a single README file.

Depending on the quality of the README file, this could be imperfect
but adequate, or it could be lousy.  Please don't accept it as a standard
to judge other things by.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 22:38                                                 ` Richard Stallman
@ 2022-06-12 22:56                                                   ` Tim Cross
  2022-06-13  0:39                                                     ` Ihor Radchenko
  2022-06-14 13:18                                                   ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-12 22:56 UTC (permalink / raw)
  To: rms; +Cc: Ihor Radchenko, eliz, monnier, acm, emacs-devel


Richard Stallman <rms@gnu.org> writes:

>   > One key reason I worry about going down that road is that I suspect it
>   > would complicate org's syntax. Two key benefits of org mode is that the
>   > basic syntax is simple and it maps reasonably consistently acorss
>   > different output formats. However, this flexibility does come at a cost.
>   > To provide consistency across export formats, the basic formatting
>   > 'concepts' need to be somewhat 'generalised', which means at times you
>   > will loose some of the more advanced or sophisticated formatting power
>   > of some export back-ends. 
>
> I suspect we are slightly miscommunicating, because Texinfo already
> generates several different formats of output, and each markup construct
> is carefully defined about how it should appear in each output format.
>
> So I'm sure it is possible to define additional markup constucts and
> make each one do, in each output format, what Texinfo would (or does)
> do with it.
>
> The only hard part is finding syntax for them.
>

Perhaps, though I tend to feel you may have misconceptions regarding the
purpose of org and its main design goals. 

The org mode syntax, especially the syntax for markup (markdown), is
very small and concise. This is one of its big strengths. It is also one
of the main reasons various markdown dialects have become so popular and
why we have so many different markdown dialects - people don't want a
rich syntax supporting numerous different sematic elements. They just
want a very few. In fact, despite the strong arguments in favour of
semantic markup, people tend to prefer typographical markup - italics,
bold, underline, etc and they want a markup where they don't have to
constantly wonder if they have selected the correct semantic tag for a
certain bit of text. They want something which is easy to write in,
which does not require referring to a manual to lookup different
semantic tags and something which is easier than HTML, LaTeX, TexInfo,
odt, xml, etc.

I also think the statement "The only hard part is finding syntax for
them." is a huge understatement. There are already a couple of requested
syntax changes for markup which IMO have far greater merit than
supporting more texinfo sematnic markup, which have not been implemented
primarily due to the difficulty of ding so in a consistent manner which
will not also break the large amount of existing org data out there. 

You mentioned in another thread that one of the big issues with texinfo
was that maintenance of the tex part, used primarily for the printed
representation, was difficult to maintain these days (assuming lack of
people with expertise in tex). I just wanted to point out that much of
the output formatting org supports is also achieved via tex and latex.
Chances are that if we extended org syntax to support the broader set of
texinfo markup, we would also inherit those same issues. At the very
least, we would not be escaping from tex simply because we moved to org
mode. 

Where I think there is a misunderstanding of org is with the emphasis
which appears to be occurring on the markup and formatting aspects of
org. Org's ability to support a small markdown dialect and generate
output in various formats is not org's main emphasis. Org's primary role
is not that of being a documentation authoring tool. Org is about a lot
more, it is about information gathering and management, task and time
management, information capture, 'executable' notes and literate
programming and lab books with convenient outline editing support. As
the name suggests, it is about helping to organise your life. It isn't
about writing documentation. The fact it can support multiple output
formats and produce reasonable looking documents is certainly part of
it, but in some ways, that was the low hanging fruit which was
relatively easy to achieve once all the other parts were in place.

If a replacement for texinfo is required, then lets take the lessons we
have learned from texinfo, latex/tex, xml, html, markdown and org and
implement something appropriate. This might seem like too much of a
task, but that is probably what people said when Carsten started
developing org mode.

There are many ways org mode can be improved. Current work to clarify
and refine the syntax, provide a solid and efficient parser, improved
font-locking and formatting and more consistent and rich API for both
extensions and new export backends are far more critical than adding
additional markup/markdown syntax to enable org mode to replace texinfo.
IMO such changes are not only misguided, they have the real potential to
adversely impact org by making things more complicated and harder to
maintain, destroying the advantage of a very small and easy to use
markup and moving the focus towards becoming an authoring tool rather
than a personal data organisation aid.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 22:56                                                   ` Tim Cross
@ 2022-06-13  0:39                                                     ` Ihor Radchenko
  2022-06-13  1:42                                                       ` Tim Cross
                                                                         ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-13  0:39 UTC (permalink / raw)
  To: Tim Cross; +Cc: rms, eliz, monnier, acm, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> There are many ways org mode can be improved. Current work to clarify
> and refine the syntax, provide a solid and efficient parser, improved
> font-locking and formatting and more consistent and rich API for both
> extensions and new export backends are far more critical than adding
> additional markup/markdown syntax to enable org mode to replace texinfo.
> IMO such changes are not only misguided, they have the real potential to
> adversely impact org by making things more complicated and harder to
> maintain, destroying the advantage of a very small and easy to use
> markup and moving the focus towards becoming an authoring tool rather
> than a personal data organisation aid.

Tim, I afraid that we a slipping to overgeneralisation again.
We may or may not need to make changes to Org syntax for the purpose of
writing documentation using Org.

Can someone please prepare a small example texi file containing examples
of various texi features that are commonly needed and possibly also
demonstrating some concrete instances of problems that exist in texinfo?

If we have such an example and know the expected outputs, someone can
then create an equivalent Org file. Then, the discussion will hopefuly
become more constructive.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  0:39                                                     ` Ihor Radchenko
@ 2022-06-13  1:42                                                       ` Tim Cross
  2022-06-13  2:40                                                         ` Ihor Radchenko
  2022-06-13  1:47                                                       ` Christopher Dimech
  2022-06-13 11:54                                                       ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  2 siblings, 1 reply; 376+ messages in thread
From: Tim Cross @ 2022-06-13  1:42 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: rms, eliz, monnier, acm, emacs-devel


Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> There are many ways org mode can be improved. Current work to clarify
>> and refine the syntax, provide a solid and efficient parser, improved
>> font-locking and formatting and more consistent and rich API for both
>> extensions and new export backends are far more critical than adding
>> additional markup/markdown syntax to enable org mode to replace texinfo.
>> IMO such changes are not only misguided, they have the real potential to
>> adversely impact org by making things more complicated and harder to
>> maintain, destroying the advantage of a very small and easy to use
>> markup and moving the focus towards becoming an authoring tool rather
>> than a personal data organisation aid.
>
> Tim, I afraid that we a slipping to overgeneralisation again.
> We may or may not need to make changes to Org syntax for the purpose of
> writing documentation using Org.
>
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?
>
> If we have such an example and know the expected outputs, someone can
> then create an equivalent Org file. Then, the discussion will hopefuly
> become more constructive.
>

Well, I think Richard was pretty clear when he asked for the additional
semantic tags listed in the Indicating Definitions, Commands & etc
section of the texinfo manual (6.1 in my copy). Looking at just that
section and it is clear current org markup syntax can handle less than
half. Some of those not handled are the types of semantic elements you
would tend to encounter in software manuals and that is just one section. 

However, I think I've made my position clear and have little more to
add, so I will bow out of any further discussion. 




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  0:39                                                     ` Ihor Radchenko
  2022-06-13  1:42                                                       ` Tim Cross
@ 2022-06-13  1:47                                                       ` Christopher Dimech
  2022-06-13  2:47                                                         ` Ihor Radchenko
  2022-06-13 11:54                                                       ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  2 siblings, 1 reply; 376+ messages in thread
From: Christopher Dimech @ 2022-06-13  1:47 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Tim Cross, rms, eliz, monnier, acm, emacs-devel



> Sent: Monday, June 13, 2022 at 12:39 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Tim Cross" <theophilusx@gmail.com>
> Cc: rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Tim Cross <theophilusx@gmail.com> writes:
>
> > There are many ways org mode can be improved. Current work to clarify
> > and refine the syntax, provide a solid and efficient parser, improved
> > font-locking and formatting and more consistent and rich API for both
> > extensions and new export backends are far more critical than adding
> > additional markup/markdown syntax to enable org mode to replace texinfo.
> > IMO such changes are not only misguided, they have the real potential to
> > adversely impact org by making things more complicated and harder to
> > maintain, destroying the advantage of a very small and easy to use
> > markup and moving the focus towards becoming an authoring tool rather
> > than a personal data organisation aid.
>
> Tim, I afraid that we a slipping to overgeneralisation again.
> We may or may not need to make changes to Org syntax for the purpose of
> writing documentation using Org.
>
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?

Does org accept latex structures?  Because texinfo is just plain tex, and as
consequence remained limited (e.g. no colouring, no bold mathematical expressions'
no boxes for formal end of proof).

> If we have such an example and know the expected outputs, someone can
> then create an equivalent Org file. Then, the discussion will hopefuly
> become more constructive.
>
> Best,
> Ihor
>
>



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 22:38                                           ` Richard Stallman
@ 2022-06-13  2:29                                             ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-13  2:29 UTC (permalink / raw)
  To: rms; +Cc: theophilusx, monnier, acm, emacs-devel

> From: Richard Stallman <rms@gnu.org>
> Cc: theophilusx@gmail.com, monnier@iro.umontreal.ca, acm@muc.de,
> 	emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 18:38:46 -0400
> 
>   > The main problem with Texinfo that I'm aware of is its reliance on TeX
>   > for the printed output formats (PDF, PS, etc.).  Many restrictions in
>   > Texinfo come from the fact that something is impossible to implement
>   > in TeX.
> 
> I am not sure which kind of "something" this refers to.  Formatting
> capabilities?  Input syntax?

The fact that Texinfo's TeX implementation is via TeX macros, AFAIU.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  1:42                                                       ` Tim Cross
@ 2022-06-13  2:40                                                         ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-13  2:40 UTC (permalink / raw)
  To: Tim Cross; +Cc: rms, eliz, monnier, acm, emacs-devel

Tim Cross <theophilusx@gmail.com> writes:

> Well, I think Richard was pretty clear when he asked for the additional
> semantic tags listed in the Indicating Definitions, Commands & etc
> section of the texinfo manual (6.1 in my copy). Looking at just that
> section and it is clear current org markup syntax can handle less than
> half. Some of those not handled are the types of semantic elements you
> would tend to encounter in software manuals and that is just one section. 

To clarify, all those extra semantics boil down to either quoting the
text inside in ASCII export or transforming them to /italics/ or
~fixed-width~ in LaTeX/HTML export. Moreover, the majority of extra
semantics is simply aliases (for example, @command is an alias to
@code).

Two exceptions are @acronym and @abbr, which can be easily added as
special link types.

In general, I so nothing in the 6.1 Indicating Definitions, Commands, etc.
that cannot be replicated using Org links or existing emphasis objects:
@code{sample} = ~sample~
@kbd{M-a}     = [[kbd:M-a]]  (with @kbdinputstyle being document keyword)
@key{RET}     = \RET (new entity type)
@samp{text}   = [[samp:text]]
@verb{text}   = =text=
@var{name}    = [[var:name]]
@env{variable}= [[env:variable]]
@file{path}   = [[file:path]]
@command{cmd} = [[command:cmd]]
@option{opt}  = [[option:opt]]
@dfn{def}     = [[dfn:def]]
@abbr{abbrev, meaning} = [[abbr:meaning][abbrev]]
same for @acronym.
@email{address,text} = [[mailto:address][text]]
@indicateurl{url} = [[url]]
@emph{text} = /text/
@strong{text} = *text*
@sc{text} = [[sc:][text]]

Note that for abbrevs, we have WIP
https://github.com/tecosaur/org-glossary

7 Quotations and Examples
@quotation: Block Quotations                    #+begin_verbatim
@indentedblock: Indented text blocks            N/A but can be a special block
@example: Example Text                          #+begin_example
@verbatim: Literal Text                         #+begin_verse
@lisp: Marking a Lisp Example                   #+begin_src
@display: Examples Using the Text Font          N/A (not sure here) but can also be a special block
@format: Examples Using the Full Line Width     I am a bit lost here, but again, anything can be a special block basically
@exdent: Undoing a Line’s Indentation           ...
@flushleft and @flushright                      ...
@raggedright: Ragged Right Text                 ...
@noindent: Omitting Indentation                 \\ I think
@indent: Forcing Indentation                    Not sure here
@cartouche: Rounded Rectangles                  Can be a block option (I am not sure if we can make it universal for all possible backends)
@small… Block Commands                          Can be a block option

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  1:47                                                       ` Christopher Dimech
@ 2022-06-13  2:47                                                         ` Ihor Radchenko
  2022-06-13  5:04                                                           ` Christopher Dimech
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-13  2:47 UTC (permalink / raw)
  To: Christopher Dimech; +Cc: Tim Cross, rms, eliz, monnier, acm, emacs-devel

Christopher Dimech <dimech@gmx.com> writes:

> Does org accept latex structures?  Because texinfo is just plain tex, and as
> consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> no boxes for formal end of proof).

Yes, latex is supported natively. You can mix it into Org text. For
non-latex export, it will be either converted to image of left as-is.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12 22:38                                               ` Richard Stallman
@ 2022-06-13  4:38                                                 ` Werner LEMBERG
  2022-06-13 16:28                                                 ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Christopher Dimech
  1 sibling, 0 replies; 376+ messages in thread
From: Werner LEMBERG @ 2022-06-13  4:38 UTC (permalink / raw)
  To: rms; +Cc: eliz, theophilusx, yantar92, monnier, acm, emacs-devel


> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.

Well, the thing is backward compatibility – for the sake of the GNU
project.  If Texinfo was allowed to drop that (in particular, the many
unfortunate restrictions of the `.info` file format), everything would
be much easier.

> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.

How do you come to this conclusion?  `texinfo.tex` is actively
maintained, and bug reports (which I submitted a lot) are usually
fixed within days.

Of course, there are some areas where more development would be
desirable (in particular, a better font interface based on ideas from
LaTeX).  On the other hand, the next Texinfo version will contain a
converter from Texinfo source to LaTeX source, which should help
overcome (at least some of) the font problems.

> (In the Info version of the file, the title of the node for @code is
> actually formatted wrong.  It has single quotes around `@code' in
> the title itself.  Texinfo does have some problems that need fixing,
> and it is hard to fix anything in it.)

I strongly suggest that you write to bug-texinfo, discussing this
issue and your other concerns.


    Werner

^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  2:47                                                         ` Ihor Radchenko
@ 2022-06-13  5:04                                                           ` Christopher Dimech
  2022-06-13  5:22                                                             ` Org mode and Emacs Werner LEMBERG
  0 siblings, 1 reply; 376+ messages in thread
From: Christopher Dimech @ 2022-06-13  5:04 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Tim Cross, rms, eliz, monnier, acm, emacs-devel


> Sent: Monday, June 13, 2022 at 2:47 PM
> From: "Ihor Radchenko" <yantar92@gmail.com>
> To: "Christopher Dimech" <dimech@gmx.com>
> Cc: "Tim Cross" <theophilusx@gmail.com>, rms@gnu.org, eliz@gnu.org, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> Christopher Dimech <dimech@gmx.com> writes:
>
> > Does org accept latex structures?  Because texinfo is just plain tex, and as
> > consequence remained limited (e.g. no colouring, no bold mathematical expressions'
> > no boxes for formal end of proof).
>
> Yes, latex is supported natively. You can mix it into Org text. For
> non-latex export, it will be either converted to image of left as-is.
>
> Best,
> Ihor

That would be very good development, using org for Official Gnu Documentation.
Had discussed with Gavin Smith about latex a couple of years ago, who explained
that texinfo would have to be rewritten completely if we are to have a latex engine
for it.  Lot of work  and unlikely to happen.






^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-13  5:04                                                           ` Christopher Dimech
@ 2022-06-13  5:22                                                             ` Werner LEMBERG
  2022-06-13  5:59                                                               ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Werner LEMBERG @ 2022-06-13  5:22 UTC (permalink / raw)
  To: dimech; +Cc: yantar92, theophilusx, rms, eliz, monnier, acm, emacs-devel


> Had discussed with Gavin Smith about latex a couple of years ago,
> who explained that texinfo would have to be rewritten completely if
> we are to have a latex engine for it.  Lot of work and unlikely to
> happen.

By the way, has someone informed the Texinfo maintainers that people
are whetting knives here on the list, discussing quite fundamental
changes about the future of the project?


    Werner



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-13  5:22                                                             ` Org mode and Emacs Werner LEMBERG
@ 2022-06-13  5:59                                                               ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-13  5:59 UTC (permalink / raw)
  To: emacs-devel, Werner LEMBERG, dimech
  Cc: yantar92, theophilusx, rms, monnier, acm

On June 13, 2022 8:22:57 AM GMT+03:00, Werner LEMBERG <wl@gnu.org> wrote:
> 
> > Had discussed with Gavin Smith about latex a couple of years ago,
> > who explained that texinfo would have to be rewritten completely if
> > we are to have a latex engine for it.  Lot of work and unlikely to
> > happen.
> 
> By the way, has someone informed the Texinfo maintainers that people
> are whetting knives here on the list, discussing quite fundamental
> changes about the future of the project?

Please don't over-dramatize.  No one is "wetting any knives" here, we are just talking.  If you followed the discussion, supplanting Texinfo is nowhere in sight yet.  This is Emacs, not GCC.

And yes, this discussion should move to the Texinfo list, as has been suggested already.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13  0:39                                                     ` Ihor Radchenko
  2022-06-13  1:42                                                       ` Tim Cross
  2022-06-13  1:47                                                       ` Christopher Dimech
@ 2022-06-13 11:54                                                       ` Eli Zaretskii
  2022-06-14 13:32                                                         ` Ihor Radchenko
  2 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-13 11:54 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  eliz@gnu.org,  monnier@iro.umontreal.ca,  acm@muc.de,
>   emacs-devel@gnu.org
> Date: Mon, 13 Jun 2022 08:39:42 +0800
> 
> Can someone please prepare a small example texi file containing examples
> of various texi features that are commonly needed and possibly also
> demonstrating some concrete instances of problems that exist in texinfo?

Given that you already produced a list of markups, is such an example
still wanted?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 19:25                                                     ` Jean Louis
@ 2022-06-13 12:14                                                       ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-13 12:14 UTC (permalink / raw)
  To: Jean Louis; +Cc: yantar92, theophilusx, rms, monnier, acm, emacs-devel

> Date: Sun, 12 Jun 2022 22:25:47 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Eli Zaretskii <eliz@gnu.org>, theophilusx@gmail.com, rms@gnu.org,
>  monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> 
> Why not export texi internally in Org project, then let Emacs build
> process use texi file only.

Because the real source is in Org, and we in the GNU Project are
always providing the real source files, not the products of their
conversion.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 22:38                                               ` Richard Stallman
  2022-06-13  4:38                                                 ` Org mode and Emacs Werner LEMBERG
@ 2022-06-13 16:28                                                 ` Christopher Dimech
  1 sibling, 0 replies; 376+ messages in thread
From: Christopher Dimech @ 2022-06-13 16:28 UTC (permalink / raw)
  To: rms; +Cc: Eli Zaretskii, theophilusx, yantar92, monnier, acm, emacs-devel


> Sent: Monday, June 13, 2022 at 10:38 AM
> From: "Richard Stallman" <rms@gnu.org>
> To: "Eli Zaretskii" <eliz@gnu.org>
> Cc: theophilusx@gmail.com, yantar92@gmail.com, monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> Subject: Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
>
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > some relating to org, some relating to the fact I'm not convinced there
>   > > is a need to replace texinfo, especially given that in some areas, I
>   > > think texinfo is more feature rich and consistent and some relating to the feeling I have
>   > > that the result is hard to justify given the amount of work it would
>   > > require.
>
> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.
>
> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.
>

Correct.  You got to move on beyond tex no matter the amount of work.

>
> --
> Dr Richard Stallman (https://stallman.org)
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)
>
>
>
>



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12  9:05                                               ` Ihor Radchenko
  2022-06-12  9:18                                                 ` Eli Zaretskii
@ 2022-06-13 22:37                                                 ` Richard Stallman
  2022-06-14  0:43                                                   ` Ihor Radchenko
  2022-06-14  2:28                                                   ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  1 sibling, 2 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-13 22:37 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: eliz, theophilusx, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So, now we have our manual written in Org mode and we never had reasons
  > to come back to texi.

I suspect that it doesn't fully follow the markup conventions
for GNU manuals.  That's because we designed Texinfo to have markup commands
to make all the proper semantic distinctions.  If the manual source
is written in a language which doesn't have the full gamut of markup
distinctions, there is no way to do the markup correctly.

It would be useful for someone who understands these conventions
to check the Org manual and see.

I don't think the slowness of processing nowadays is the crucial issue
here.  Computers are much faster now than in the 1990s.  It used
to take a long time for TeX to process the Emacs Lisp Reference Manual.
Now it is perhaps 10 times as fast.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12 11:41                           ` Jean Louis
  2022-06-12 21:58                             ` [External] : " Drew Adams
@ 2022-06-13 22:37                             ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Richard Stallman @ 2022-06-13 22:37 UTC (permalink / raw)
  To: Jean Louis; +Cc: theophilusx, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


  > I did not. I speak of understanding of package descriptions. When user
  > clicks on a package within Emacs that user expects package
  > description, not a markup.

  > It is totally irrelevant for user if markup and tags, properties are
  > plain text in the context of editors. 

The idea of adding markup to text is that a program will read that
marked-up text and output text which is nicely formatted.  The
marked-up text is harder to read, but the markup effects in the output
make it easier to read.

If we use markup in some text, we shouldn't normally show users that
marked-up source text; we should normally show users the result of
processing the markup.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13 22:37                                                 ` Richard Stallman
@ 2022-06-14  0:43                                                   ` Ihor Radchenko
       [not found]                                                     ` <87h74mv56b.fsf@localhost>
  2022-06-14  2:28                                                   ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14  0:43 UTC (permalink / raw)
  To: rms; +Cc: eliz, theophilusx, monnier, acm, emacs-devel, emacs-orgmode

I am CCing Org ML from now signifying that this branch of the thread is
directly relevent to Org mode and might be of interest for other Org
contributors.

Richard Stallman <rms@gnu.org> writes:

>   > So, now we have our manual written in Org mode and we never had reasons
>   > to come back to texi.
>
> I suspect that it doesn't fully follow the markup conventions
> for GNU manuals.  That's because we designed Texinfo to have markup commands
> to make all the proper semantic distinctions.  If the manual source
> is written in a language which doesn't have the full gamut of markup
> distinctions, there is no way to do the markup correctly.
>
> It would be useful for someone who understands these conventions
> to check the Org manual and see.

Yes, it would certainly help!
The work on manual has been done a long time ago and we also extended
our texinfo exported to suit the manual at that time. Assumingly, just
enough to handle the Org manual use-cases.

Note that we have doc/Documentation_Standards.org explaining some of the
conventions.

Here is a possibly relevant note inside it:

 - Only two of the standard Texinfo indexes are used; those for
   concepts and keys.  This has some implications:

   + The preference is to document commands by key rather than by name

   + Texinfo commands such as @var and @defoption are not used.  The
     preference for this type of thing is that the user browses the
     customize groups.  If you want or need to refer to, say, a
     variable then document it as "the variable
     @code{org-startup-folded}"
 
   + Entries in the concept index are normally all lower case unless
     some other rule dictates otherwise.

Without knowing texinfo, the above paragraphs do not make a whole lot of
sense for me. So, if someone points out any omissions, it would be
helpful for future Org contributors.

> I don't think the slowness of processing nowadays is the crucial issue
> here.  Computers are much faster now than in the 1990s.  It used
> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> Now it is perhaps 10 times as fast.

Generally, there is no way Org export to .info gets any faster than
texinfo. Org is only able to export to other text formats: org->texi;
org->tex; org->html; etc. Convertion to more low-level formats is left
to the external tools like texinfo and pdflatex.

As for reports on the slow performance, they are still useful as long as
they reveal some bottlenecks in Org exporter. Org is being used to
export large documents and whole websites. Hence, improving performance
in this area is generally helpful, even if it is not strictly a critical
blocking issue.

What I want to say is: do not expect Org export to be faster than native
binary tools, but do not hesitate to report performance issues either.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13 22:37                                                 ` Richard Stallman
  2022-06-14  0:43                                                   ` Ihor Radchenko
@ 2022-06-14  2:28                                                   ` Eli Zaretskii
  2022-06-14  2:45                                                     ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-14  2:28 UTC (permalink / raw)
  To: rms; +Cc: yantar92, theophilusx, monnier, acm, emacs-devel

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, theophilusx@gmail.com, monnier@iro.umontreal.ca,
>  acm@muc.de, emacs-devel@gnu.org
> Date: Mon, 13 Jun 2022 18:37:04 -0400
> 
> I don't think the slowness of processing nowadays is the crucial issue
> here.  Computers are much faster now than in the 1990s.  It used
> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> Now it is perhaps 10 times as fast.

The Emacs Lisp Reference manual gets processed in about 15 seconds
here.  So 2.5 minutes required to process org.org into org.texi is
quite annoying.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14  2:28                                                   ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-14  2:45                                                     ` Ihor Radchenko
  2022-06-14 11:04                                                       ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14  2:45 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: rms, theophilusx, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I don't think the slowness of processing nowadays is the crucial issue
>> here.  Computers are much faster now than in the 1990s.  It used
>> to take a long time for TeX to process the Emacs Lisp Reference Manual.
>> Now it is perhaps 10 times as fast.
>
> The Emacs Lisp Reference manual gets processed in about 15 seconds
> here.  So 2.5 minutes required to process org.org into org.texi is
> quite annoying.

To be fair, 2.5 minutes are likely caused by unoptimized Emacs build
because Org has to use the same unoptimized Emacs executable. In
contrast, texinfo is an external tool with release build.

So, while some generic improvements can likely be made on Org side, do
not expect orders of magnitude improvements.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-12 18:29                                                       ` David Masterson
@ 2022-06-14  5:09                                                         ` Ihor Radchenko
  2022-06-19 23:48                                                           ` David Masterson
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14  5:09 UTC (permalink / raw)
  To: David Masterson
  Cc: Tim Cross, Po Lu, Richard Stallman, monnier, acm, emacs-devel

David Masterson <dsmasterson@gmail.com> writes:

>>> Semantic/Bovine ??
>>
>> Org is not context-free.
>
> But could it be moved in that direction? (ie. Organice)

I don't think so. It is motivated by the fundamental Org syntax design,
AFAIU. (mostly by first match wins design). We are not going to change
fundamentals of the Org syntax. It will break backward compatibility.

>> Also, Org maintaners previously rejected the idea of implementing Org
>> parser not in Elisp. Mainly because it would limit the ability to
>> maintain and contribute to Org - one would need to learn another
>> programming language to alter anything in Org syntax.
>
> Hmmm. That would make it difficult to keep the language "parseable" by a
> different parser.  Elisp would not provide the checks for (say) keeping
> the language context-free.

At this point, we are trying to "freeze" Org syntax as much as possible.
So, major changes are not expected. Different parsers should not suffer
from future changes (if they do, we should not make those changes to
start with).

As for keeping checks, we do have a set of parser tests using ERT. So,
major breakage will be prevented. On top of this, we plan to make the
parser tests more friendly to third-party tools:
https://orgmode.org/list/87fsqzi4tw.fsf@localhost

Best,
Ihor





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14  2:45                                                     ` Ihor Radchenko
@ 2022-06-14 11:04                                                       ` Eli Zaretskii
  2022-06-14 11:18                                                         ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-14 11:04 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: rms, theophilusx, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 10:45:02 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I don't think the slowness of processing nowadays is the crucial issue
> >> here.  Computers are much faster now than in the 1990s.  It used
> >> to take a long time for TeX to process the Emacs Lisp Reference Manual.
> >> Now it is perhaps 10 times as fast.
> >
> > The Emacs Lisp Reference manual gets processed in about 15 seconds
> > here.  So 2.5 minutes required to process org.org into org.texi is
> > quite annoying.
> 
> To be fair, 2.5 minutes are likely caused by unoptimized Emacs build
> because Org has to use the same unoptimized Emacs executable.

As an Emacs maintainer, I must have the development version built
without optimizations in at least one branch, because I need to be
able to debug the latest Emacs code at all times.  So for me (and I
presume other frequent contributors to the development) building and
using an unoptimized binary is a must, part of my daily routine.

An optimized build of Emacs produces org.texi in about 30 to 40
seconds here, and that is also quite annoying for a production build.
Lars recently made significant changes in our build scripts and
support code because he didn't like much shorter delays.  We build
Emacs many times a week, so these delays add up.

> So, while some generic improvements can likely be made on Org side, do
> not expect orders of magnitude improvements.

Any improvement will be welcome, thank you.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14 11:04                                                       ` Eli Zaretskii
@ 2022-06-14 11:18                                                         ` Ihor Radchenko
  2022-06-14 11:44                                                           ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14 11:18 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: rms, theophilusx, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> An optimized build of Emacs produces org.texi in about 30 to 40
> seconds here, and that is also quite annoying for a production build.
> Lars recently made significant changes in our build scripts and
> support code because he didn't like much shorter delays.  We build
> Emacs many times a week, so these delays add up.

Is there any practical reason to re-build Org manual every time you
rebuild Emacs, even if the source did not change? I though that make
would skip building unchanged sources.

Best,
Ihor






^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14 11:18                                                         ` Ihor Radchenko
@ 2022-06-14 11:44                                                           ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-14 11:44 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: rms, theophilusx, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: rms@gnu.org,  theophilusx@gmail.com,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 19:18:57 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > An optimized build of Emacs produces org.texi in about 30 to 40
> > seconds here, and that is also quite annoying for a production build.
> > Lars recently made significant changes in our build scripts and
> > support code because he didn't like much shorter delays.  We build
> > Emacs many times a week, so these delays add up.
> 
> Is there any practical reason to re-build Org manual every time you
> rebuild Emacs, even if the source did not change?

No, and it doesn't happen unless I bootstrap.

> I though that make would skip building unchanged sources.

Indeed, it does (except as part of "make bootstrap", which rebuilds
everything).  But even without a bootstrap, when org.org changes,
building several branches needs to process it once per branch.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 22:38                                                 ` Richard Stallman
  2022-06-12 22:56                                                   ` Tim Cross
@ 2022-06-14 13:18                                                   ` Ihor Radchenko
  2022-06-14 13:38                                                     ` Org mode and Emacs Robert Pluim
  2022-06-15 23:15                                                     ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  1 sibling, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14 13:18 UTC (permalink / raw)
  To: rms; +Cc: theophilusx, eliz, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

> They are in the Texinfo manual.  Look at node "Indicating", which is
> section 7.1 in the version I have in Info.

Thanks!

>   > One key reason I worry about going down that road is that I suspect it
>   > would complicate org's syntax. Two key benefits of org mode is that the
>   > basic syntax is simple and it maps reasonably consistently acorss
>   > different output formats. However, this flexibility does come at a cost.
>   > To provide consistency across export formats, the basic formatting
>   > 'concepts' need to be somewhat 'generalised', which means at times you
>   > will loose some of the more advanced or sophisticated formatting power
>   > of some export back-ends. 
>
> I suspect we are slightly miscommunicating, because Texinfo already
> generates several different formats of output, and each markup construct
> is carefully defined about how it should appear in each output format.
>
> So I'm sure it is possible to define additional markup constucts and
> make each one do, in each output format, what Texinfo would (or does)
> do with it.
>
> The only hard part is finding syntax for them.

As I stated in another message, we may not need to define additional
markup constructs. texinfo-specific markup appears to be a little bit
too specific for general-purpose formatting aimed by Org. Instead of
modifying Org syntax (and forcing all third-party packages adapt), we
can leverage the fact that parts of Org syntax are programmable in
Elisp.

Org provides links as a flexible markup element - its export to
different formats can be arbitrarily customized (via function).
See A.3 Adding Hyperlink Types of Org manual.

For paragraph-level constructs, there is
org-export-filter-parse-tree-functions and external
https://github.com/alhassy/org-special-block-extras
We may implement something more reminiscent to link customization in
future. org-export-filter-parse-tree-functions is a bit too generic (one
function for everything).

In summary, Org does not really need to change syntax in order to
support full texinfo markup. AFAIU, the missing parts are
some of the texinfo-specific markup elements, abbreviations, and
generating index. They all can be implemented as new libraries.
Abbreviations and index are WIP at
https://github.com/tecosaur/org-glossary
The texinfo-specific markup can be another (optionally loaded) Org
library.

>   > As pointed out elswhere in this thread, org could support missing
>   > texinfo syntax using texinfo specific blocks. However, that isn't a
>   > great experience from an authoring perspective. It is fine if you only
>   > need to do it occasionally, but if you have to constantly add such
>   > blocks in order to get really well formatted texinfo output,
>
> This is rather vague.  Why do you thik this would be difficult to use
> Can we find a solution to make it easy to use?

This was a reference to Org export blocks:

#+begin_export latex
This text will only appear when exported to \LaTeX, not to any other format.
#+end_export

They may be not great experience what one needs to do something like

#+begin_export texinfo
@var{test} will only appear in .texi output.
#+end_export

#+begin_export latex
\emph{test} will only appear in .tex output.
#+end_export

many times in the same .org document.

However, we have special blocks for this purpose. They are extendable,
as I descibed above. Though some improvements in Org core might be
needed if we have to use this extensibility more frequently.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-13 11:54                                                       ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-14 13:32                                                         ` Ihor Radchenko
  2022-06-14 13:45                                                           ` Eli Zaretskii
  2022-06-15 23:15                                                           ` Richard Stallman
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-14 13:32 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> Can someone please prepare a small example texi file containing examples
>> of various texi features that are commonly needed and possibly also
>> demonstrating some concrete instances of problems that exist in texinfo?
>
> Given that you already produced a list of markups, is such an example
> still wanted?

It would certainly help with development. For example, I did not look
close into allowed markup nesting. File with tricky examples will be
helpful as a benchmark for any attempted implementation.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-14 13:18                                                   ` Ihor Radchenko
@ 2022-06-14 13:38                                                     ` Robert Pluim
  2022-06-15  6:13                                                       ` Cusom special block export, similar org org-link :export parameter (was: Org mode and Emacs) Ihor Radchenko
  2022-06-15 23:15                                                     ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
  1 sibling, 1 reply; 376+ messages in thread
From: Robert Pluim @ 2022-06-14 13:38 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: rms, theophilusx, eliz, monnier, acm, emacs-devel

>>>>> On Tue, 14 Jun 2022 21:18:58 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> #+begin_export latex
    Ihor> This text will only appear when exported to \LaTeX, not to any other format.
    Ihor> #+end_export


    Ihor> They may be not great experience what one needs to do something like

    Ihor> #+begin_export texinfo
    Ihor> @var{test} will only appear in .texi output.
    Ihor> #+end_export

    Ihor> #+begin_export latex
    Ihor> \emph{test} will only appear in .tex output.
    Ihor> #+end_export

    Ihor> many times in the same .org document.

    Ihor> However, we have special blocks for this purpose. They are extendable,
    Ihor> as I descibed above. Though some improvements in Org core might be
    Ihor> needed if we have to use this extensibility more frequently.

My first thought here was Org macros, but as far as I can tell they canʼt
be multi-line, which would make them a bit cumbersome to use. Tell us
more about special blocks, the documentation on them is a bit sparse.

Robert
-- 



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14 13:32                                                         ` Ihor Radchenko
@ 2022-06-14 13:45                                                           ` Eli Zaretskii
  2022-06-15 23:15                                                           ` Richard Stallman
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-14 13:45 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Tue, 14 Jun 2022 21:32:33 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Can someone please prepare a small example texi file containing examples
> >> of various texi features that are commonly needed and possibly also
> >> demonstrating some concrete instances of problems that exist in texinfo?
> >
> > Given that you already produced a list of markups, is such an example
> > still wanted?
> 
> It would certainly help with development. For example, I did not look
> close into allowed markup nesting. File with tricky examples will be
> helpful as a benchmark for any attempted implementation.

I'm afraid that would take someone who knows more than I do about Org,
because I don't have a clear idea what examples to provide to be
useful for this purpose.  In general the Texinfo manual has examples
of every Texinfo feature, of course.  But it is a large manual.

Specifically wrt nested markups, the most frequent ones I can think
off the top of my head are:

  . @var inside @code
  . @key inside @kbd
  . any in-line markup inside @w
  . @r, which is "escape to regular typeface" and can be used inside
    any markup that produces other typefaces

HTH



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-12  9:59                                                   ` Eli Zaretskii
@ 2022-06-15  5:13                                                     ` Ihor Radchenko
  2022-06-15 12:49                                                       ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-15  5:13 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> The syntax of TeX is dictated by an external tool.  By contrast, the
> syntax of Org is determined by Org itself.

Not exactly. We have to keep backwards compatibility and keep in mind 
external packages. So, syntax changes in Org must be avoided unless
strictly necessary. And they should be backwards compatible even if the
changes are necessary.

> Is an Org table always preceded by some directive that makes it
> impossible to interpret innocent text as a table?  If so, perhaps the
> problems that were bothering me don't actually exist.
>
> But if Org interprets some text patterns as meaning that there's a
> table here, that could be contrary to user expectations.

Org tables are always preceded by ^[ \t]*|
I am not sure what is the problem here. One can just not start lines
with | or put zero-width space if starting lines from | is absolutely
necessary.

>> We do not assume that users always expect specific behavior. That's wy
>> org-special-ctrl-o is customization.
>
> Relying on a user option for something that can need frequent
> adjustment is not the best UX.  defcustom is only a good solution when
> it expresses a more-or-less constant user preference that change only
> very rarely.  If I may need to change the value before invoking a
> command, that's inconvenient.

I am not sure why you need a frequent adjustment of org-special-ctrl-o.
This is the first time I hear about such a need.

>> It's default value has been agreed upon and has not been questioned
>> by many.
>
> Maybe because most Org users are frequent Org users and always know
> what they want well enough?  As mentioned, I have other kind of users
> in mind.

I understand, but we have to consider the needs to majority of users,
right? Customizations are there for other kind of users. Almost any kind
of default behavior, be it Org or Emacs, will suffer from some users not
liking it.

>> > I'm not arguing with the need, I'm arguing with the particular
>> > implementation that caters to that need.
>> 
>> Sorry, but I do not understand what you mean here, except that you are
>> dissatisfied with the existing implementation. AFAIU, you objected the
>> number of Org bindings. That many of them are not needed. I think my
>> reply was targeting your objection. I did not recognize any kind of
>> reference to implementation specifics.
>
> Adding keybindings is a solution to a problem.  I'm saying that
> alternatives to that solution were not seriously explored fro those
> unbundled packages.

How can you tell it with confidence? I, personally, cannot come up with
any better solution and haven't seen any alternative to the existing
numerous key bindings, except context-depended dwim bindings (which are
also frown upon). If you have an ideas about any better way to deal with
editing complex markups, please share it.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 14:36                                                         ` Eli Zaretskii
  2022-06-12 15:31                                                           ` Org mode and Emacs Colin Baxter
@ 2022-06-15  5:19                                                           ` Ihor Radchenko
  2022-06-15  6:46                                                             ` Org mode and Emacs David Engster
                                                                               ` (2 more replies)
  1 sibling, 3 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-15  5:19 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> We rarely have bugs related to manual builds. I recall two in many
>> years.
>> 
>> Usually documentation is built automatically on ELPA and by our
>> publishing scripts on orgmode.org.
>
> So basically no one builds Org, including the manual, on their own
> system?  Even not the Org developers?

Org releases are not frequent for most users and occasional build time
should not matter too much. Nobody complained on the list.

Org developers (at least, me) usually run make test. It does not touch
documentation. Full re-build is rarely needed because the documentation
does not change often.

>> > Thanks, I hope to see this soon in the Emacs repository.
>> 
>> Not soon. Unless you want major changes for Emacs 28.2. We restricted
>> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
>
> This is not needed for the emacs-28 branch, so I meant master.

I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
master only tracks our stable tagged releases. Not the working branches
(both bugfix and main). I do not know the motivation behind it and I do
not know if it is a good idea to change it. Do you think that Emacs
master should instead track our Org dev branch?

>> >> It may also help if you try to profile org-make-manuals from
>> >> mk/org-fixup.el and share the results.
>> >
>> > If profiling can help, wouldn't it be simpler to invoke the same
>> > commands from an interactive Emacs session, then show the profile?
>> 
>> This is exactly what I meant. To run org-make-manuals from interactive
>> Emacs session.
>
> Then why would I need org-fixup.el?

I assume that you figured that org-fixup.el contains functions to build
manual.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-12 14:58                                                         ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-15  5:48                                                           ` Ihor Radchenko
  2022-06-15 16:49                                                             ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-15  5:48 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> This is exactly what I meant. To run org-make-manuals from interactive
>> Emacs session.
>
> I've run org-texinfo-export-to-texinfo instead.  Is that OK?

It is a valid option. org-make-manuals just opens the file and runs
org-texinfo-export-to-texinfo.

> Here's the profile, collected after loading ox-texinfo.el (NOT .elc):
>
>           35  54% - ...
>           31  48%    Automatic GC
>           10  15%             - org-macro--collect-macros
>            5   7%                org-macro--find-keyword-value
>            2   3%              - org-macro--find-date
>            2   3%                 org-macro--find-keyword-value
>            6   9%              org-export--delete-comment-trees
>            4   6%              org-export-expand-include-keyword

The common feature of all the above functions is that they are
structured as:
(goto-char 1) (while (re-search-forward regexp) (do-stuff))

The fact that no-sublevels are revealed by the profiler signifies that
it is regexp search that is likely taking a long time.

Note that I currently experience a significant degradation in
re-search-forward performance on some Org file using Emacs 29 master,
but not Emacs 28 and earlier.

What will happen if you run
(progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
right after exporting. Does it take siginficant amount of time?
What if you run M-: (org-macro--collect-macros) directly in the buffer?
Is it also slow? Is the profiler data more detailed in such case?

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Cusom special block export, similar org org-link :export parameter (was: Org mode and Emacs)
  2022-06-14 13:38                                                     ` Org mode and Emacs Robert Pluim
@ 2022-06-15  6:13                                                       ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-15  6:13 UTC (permalink / raw)
  To: Robert Pluim
  Cc: rms, theophilusx, eliz, monnier, acm, emacs-devel, emacs-orgmode

Robert Pluim <rpluim@gmail.com> writes:

>     Ihor> However, we have special blocks for this purpose. They are extendable,
>     Ihor> as I descibed above. Though some improvements in Org core might be
>     Ihor> needed if we have to use this extensibility more frequently.
>
> My first thought here was Org macros, but as far as I can tell they canʼt
> be multi-line, which would make them a bit cumbersome to use. Tell us
> more about special blocks, the documentation on them is a bit sparse.

Macros are meant to configure per-document level of export. Not global,
usually.

Special blocks are Org blocks with the following syntax:

#+begin_environment [optional parameters]
#+end_environment

With "environment" being any word except "center", "quote", "comment",
"example", "export", "src", and "verse".

Currently, latex export directly translates such blocks into
\begin{environment}
\end{environment}

plain text export supports special handling of
environment=justifyleft, justifyright.

html export supports
'("article" "aside" "audio" "canvas" "details" "figcaption"
    "figure" "footer" "header" "menu" "meter" "nav" "output"
    "progress" "section" "summary" "video")
special blocks

texinfo export translates special blocks into
@environment [options]
@end

odt exprot supports "annotation" and "textbox" blocks.

etc.

Because "environment" can be pretty much anything, we can have a
pre-processor in org-export-filter-parse-tree-functions that transforms
different special blocks into appropriate form for a given export
backend (texinfo, latex, ascii, html, etc). This way, we will not have
to rely on backend-specific handling.

Such pre-processor currently does not exist, but it would be fairly
trivial to implement. We may even introduce an interface similar to
org-link-set-parameters for easier configuration of various types of
special blocks.

WDYT?

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15  5:19                                                           ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Ihor Radchenko
@ 2022-06-15  6:46                                                             ` David Engster
  2022-06-15  7:36                                                               ` Ihor Radchenko
  2022-06-15 12:50                                                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
  2022-06-16  3:19                                                             ` Org mode and Emacs Pankaj Jangid
  2 siblings, 1 reply; 376+ messages in thread
From: David Engster @ 2022-06-15  6:46 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We rarely have bugs related to manual builds. I recall two in many
>>> years.
>>> 
>>> Usually documentation is built automatically on ELPA and by our
>>> publishing scripts on orgmode.org.
>>
>> So basically no one builds Org, including the manual, on their own
>> system?  Even not the Org developers?
>
> Org releases are not frequent for most users and occasional build time
> should not matter too much. Nobody complained on the list.

Well, for the record, I did complain, but that was over 11 years ago. :-)

https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

I still think Org is way too slow to be a viable documentation system
for a project like Emacs.

-David



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15  6:46                                                             ` Org mode and Emacs David Engster
@ 2022-06-15  7:36                                                               ` Ihor Radchenko
  2022-06-15 13:01                                                                 ` Eli Zaretskii
  2022-06-15 13:34                                                                 ` David Engster
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-15  7:36 UTC (permalink / raw)
  To: David Engster; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

David Engster <deng@randomsample.de> writes:

>>> So basically no one builds Org, including the manual, on their own
>>> system?  Even not the Org developers?
>>
>> Org releases are not frequent for most users and occasional build time
>> should not matter too much. Nobody complained on the list.
>
> Well, for the record, I did complain, but that was over 11 years ago. :-)
>
> https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

Point taken.

> I still think Org is way too slow to be a viable documentation system
> for a project like Emacs.

Things have changed since 2014.
On my system, using Emacs 28 and latest Org, exporting org-manual +
org-guide takes 18.8 sec. This time includes genering the .texi files
and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

org export takes: 15.4 sec, which is 4.5 x texinfo time.
Considering that Org export is more sofisticated compared to texinfo and
that Org export is written in Elisp, I see this performance as
acceptable for a documentation system.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-15  5:13                                                     ` Ihor Radchenko
@ 2022-06-15 12:49                                                       ` Eli Zaretskii
  2022-06-17  5:55                                                         ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-15 12:49 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:13:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The syntax of TeX is dictated by an external tool.  By contrast, the
> > syntax of Org is determined by Org itself.
> 
> Not exactly. We have to keep backwards compatibility and keep in mind 
> external packages. So, syntax changes in Org must be avoided unless
> strictly necessary. And they should be backwards compatible even if the
> changes are necessary.

This is a tangent, entirely unrelated to what I said.  My point was
that, unlike TeX source, which can only be submitted to TeX, and
therefore must follow TeX syntax and is written by people who are
familiar with TeX, Org is being advertised as a mode for editing
mostly-plain text.  And in plain text, what Org decided to take as an
indication of a table could very well be something else, because the
user just typed plain text, and was oblivious to its meaning in Org.

So Org is _unlike_ TeX mode, in that the assumption that every
environment or markup are only used where their Org meaning is
intended -- that assumption is not necessarily true, while it is in
TeX.

> > Is an Org table always preceded by some directive that makes it
> > impossible to interpret innocent text as a table?  If so, perhaps the
> > problems that were bothering me don't actually exist.
> >
> > But if Org interprets some text patterns as meaning that there's a
> > table here, that could be contrary to user expectations.
> 
> Org tables are always preceded by ^[ \t]*|

That's what I remembered, and that is exactly what bothers me in the
context of this discussion: seemingly innocent text sequences are
interpreted by Org in some very special way, and the user doesn't
always have good means to disable that interpretation when it is not
intended.

> I am not sure what is the problem here. One can just not start lines
> with | or put zero-width space if starting lines from | is absolutely
> necessary.

Imagine a user who has no idea that a space and a | at the beginning
of a line means it's an Org table, and thus won't realize that this
magically changes how commands behave in that chunk of text.  That is
the problem that was in my mind when I said that the special
table-related behavior should be better controllable and perhaps off
by default.

> >> We do not assume that users always expect specific behavior. That's wy
> >> org-special-ctrl-o is customization.
> >
> > Relying on a user option for something that can need frequent
> > adjustment is not the best UX.  defcustom is only a good solution when
> > it expresses a more-or-less constant user preference that change only
> > very rarely.  If I may need to change the value before invoking a
> > command, that's inconvenient.
> 
> I am not sure why you need a frequent adjustment of org-special-ctrl-o.

If some instances of the same sequence of characters are indeed meant
to be a table, and other instances in the same file aren't, or if I
need to edit a file where they have this meaning and soon after a file
where they don't, I'd need to flip the option on and off very often.

> >> It's default value has been agreed upon and has not been questioned
> >> by many.
> >
> > Maybe because most Org users are frequent Org users and always know
> > what they want well enough?  As mentioned, I have other kind of users
> > in mind.
> 
> I understand, but we have to consider the needs to majority of users,
> right?

Majority of Org users or majority of Emacs users?

If we want to promote Org to be used more widely and frequently, that
would inevitably mean it will be used by Emacs users who use Org only
infrequently, and those are the ones I'm thinking about.  We should
make it easier to use Org infrequently, by people who don't do
everything in Org.

> >> > I'm not arguing with the need, I'm arguing with the particular
> >> > implementation that caters to that need.
> >> 
> >> Sorry, but I do not understand what you mean here, except that you are
> >> dissatisfied with the existing implementation. AFAIU, you objected the
> >> number of Org bindings. That many of them are not needed. I think my
> >> reply was targeting your objection. I did not recognize any kind of
> >> reference to implementation specifics.
> >
> > Adding keybindings is a solution to a problem.  I'm saying that
> > alternatives to that solution were not seriously explored fro those
> > unbundled packages.
> 
> How can you tell it with confidence?

I can, because I'm talking about what the Emacs core developers did
(or, rather, did not do).  If you somehow interpreted that to allude
to the developers of the respective packages, that was not the intent.

IOW, I'm saying that when you compare the ELPA packages to Org in
these aspects, the comparison is not really valid, because Org gets
much more attention from the core developers than the ELPA packages,
especially since there's a tendency to use Org in more situations.

> If you have an ideas about any better way to deal with editing
> complex markups, please share it.

As I said before, these issues should be discussed one by one.
There's no reason to believe there will be a single solution for all
of them.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15  5:19                                                           ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Ihor Radchenko
  2022-06-15  6:46                                                             ` Org mode and Emacs David Engster
@ 2022-06-15 12:50                                                             ` Eli Zaretskii
  2022-06-16  7:03                                                               ` Ihor Radchenko
  2022-06-16  3:19                                                             ` Org mode and Emacs Pankaj Jangid
  2 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-15 12:50 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:19:10 +0800
> 
> >> > Thanks, I hope to see this soon in the Emacs repository.
> >> 
> >> Not soon. Unless you want major changes for Emacs 28.2. We restricted
> >> stable Org branch to critical-only bugfixes until Emacs 28.2 is out.
> >
> > This is not needed for the emacs-28 branch, so I meant master.
> 
> I am not too familiar with how Emacs handles Org updates. AFAIK, Emacs
> master only tracks our stable tagged releases. Not the working branches
> (both bugfix and main). I do not know the motivation behind it and I do
> not know if it is a good idea to change it. Do you think that Emacs
> master should instead track our Org dev branch?

I see no need for such significant changes in our merging practices.
Rather, I hoped that the speedup you achieved will be important enough
to make an exception and install just it in the Emacs master branch.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15  7:36                                                               ` Ihor Radchenko
@ 2022-06-15 13:01                                                                 ` Eli Zaretskii
  2022-06-16  5:36                                                                   ` Ihor Radchenko
  2022-06-15 13:34                                                                 ` David Engster
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-15 13:01 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: deng, theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 15:36:16 +0800
> 
> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
> 
> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

I don't know about "acceptable", sorry.  It is definitely "endurable",
but 18.8 seconds is annoyingly long for an optimized build of Emacs to
do anything this simple.

Can you find any other single command that's part of building Emacs
which takes a comparable amount of time in an optimized build?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15  7:36                                                               ` Ihor Radchenko
  2022-06-15 13:01                                                                 ` Eli Zaretskii
@ 2022-06-15 13:34                                                                 ` David Engster
  2022-06-16  6:50                                                                   ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: David Engster @ 2022-06-15 13:34 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

On my system, generating org.texi from org.org takes 21s, makeinfo
org.texi takes 2.7s. But it's nice to see that it has become much faster
compared to 2014.

> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

Why does the implementation language matter whether a documentation
system is acceptable? And while sophisticated, Org is actually still
missing features (like a proper index) to be a suitable replacement.

Org is about 1/16th of the whole Emacs documentation, so we're looking
at over 5min if everything was written in Org, give or take. People were
already up in arms when the switch from Texinfo v4 to v5 was done (which
switched to a Perl implementation). When documentation generation takes
a long time, writing it becomes more painful, as you cannot quickly
check the resulting output. And I think you underestimate how important
a quick build process is. Apart from developer annoyance, you need less
resources for CI, for instance.

-David



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15  5:48                                                           ` Ihor Radchenko
@ 2022-06-15 16:49                                                             ` Eli Zaretskii
  2022-06-17  6:11                                                               ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-15 16:49 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 13:48:05 +0800
> 
> What will happen if you run
> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
> right after exporting. Does it take siginficant amount of time?

It's instantaneous.

> What if you run M-: (org-macro--collect-macros) directly in the buffer?

Also instantaneous.

> Is the profiler data more detailed in such case?

Sorry, I don't understand what you mean here.  What should I do to get
that possibly more detailed profiler data?



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14 13:18                                                   ` Ihor Radchenko
  2022-06-14 13:38                                                     ` Org mode and Emacs Robert Pluim
@ 2022-06-15 23:15                                                     ` Richard Stallman
  2022-06-17  6:52                                                       ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Richard Stallman @ 2022-06-15 23:15 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, eliz, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They may be not great experience what one needs to do something like

  > #+begin_export texinfo
  > @var{test} will only appear in .texi output.
  > #+end_export

  > #+begin_export latex
  > \emph{test} will only appear in .tex output.
  > #+end_export

  > many times in the same .org document.

  > However, we have special blocks for this purpose. They are extendable,
  > as I descibed above. Though some improvements in Org core might be
  > needed if we have to use this extensibility more frequently.

Maybe that is sufficient.  It would be ok to use, for manuals, a
submode which extend the usual Org facilities.

But it would be good for that submode to define shorter command names
for these.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-14 13:32                                                         ` Ihor Radchenko
  2022-06-14 13:45                                                           ` Eli Zaretskii
@ 2022-06-15 23:15                                                           ` Richard Stallman
  2022-06-17  6:43                                                             ` Ihor Radchenko
  1 sibling, 1 reply; 376+ messages in thread
From: Richard Stallman @ 2022-06-15 23:15 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: eliz, theophilusx, monnier, acm, emacs-devel

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It would certainly help with development. For example, I did not look
  > close into allowed markup nesting. File with tricky examples will be
  > helpful as a benchmark for any attempted implementation.

I don't know for certain that every possible nesting "does the right
thing".  I do know that @var{} is used inside many other constructs.
By contrast, @dfn{} would not be nested inside or around other
contructs very much.  @key can be nested inside @kbd, and it behaves
a little differently when nested.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)





^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15  5:19                                                           ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Ihor Radchenko
  2022-06-15  6:46                                                             ` Org mode and Emacs David Engster
  2022-06-15 12:50                                                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-16  3:19                                                             ` Pankaj Jangid
  2022-06-16  4:03                                                               ` Visuwesh
  2 siblings, 1 reply; 376+ messages in thread
From: Pankaj Jangid @ 2022-06-16  3:19 UTC (permalink / raw)
  To: emacs-devel

Ihor Radchenko <yantar92@gmail.com> writes:

>> This is not needed for the emacs-28 branch,
>> so I meant master.
>
> I am not too familiar with how Emacs handles
> Org updates. AFAIK, Emacs master only tracks
> our stable tagged releases. Not the working
> branches (both bugfix and main). I do not
> know the motivation behind it and I do not
> know if it is a good idea to change it. Do
> you think that Emacs master should instead
> track our Org dev branch?
>

Emacs master is the development branch of
Emacs. I have always wanted Org dev branch to
be part of that. But may be that Org’s dev
branch is not as stable as Emacs’. Hence it is
kept that way.

I have complained about this branch confusion
once on the emacs-devel mailing list. Only to
learn that there is a separate mailing list for
org development related things.

Even for reporting bugs there was a different
mailing list for Org. I am not sure what is the
situation now. I use whichever version comes
with master branch of Emacs. And I report bugs
using ‘M-x report-emacs-bug’ only.




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-16  3:19                                                             ` Org mode and Emacs Pankaj Jangid
@ 2022-06-16  4:03                                                               ` Visuwesh
  0 siblings, 0 replies; 376+ messages in thread
From: Visuwesh @ 2022-06-16  4:03 UTC (permalink / raw)
  To: Pankaj Jangid; +Cc: emacs-devel

[வியாழன் ஜூன் 16, 2022 08:49] Pankaj Jangid wrote:

> Even for reporting bugs there was a different
> mailing list for Org. I am not sure what is the
> situation now. I use whichever version comes
> with master branch of Emacs. And I report bugs
> using ‘M-x report-emacs-bug’ only.

The situation is still the same and you're supposed to report org-mode
bugs with M-x org-submit-bug-report.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15 13:01                                                                 ` Eli Zaretskii
@ 2022-06-16  5:36                                                                   ` Ihor Radchenko
  2022-06-16  5:58                                                                     ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-16  5:36 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: deng, theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>> 
>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> I don't know about "acceptable", sorry.  It is definitely "endurable",
> but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> do anything this simple.

Ok. I pushed things a bit further. The latest Org main takes ~8sec to
generate Org manual. This _includes_ makeinfo call. Now, Org export
takes the same time with texinfo - ~4sec on my system.

Further improvements are not trivial, given my familiarity with Org
babel and Org export code.

> Can you find any other single command that's part of building Emacs
> which takes a comparable amount of time in an optimized build?

Optimized Emacs build takes >30 minutes on my system. Mainly because of
native compilation. 40, 20, or 10 seconds spent generating Org
documentation are negligible for me.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-16  5:36                                                                   ` Ihor Radchenko
@ 2022-06-16  5:58                                                                     ` Eli Zaretskii
  2022-06-16  9:55                                                                       ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-16  5:58 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: deng, theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: deng@randomsample.de,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 13:36:35 +0800
> 
> > I don't know about "acceptable", sorry.  It is definitely "endurable",
> > but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> > do anything this simple.
> 
> Ok. I pushed things a bit further. The latest Org main takes ~8sec to
> generate Org manual. This _includes_ makeinfo call. Now, Org export
> takes the same time with texinfo - ~4sec on my system.

Thanks, this is awesome.  Hope to see this speedup in the Emacs
repository soon.

> > Can you find any other single command that's part of building Emacs
> > which takes a comparable amount of time in an optimized build?
> 
> Optimized Emacs build takes >30 minutes on my system. Mainly because of
> native compilation.

The slowness of native-compilation is a long-standing bug report, so
I'd rather not use that as a reference for acceptable speed.  I meant
to compare with other commands run by the build.

> 40, 20, or 10 seconds spent generating Org documentation are
> negligible for me.

They aren't negligible, even in a native-compilation build, because
they happen at the end, when all the rest was already done, and no
large N in "-j N" of the Make command can help with that last long
wait.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-15 13:34                                                                 ` David Engster
@ 2022-06-16  6:50                                                                   ` Ihor Radchenko
  2022-06-16 10:21                                                                     ` David Engster
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-16  6:50 UTC (permalink / raw)
  To: David Engster; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

David Engster <deng@randomsample.de> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>
> On my system, generating org.texi from org.org takes 21s, makeinfo
> org.texi takes 2.7s. But it's nice to see that it has become much faster
> compared to 2014.

Can you test the latest Org main?

>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> Why does the implementation language matter whether a documentation
> system is acceptable? And while sophisticated, Org is actually still
> missing features (like a proper index) to be a suitable replacement.

Well. It was a rather hand-waving argument: Elisp is not as fast of many
other programming languages. So software written in Elisp may be slow
despite the best effort. etc etc

In any case, that argument is futile now. I managed to get Org export
work as fast as texinfo.

As for proper index, we do support index for texinfo export
specifically and for publishing websites. A more general-purpose index
support is in the works. See https://github.com/tecosaur/org-glossary

> Org is about 1/16th of the whole Emacs documentation, so we're looking
> at over 5min if everything was written in Org, give or take. People were
> already up in arms when the switch from Texinfo v4 to v5 was done (which
> switched to a Perl implementation). When documentation generation takes
> a long time, writing it becomes more painful, as you cannot quickly
> check the resulting output. And I think you underestimate how important
> a quick build process is. Apart from developer annoyance, you need less
> resources for CI, for instance.

Fair point.

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15 12:50                                                             ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Eli Zaretskii
@ 2022-06-16  7:03                                                               ` Ihor Radchenko
  2022-06-16  8:13                                                                 ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-16  7:03 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> I see no need for such significant changes in our merging practices.
> Rather, I hoped that the speedup you achieved will be important enough
> to make an exception and install just it in the Emacs master branch.

See the attached patches.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-export-get-footnote-definition-Pre-cache-referen.patch --]
[-- Type: text/x-patch, Size: 3096 bytes --]

From 3342137359f122ed7168dc75096c6a5d3839a0c2 Mon Sep 17 00:00:00 2001
Message-Id: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:05:16 +0800
Subject: [PATCH 1/8] org-export-get-footnote-definition: Pre-cache references
 in parse tree

* lisp/ox.el (org-export-get-footnote-definition): Pre-process parse
tree once to filter out all non-footnote elements.  This speeds up
subsequent footnote definition searches.
---
 lisp/ox.el | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 2a3edaa50..7f90dc36f 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3748,28 +3748,33 @@ (defun org-export-get-footnote-definition (footnote-reference info)
     (if (not label) (org-element-contents footnote-reference)
       (let ((cache (or (plist-get info :footnote-definition-cache)
 		       (let ((hash (make-hash-table :test #'equal)))
+                         ;; Cache all the footnotes in document for
+                         ;; later search.
+                         (org-element-map (plist-get info :parse-tree)
+                             '(footnote-definition footnote-reference)
+                           (lambda (f)
+		             ;; Skip any standard footnote reference
+		             ;; since those cannot contain a
+		             ;; definition.
+                             (unless (eq (org-element-property :type f) 'standard)
+                               (puthash
+                                (cons :element (org-element-property :label f))
+                                f
+                                hash)))
+                           info)
 			 (plist-put info :footnote-definition-cache hash)
 			 hash))))
 	(or
 	 (gethash label cache)
 	 (puthash label
-		  (org-element-map (plist-get info :parse-tree)
-		      '(footnote-definition footnote-reference)
-		    (lambda (f)
-		      (cond
-		       ;; Skip any footnote with a different label.
-		       ;; Also skip any standard footnote reference
-		       ;; with the same label since those cannot
-		       ;; contain a definition.
-		       ((not (equal (org-element-property :label f) label)) nil)
-		       ((eq (org-element-property :type f) 'standard) nil)
-		       ((org-element-contents f))
-		       ;; Even if the contents are empty, we can not
-		       ;; return nil since that would eventually raise
-		       ;; the error.  Instead, return the equivalent
-		       ;; empty string.
-		       (t "")))
-		    info t)
+                  (let ((hashed (gethash (cons :element label) cache)))
+                    (when hashed
+                      (or (org-element-contents hashed)
+		          ;; Even if the contents are empty, we can not
+		          ;; return nil since that would eventually raise
+		          ;; the error.  Instead, return the equivalent
+		          ;; empty string.
+                          "")))
 		  cache)
 	 (error "Definition not found for footnote %s" label))))))
 
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-export-resolve-fuzyy-link-Pre-cache-all-possible.patch --]
[-- Type: text/x-patch, Size: 2598 bytes --]

From cff9ed2f7660abaa5bd67e5de87416cb393b2cb8 Mon Sep 17 00:00:00 2001
Message-Id: <cff9ed2f7660abaa5bd67e5de87416cb393b2cb8.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:06:47 +0800
Subject: [PATCH 2/8] org-export-resolve-fuzyy-link: Pre-cache all possible
 search cells

* lisp/ox.el (org-export-resolve-fuzzy-link): Before matching LINK,
pre-process and cache all the non-nil search cells in the parse tree.
When matching, use the pre-processed info.  Fix the :test function for
the cache hash table.
---
 lisp/ox.el | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 7f90dc36f..4a9387519 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4346,17 +4346,27 @@ (defun org-export-resolve-fuzzy-link (link info &rest pseudo-types)
   (let* ((search-cells (org-export-string-to-search-cell
 			(org-element-property :path link)))
 	 (link-cache (or (plist-get info :resolve-fuzzy-link-cache)
-			 (let ((table (make-hash-table :test #'eq)))
+			 (let ((table (make-hash-table :test #'equal)))
+                           ;; Cache all the element search cells.
+                           (org-element-map (plist-get info :parse-tree)
+		               (append pseudo-types '(target) org-element-all-elements)
+	                     (lambda (datum)
+		               (dolist (cell (org-export-search-cells datum))
+		                 (if (gethash cell table)
+                                     (push datum (gethash cell table))
+                                   (puthash cell (list datum) table)))))
 			   (plist-put info :resolve-fuzzy-link-cache table)
 			   table)))
 	 (cached (gethash search-cells link-cache 'not-found)))
     (if (not (eq cached 'not-found)) cached
       (let ((matches
-	     (org-element-map (plist-get info :parse-tree)
-		 (append pseudo-types '(target) org-element-all-elements)
-	       (lambda (datum)
-		 (and (org-export-match-search-cell-p datum search-cells)
-		      datum)))))
+             (let (result)
+               (dolist (search-cell search-cells)
+                 (setq result
+                       (nconc
+                        result
+	                (gethash search-cell link-cache))))
+               (delq nil result))))
 	(unless matches
 	  (signal 'org-link-broken (list (org-element-property :path link))))
 	(puthash
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-org-export-resolve-id-link-Pre-cache-all-the-ids-in-.patch --]
[-- Type: text/x-patch, Size: 2832 bytes --]

From eca7a1d4dea1e8980aee6a02967d4c596114f759 Mon Sep 17 00:00:00 2001
Message-Id: <eca7a1d4dea1e8980aee6a02967d4c596114f759.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sun, 12 Jun 2022 13:32:35 +0800
Subject: [PATCH 3/8] org-export-resolve-id-link: Pre-cache all the ids in the
 parse tree

* lisp/ox.el (org-export-resolve-id-link): Pre-cache all the ids in
the parse tree for faster lookup.
---
 lisp/ox.el | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 4a9387519..b431d7119 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -4393,15 +4393,27 @@ (defun org-export-resolve-id-link (link info)
 \"custom-id\".  Throw an error if no match is found."
   (let ((id (org-element-property :path link)))
     ;; First check if id is within the current parse tree.
-    (or (org-element-map (plist-get info :parse-tree) 'headline
-	  (lambda (headline)
-	    (when (or (equal (org-element-property :ID headline) id)
-		      (equal (org-element-property :CUSTOM_ID headline) id))
-	      headline))
-	  info 'first-match)
-	;; Otherwise, look for external files.
-	(cdr (assoc id (plist-get info :id-alist)))
-	(signal 'org-link-broken (list id)))))
+    (or (let ((local-ids (or (plist-get info :id-local-cache)
+                             (let ((table (make-hash-table :test #'equal)))
+                               (org-element-map
+                                   (plist-get info :parse-tree)
+                                   'headline
+                                 (lambda (headline)
+                                   (let ((id (org-element-property :ID headline))
+                                         (custom-id (org-element-property :CUSTOM_ID headline)))
+                                     (when id
+                                       (unless (gethash id table)
+                                         (puthash id headline table)))
+                                     (when custom-id
+                                       (unless (gethash custom-id table)
+                                         (puthash custom-id headline table)))))
+                                 info)
+                               (plist-put info :id-local-cache table)
+                               table))))
+          (gethash id local-ids))
+        ;; Otherwise, look for external files.
+        (cdr (assoc id (plist-get info :id-alist)))
+        (signal 'org-link-broken (list id)))))
 
 (defun org-export-resolve-radio-link (link info)
   "Return radio-target object referenced as LINK destination.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-org-export-as-Do-not-update-buffer-settings-when-not.patch --]
[-- Type: text/x-patch, Size: 3174 bytes --]

From fb8b921066ca84aaa7034c7f13e2cc62da40d7ad Mon Sep 17 00:00:00 2001
Message-Id: <fb8b921066ca84aaa7034c7f13e2cc62da40d7ad.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:03:18 +0800
Subject: [PATCH 4/8] org-export-as: Do not update buffer settings when not
 modified

* lisp/ox.el (org-export-as): Use `buffer-chars-modified-tick' and
avoid extra invocations of `org-set-regexps-and-options' and
`org-update-radio-target-regexp' when the buffer is not changed.
Also, disable folding checks.  Folding is irrelevant inside export
buffer.
---
 lisp/ox.el | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index b431d7119..a4512270c 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2956,11 +2956,12 @@ (defun org-export-as
 		    (mapcar (lambda (o) (and (eq (nth 4 o) 'parse) (nth 1 o)))
 			    (append (org-export-get-all-options backend)
 				    org-export-options-alist))))
-	     tree)
+	     tree modified-tick)
 	;; Update communication channel and get parse tree.  Buffer
 	;; isn't parsed directly.  Instead, all buffer modifications
 	;; and consequent parsing are undertaken in a temporary copy.
 	(org-export-with-buffer-copy
+         (font-lock-mode -1)
 	 ;; Run first hook with current back-end's name as argument.
 	 (run-hook-with-args 'org-export-before-processing-hook
 			     (org-export-backend-name backend))
@@ -2972,6 +2973,7 @@ (defun org-export-as
 	 ;; potentially invasive changes.
 	 (org-set-regexps-and-options)
 	 (org-update-radio-target-regexp)
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;;  Possibly execute Babel code.  Re-run a macro expansion
 	 ;;  specifically for {{{results}}} since inline source blocks
 	 ;;  may have generated some more.  Refresh buffer properties
@@ -2979,8 +2981,10 @@ (defun org-export-as
 	 (when org-export-use-babel
 	   (org-babel-exp-process-buffer)
 	   (org-macro-replace-all '(("results" . "$1")) parsed-keywords)
-	   (org-set-regexps-and-options)
-	   (org-update-radio-target-regexp))
+           (unless (eq modified-tick (buffer-chars-modified-tick))
+	     (org-set-regexps-and-options)
+	     (org-update-radio-target-regexp))
+           (setq modified-tick (buffer-chars-modified-tick)))
 	 ;; Run last hook with current back-end's name as argument.
 	 ;; Update buffer properties and radio targets one last time
 	 ;; before parsing.
@@ -2988,8 +2992,10 @@ (defun org-export-as
 	 (save-excursion
 	   (run-hook-with-args 'org-export-before-parsing-hook
 			       (org-export-backend-name backend)))
-	 (org-set-regexps-and-options)
-	 (org-update-radio-target-regexp)
+         (unless (eq modified-tick (buffer-chars-modified-tick))
+	   (org-set-regexps-and-options)
+	   (org-update-radio-target-regexp))
+         (setq modified-tick (buffer-chars-modified-tick))
 	 ;; Update communication channel with environment.
 	 (setq info
 	       (org-combine-plists
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-doc-Makefile-Disable-GC-during-export.patch --]
[-- Type: text/x-patch, Size: 1396 bytes --]

From 7f3ee582c1105c66a66fb61786c4e086e30460b4 Mon Sep 17 00:00:00 2001
Message-Id: <7f3ee582c1105c66a66fb61786c4e086e30460b4.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 07:29:06 +0800
Subject: [PATCH 5/8] doc/Makefile: Disable GC during export

* doc/Makefile (org.texi):
(orgguide.texi): Set `gc-cons-threshold` to `most-positive=fixnum' and
thus disable garbage collection while exporting manuals.  This reduces
the manual generation time.
---
 doc/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/Makefile b/doc/Makefile
index cb6d72bdc..5911bd08a 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -31,12 +31,14 @@ org.texi:	org-manual.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-manual)'
 
 orgguide.texi:	org-guide.org
 	$(BATCH) 				      \
 	  --eval '(add-to-list `load-path "../lisp")' \
 	  --eval '(load "../mk/org-fixup.el")' 	      \
+	  --eval '(setq gc-cons-threshold most-positive-fixnum)' \
 	  --eval '(org-make-guide)'
 
 org-version.inc:	org.texi
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #7: 0006-org-export-data-Concatenate-strings-in-temporary-buf.patch --]
[-- Type: text/x-patch, Size: 3492 bytes --]

From 4a466e28999bbaa5169b6fad414cf08c6d23cd7b Mon Sep 17 00:00:00 2001
Message-Id: <4a466e28999bbaa5169b6fad414cf08c6d23cd7b.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 01:01:53 +0800
Subject: [PATCH 6/8] org-export-data: Concatenate strings in temporary buffer
 for performance

* lisp/ox.el (org-export-data): Use temporary buffer to collect export
data instead of `mapconcat'.  Using buffer puts less load on garbage
collector.
---
 lisp/ox.el | 50 ++++++++++++++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 22 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index a4512270c..ae7e41e57 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1923,28 +1923,34 @@ (defun org-export-data (data info)
 			      (and (not greaterp)
 				   (memq type org-element-recursive-objects)))
 			     (contents
-			      (mapconcat
-			       (lambda (element) (org-export-data element info))
-			       (org-element-contents
-				(if (or greaterp objectp) data
-				  ;; Elements directly containing
-				  ;; objects must have their indentation
-				  ;; normalized first.
-				  (org-element-normalize-contents
-				   data
-				   ;; When normalizing first paragraph
-				   ;; of an item or
-				   ;; a footnote-definition, ignore
-				   ;; first line's indentation.
-				   (and
-				    (eq type 'paragraph)
-				    (memq (org-element-type parent)
-					  '(footnote-definition item))
-				    (eq (car (org-element-contents parent))
-					data)
-				    (eq (org-element-property :pre-blank parent)
-					0)))))
-			       "")))
+                              (let ((export-buffer (current-buffer)))
+                                (with-temp-buffer
+                                  (dolist (element (org-element-contents
+				                    (if (or greaterp objectp) data
+				                      ;; Elements directly containing
+				                      ;; objects must have their indentation
+				                      ;; normalized first.
+				                      (org-element-normalize-contents
+				                       data
+				                       ;; When normalizing first paragraph
+				                       ;; of an item or
+				                       ;; a footnote-definition, ignore
+				                       ;; first line's indentation.
+				                       (and
+				                        (eq type 'paragraph)
+				                        (memq (org-element-type parent)
+					                      '(footnote-definition item))
+				                        (eq (car (org-element-contents parent))
+					                    data)
+				                        (eq (org-element-property :pre-blank parent)
+					                    0))))))
+                                    (insert
+                                     ;; Use right local variable
+                                     ;; environment if there are, for
+                                     ;; example, #+BIND variables.
+                                     (with-current-buffer export-buffer
+                                       (org-export-data element info))))
+                                  (buffer-string)))))
 			(broken-link-handler
 			 (funcall transcoder data
 				  (if (not greaterp) contents
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #8: 0007-org-element-map-Avoid-repetitive-plist-get-call.patch --]
[-- Type: text/x-patch, Size: 1486 bytes --]

From efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc Mon Sep 17 00:00:00 2001
Message-Id: <efd91d5f654cc0e734cd3f5cfd1f39a6219fd0dc.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 09:28:27 +0800
Subject: [PATCH 7/8] org-element-map: Avoid repetitive `plist-get' call

* lisp/org-element.el (org-element-map): Do not call `(plist-get info
:ignore-list)' on every iteration.
---
 lisp/org-element.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index 9db1406b3..20b5b0303 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -4391,6 +4391,7 @@ (defun org-element-map
 		       ;; every element it encounters.
 		       (and (not (eq category 'elements))
 			    (setq category 'elements))))))))
+         (--ignore-list (plist-get info :ignore-list))
 	 --acc)
     (letrec ((--walk-tree
 	      (lambda (--data)
@@ -4400,7 +4401,7 @@ (defun org-element-map
 		  (cond
 		   ((not --data))
 		   ;; Ignored element in an export context.
-		   ((and info (memq --data (plist-get info :ignore-list))))
+		   ((and info (memq --data --ignore-list)))
 		   ;; List of elements or objects.
 		   ((not --type) (mapc --walk-tree --data))
 		   ;; Unconditionally enter parse trees.
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: 0008-org-cite-list-citations-Cache-footnote-definition-se.patch --]
[-- Type: text/x-patch, Size: 3309 bytes --]

From 56830c1792b548e9157cdd95f4bb7c093e390a05 Mon Sep 17 00:00:00 2001
Message-Id: <56830c1792b548e9157cdd95f4bb7c093e390a05.1655362876.git.yantar92@gmail.com>
In-Reply-To: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
References: <3342137359f122ed7168dc75096c6a5d3839a0c2.1655362876.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 16 Jun 2022 10:43:29 +0800
Subject: [PATCH 8/8] org-cite-list-citations: Cache footnote-definition
 searches

* lisp/oc.el (org-cite-list-citations): Avoid quadratic complexity.
Pre-calculate list of all footnote definitions and cache the footnote
label search hits.  Do not make `org-element-map' accumulate unused
result.
---
 lisp/oc.el | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/lisp/oc.el b/lisp/oc.el
index eb5f519cb..c4cd0268c 100644
--- a/lisp/oc.el
+++ b/lisp/oc.el
@@ -808,6 +808,8 @@ (defun org-cite-list-citations (info)
   (or (plist-get info :citations)
       (letrec ((cites nil)
                (tree (plist-get info :parse-tree))
+               (definition-cache (make-hash-table :test #'equal))
+               (definition-list nil)
                (find-definition
                 ;; Find definition for standard reference LABEL.  At
                 ;; this point, it is impossible to rely on
@@ -816,11 +818,21 @@ (defun org-cite-list-citations (info)
                 ;; un-processed citation objects.  So we use
                 ;; a simplified version of the function above.
                 (lambda (label)
-                  (org-element-map tree 'footnote-definition
-                    (lambda (d)
-                      (and (equal label (org-element-property :label d))
-                           (or (org-element-contents d) "")))
-                    info t)))
+                  (or (gethash label definition-cache)
+                      (org-element-map
+                          (or definition-list
+                              (setq definition-list
+                                    (org-element-map
+                                        tree
+                                        'footnote-definition
+                                      #'identity info)))
+                          'footnote-definition
+                        (lambda (d)
+                          (and (equal label (org-element-property :label d))
+                               (puthash label
+                                        (or (org-element-contents d) "")
+                                        definition-cache)))
+                        info t))))
                (search-cites
                 (lambda (data)
                   (org-element-map data '(citation footnote-reference)
@@ -834,7 +846,8 @@ (defun org-cite-list-citations (info)
                         (_
                          (let ((label (org-element-property :label datum)))
                            (funcall search-cites
-                                    (funcall find-definition label))))))
+                                    (funcall find-definition label)))))
+                      nil)
                     info nil 'footnote-definition t))))
         (funcall search-cites tree)
         (let ((result (nreverse cites)))
-- 
2.35.1


^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-16  7:03                                                               ` Ihor Radchenko
@ 2022-06-16  8:13                                                                 ` Eli Zaretskii
  2022-06-16 11:12                                                                   ` Mattias Engdegård
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-16  8:13 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>   acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 15:03:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I see no need for such significant changes in our merging practices.
> > Rather, I hoped that the speedup you achieved will be important enough
> > to make an exception and install just it in the Emacs master branch.
> 
> See the attached patches.

Thanks, I installed these, with one exception: I limited
gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
most-positive-fixnum.  The slowdown in producing the Org manual as
result of that is negligible, and I didn't feel comfortable with
disabling GC altogether, as it caused the Emacs memory footprint go up
to 0.5GB.

These changes slash the conversion from 2.5 min to just 35 sec on my
system, with an unoptimized build of Emacs, which is a terrific
speedup.  Thanks!



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-16  5:58                                                                     ` Eli Zaretskii
@ 2022-06-16  9:55                                                                       ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-16  9:55 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: deng, theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> Optimized Emacs build takes >30 minutes on my system. Mainly because of
>> native compilation.
>
> The slowness of native-compilation is a long-standing bug report, so
> I'd rather not use that as a reference for acceptable speed.  I meant
> to compare with other commands run by the build.

The other relatively slow part of the built process without native-comp
is Scraping autoloads.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-16  6:50                                                                   ` Ihor Radchenko
@ 2022-06-16 10:21                                                                     ` David Engster
  0 siblings, 0 replies; 376+ messages in thread
From: David Engster @ 2022-06-16 10:21 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Eli Zaretskii, theophilusx, rms, monnier, acm, emacs-devel

> David Engster <deng@randomsample.de> writes:
>
>>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>>> org-guide takes 18.8 sec. This time includes genering the .texi files
>>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>>
>> On my system, generating org.texi from org.org takes 21s, makeinfo
>> org.texi takes 2.7s. But it's nice to see that it has become much faster
>> compared to 2014.
>
> Can you test the latest Org main?

I tested with latest Emacs master and with increased gc-cons-threshold
it takes ~5 seconds to generate org.texi. Great job!

-David



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-16  8:13                                                                 ` Eli Zaretskii
@ 2022-06-16 11:12                                                                   ` Mattias Engdegård
  2022-06-16 12:58                                                                     ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Mattias Engdegård @ 2022-06-16 11:12 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: Ihor Radchenko, Tim Cross, rms, Stefan Monnier, Alan Mackenzie,
	emacs-devel

16 juni 2022 kl. 10.13 skrev Eli Zaretskii <eliz@gnu.org>:

> I limited
> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
> most-positive-fixnum.

Some timings for the export to .texi (old machine, optimised build, bytecode only):

|    gc-cons-threshold | time |
|----------------------+------|
|               800000 | 14.1 | (default value)
|             25000000 |  6.2 |
|             50000000 |  5.7 |
|            100000000 |  5.7 |
| most-positive-fixnum |  5.1 |

thus Eli's choice is very good, and we really should do something about that GC of ours.

> a terrific
> speedup.  Thanks!

Echoed!




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-16 11:12                                                                   ` Mattias Engdegård
@ 2022-06-16 12:58                                                                     ` Ihor Radchenko
  2022-06-16 16:59                                                                       ` Org mode and Emacs Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-16 12:58 UTC (permalink / raw)
  To: Mattias Engdegård
  Cc: Eli Zaretskii, Tim Cross, rms, Stefan Monnier, Alan Mackenzie,
	emacs-devel

Mattias Engdegård <mattiase@acm.org> writes:

>> I limited
>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>> most-positive-fixnum.
>
> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>
> |    gc-cons-threshold | time |
> |----------------------+------|
> |               800000 | 14.1 | (default value)
> |             25000000 |  6.2 |
> |             50000000 |  5.7 |
> |            100000000 |  5.7 |
> | most-positive-fixnum |  5.1 |
>
> thus Eli's choice is very good, and we really should do something about that GC of ours.

I am wondering if there is a "universal" value suitable for one-off Emacs
batch invocations.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs
  2022-06-16 12:58                                                                     ` Ihor Radchenko
@ 2022-06-16 16:59                                                                       ` Stefan Monnier
  2022-06-17  7:18                                                                         ` Larger GC thresholds for non-interactive Emacs (was: Org mode and Emacs) Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-16 16:59 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Mattias Engdegård, Eli Zaretskii, Tim Cross, rms,
	Alan Mackenzie, emacs-devel

Ihor Radchenko [2022-06-16 20:58:07] wrote:
> Mattias Engdegård <mattiase@acm.org> writes:
>>> I limited
>>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>>> most-positive-fixnum.
>>
>> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>>
>> |    gc-cons-threshold | time |
>> |----------------------+------|
>> |               800000 | 14.1 | (default value)
>> |             25000000 |  6.2 |
>> |             50000000 |  5.7 |
>> |            100000000 |  5.7 |
>> | most-positive-fixnum |  5.1 |
>>
>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>
> I am wondering if there is a "universal" value suitable for one-off Emacs
> batch invocations.

I doubt that's the case, but of course we should try and use values that
work "overall best" on average.  Maybe we should revisit the current
knobs and their default values.

Currently the two important knobs we have are:

    gc-cons-threshold
    gc-cons-percentage

Maybe it's time for some benchmarks with various values for these knobs
to see if the values we're using now are "good enough" or might need to
be changed.

We could also try and be bit more discerning.  E.g. currently when the
program is in a phase where it builds a lot of new data-structures, we
run the GC everytime some amount of new memory has been allocated, but
presumably almost none of those objects have died yet (we're still
building the overall datastructure) so the GC will be a pure waste of
time.  OTOH in other phases the code allocates objects which are used
only temporarily, in which case it's beneficial to run the GC somewhat
frequently to avoid growing our heap unnecessarily (and suffering
fragmentation as a result).

It's hard to know beforehand whether a GC will be useful or not, tho.
But maybe we can find good heuristics.  E.g. have something like
a `gc-cons-percentage` which depends on how much garbage we collected in
the last GC: if a GC doesn't collect any garbage (or very little of it),
it's a sign that we're in a phase where running the GC is not very
useful so we should wait a bit more until the next GC, whereas if the GC
collected a fair bit of garbage, it's a sign that we're in a phase where
running the GC is beneficial and we can run it a bit more often.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-15 12:49                                                       ` Eli Zaretskii
@ 2022-06-17  5:55                                                         ` Ihor Radchenko
  2022-06-17  6:40                                                           ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  5:55 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> Not exactly. We have to keep backwards compatibility and keep in mind 
>> external packages. So, syntax changes in Org must be avoided unless
>> strictly necessary. And they should be backwards compatible even if the
>> changes are necessary.
>
> This is a tangent, entirely unrelated to what I said.  My point was
> that, unlike TeX source, which can only be submitted to TeX, and
> therefore must follow TeX syntax and is written by people who are
> familiar with TeX, Org is being advertised as a mode for editing
> mostly-plain text.  And in plain text, what Org decided to take as an
> indication of a table could very well be something else, because the
> user just typed plain text, and was oblivious to its meaning in Org.
>
> So Org is _unlike_ TeX mode, in that the assumption that every
> environment or markup are only used where their Org meaning is
> intended -- that assumption is not necessarily true, while it is in
> TeX.

I think we are mis-communicating. Org mode is not just a major mode, it
is a markup language. org-mode in Emacs is a mode for editing documents
written using Org markup. The markup is fixed with tables being a part
of it. Altering the markup rules is not currently supported for
technical reasons (specific Org markup is not parameterized in various
places in the code-base; though efforts to modulate everything, including
some markup semantics, are ongoing).

Note that Org markup has facilities to escape text constructs using
verbatim/code or #+begin_quote/#+begin_example constructs.

>> Org tables are always preceded by ^[ \t]*|
>
> That's what I remembered, and that is exactly what bothers me in the
> context of this discussion: seemingly innocent text sequences are
> interpreted by Org in some very special way, and the user doesn't
> always have good means to disable that interpretation when it is not
> intended.

I do not think that Org will support major user changes in Org syntax
any time soon or in future. At least, there is no intention to guarantee
such support.

I am not sure why you consider using pre-defined markup constructs as
"innocent". It is problematic in any kind of markup language, be it Org,
markdown, texinfo, or anything else.

>> I am not sure what is the problem here. One can just not start lines
>> with | or put zero-width space if starting lines from | is absolutely
>> necessary.
>
> Imagine a user who has no idea that a space and a | at the beginning
> of a line means it's an Org table, and thus won't realize that this
> magically changes how commands behave in that chunk of text.  That is
> the problem that was in my mind when I said that the special
> table-related behavior should be better controllable and perhaps off
> by default.

Fontification will hint about different semantic meaning.
FYI, I can blame comment-dwim for having the same problem - if user does
not know about comment syntax, comment-dwim will magically change its
meaning in different chunks of text. The same goes for open-line when
there is fill-prefix or even not-easily-visible 'left-margin text
property.

>> > Relying on a user option for something that can need frequent
>> > adjustment is not the best UX.  defcustom is only a good solution when
>> > it expresses a more-or-less constant user preference that change only
>> > very rarely.  If I may need to change the value before invoking a
>> > command, that's inconvenient.
>> 
>> I am not sure why you need a frequent adjustment of org-special-ctrl-o.
>
> If some instances of the same sequence of characters are indeed meant
> to be a table, and other instances in the same file aren't, or if I
> need to edit a file where they have this meaning and soon after a file
> where they don't, I'd need to flip the option on and off very often.

I'd say that you should not use Org on files that do not use Org syntax.
If you do what to do it, you should not be surprised that the same mode
behaves the same way despite you having different idea in mind.

More practically, you can always use file-local variables.

Or, if you think that flipping between some Emacs-global default
behavior and major-mode-specific behavior is frequent or important
use-case, why would you not introduce global Emacs functionality or at
least convention to do exactly this? I do not understand why this very
general problem is blamed on Org specifically and not on other major
modes.

>> > Maybe because most Org users are frequent Org users and always know
>> > what they want well enough?  As mentioned, I have other kind of users
>> > in mind.
>> 
>> I understand, but we have to consider the needs to majority of users,
>> right?
>
> Majority of Org users or majority of Emacs users?
>
> If we want to promote Org to be used more widely and frequently, that
> would inevitably mean it will be used by Emacs users who use Org only
> infrequently, and those are the ones I'm thinking about.  We should
> make it easier to use Org infrequently, by people who don't do
> everything in Org.

I agree here. However, I am not sure how to achieve this. We can limit
Org functionality to lower the entry barrier for infrequent Org users,
yes. However, it will necessarily mean that most of frequent Org users
have to adjust their Org mode configurations - not good.

Should we mark some of the Org commands disabled by default? Though it
would make sense to mark the whole groups of commands. Is it possible?
Is it also possible to disable commands only for users who never
customized Org?

>> > Adding keybindings is a solution to a problem.  I'm saying that
>> > alternatives to that solution were not seriously explored fro those
>> > unbundled packages.
>> 
>> How can you tell it with confidence?
>
> I can, because I'm talking about what the Emacs core developers did
> (or, rather, did not do).  If you somehow interpreted that to allude
> to the developers of the respective packages, that was not the intent.
>
> IOW, I'm saying that when you compare the ELPA packages to Org in
> these aspects, the comparison is not really valid, because Org gets
> much more attention from the core developers than the ELPA packages,
> especially since there's a tendency to use Org in more situations.

I get your point.
However, I do not see a good solution here. Suggestions are welcome.
(I'd prefer if the discussion about the principles of key binding /
editing design were in a separate thread, CCing both emacs-devel and Org
mode ML). This thread is raising so many issues of various levels of
criticality, generality, and validity that it is very hard to stay on
topic.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15 16:49                                                             ` Eli Zaretskii
@ 2022-06-17  6:11                                                               ` Ihor Radchenko
  2022-06-17  6:41                                                                 ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  6:11 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, rms, monnier, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> What will happen if you run
>> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
>> right after exporting. Does it take siginficant amount of time?
>
> It's instantaneous.
>
>> What if you run M-: (org-macro--collect-macros) directly in the buffer?
>
> Also instantaneous.

And yet it took 15% of CPU time according to your profile. Something
fishy is going on here. My guess that it is GC.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-17  5:55                                                         ` Ihor Radchenko
@ 2022-06-17  6:40                                                           ` Eli Zaretskii
  2022-06-18  4:40                                                             ` Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package) Ihor Radchenko
                                                                               ` (2 more replies)
  0 siblings, 3 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-17  6:40 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 17 Jun 2022 13:55:41 +0800
> 
> > So Org is _unlike_ TeX mode, in that the assumption that every
> > environment or markup are only used where their Org meaning is
> > intended -- that assumption is not necessarily true, while it is in
> > TeX.
> 
> I think we are mis-communicating.

Most probably, see below.

> >> Org tables are always preceded by ^[ \t]*|
> >
> > That's what I remembered, and that is exactly what bothers me in the
> > context of this discussion: seemingly innocent text sequences are
> > interpreted by Org in some very special way, and the user doesn't
> > always have good means to disable that interpretation when it is not
> > intended.
> 
> I do not think that Org will support major user changes in Org syntax
> any time soon or in future. At least, there is no intention to guarantee
> such support.
> 
> I am not sure why you consider using pre-defined markup constructs as
> "innocent". It is problematic in any kind of markup language, be it Org,
> markdown, texinfo, or anything else.

Neither of the other markup modes is being proposed for viewing and
editing documents that were not originally edited under those modes.
By contrast, there's a fraction of Emacs contributors and developers
who repeatedly suggest to use Org for documents that were not
originally written in Org.  A notable example (not the only one) is
recent discussions of turning on Org when visiting NEWS files.

If you think these ideas are problematic from the POV of Org
developers, please voice this opinion whenever such proposals are
brought up.  Those proposals, and in general the proposals to use Org
widely in unrelated contexts, is what I had in mind all the time in
this discussion.  Perhaps now you can better understand some of my
comments and responses.

For example, what is your opinion of using Org markup in email
messages?  There are a lot of examples of that, both here and on the
bug tracker.  People use Org markup and Org-style code blocks quite a
lot, and reading that is always jarring to me.  For some reason,
people assume that I read my email in Org mode or some derivative of
Org.

> I'd say that you should not use Org on files that do not use Org syntax.

See above: there are suggestions to do just that.  People are
proposing to use Org in many situations where Org was not originally
used, and therefore the "usual" Org assumptions don't hold.

> > If we want to promote Org to be used more widely and frequently, that
> > would inevitably mean it will be used by Emacs users who use Org only
> > infrequently, and those are the ones I'm thinking about.  We should
> > make it easier to use Org infrequently, by people who don't do
> > everything in Org.
> 
> I agree here. However, I am not sure how to achieve this. We can limit
> Org functionality to lower the entry barrier for infrequent Org users,
> yes. However, it will necessarily mean that most of frequent Org users
> have to adjust their Org mode configurations - not good.
> 
> Should we mark some of the Org commands disabled by default? Though it
> would make sense to mark the whole groups of commands. Is it possible?
> Is it also possible to disable commands only for users who never
> customized Org?

I don't know the answers.  But I'd appreciate if these consequences
and prerequisites of using Org in places where it wasn't used
originally could be voiced when such suggestions are brought up, and
by Org developers on top of that.  I quite frequently opposition to
such proposals is regarded as pure conservatism, so if there are valid
technical reasons and problems to solve before such proposals can be
considered practical, it would be good to have them part of those
discussions.

> This thread is raising so many issues of various levels of
> criticality, generality, and validity that it is very hard to stay on
> topic.

Agreed.  There should be a separate thread for each specific topic.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-17  6:11                                                               ` Ihor Radchenko
@ 2022-06-17  6:41                                                                 ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-17  6:41 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, rms, monnier, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  rms@gnu.org,  monnier@iro.umontreal.ca,
>  acm@muc.de,  emacs-devel@gnu.org
> Date: Fri, 17 Jun 2022 14:11:05 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> What will happen if you run
> >> (progn (goto-char 1) (while (re-search-forward "^\\*+ " nil t)))
> >> right after exporting. Does it take siginficant amount of time?
> >
> > It's instantaneous.
> >
> >> What if you run M-: (org-macro--collect-macros) directly in the buffer?
> >
> > Also instantaneous.
> 
> And yet it took 15% of CPU time according to your profile. Something
> fishy is going on here. My guess that it is GC.

That'd be strange, because AFAIK the profiler accounts for GC
separately.

Anyway, I think your latest changes close this issue for good.
Thanks!



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Org syntax compatibility with texinfo syntax (was: Org mode and Emacs (was: Convert README.org to plain text README while installing package))
       [not found]                                                     ` <87h74mv56b.fsf@localhost>
@ 2022-06-17  6:42                                                       ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  6:42 UTC (permalink / raw)
  To: rms; +Cc: eliz, theophilusx, monnier, acm, emacs-orgmode, emacs-devel

I am forwarding yet another email from the emacs-devel thread to here.
This thread includes both emacs-devel and Org ML.
I will still keep emacs-devel in the loop because people from there can
provide a useful insight about texinfo capabilities.

Richard Stallman wrote:

> I don't know for certain that every possible nesting "does the right
> thing".  I do know that @var{} is used inside many other constructs.
> By contrast, @dfn{} would not be nested inside or around other
> contructs very much.  @key can be nested inside @kbd, and it behaves
> a little differently when nested.

signifying (in addition to previous message), that
1. Texinfo has some software documentation-specific markup elements
2. Those elements can be nested in non-trivial ways, similar to Org's
org-element-object-restrictions, the that nesting can affect formatted
export (again, similar to Org).

With regards to Org syntax, I do not think that we must include all the
possible texinfo elements into Org standard.
@dfn, @key, and @var constructs are extremely specific to software
documentation and have no sense to maintain as a part of general-purpose
markup syntax.

However, what we can do is to allow extending Org syntax to support such
elements. We do support custom syntax elements already (or are
discussing such support):

We have the following, potentially customizeable syntax elements:
- Org links (extensible via org-link-set-parameters)
- Special blocks (de-facto extensible in export backends; also,
  https://github.com/alhassy/org-special-block-extras/);
  https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af
- [FR] Inline special blocks idea being discussed in
  https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net

The links are somewhat limited wrt nesting: links cannot be nested
inside links thus limiting their usefulness as custom markup. Hence, we
may need to look more closely into the idea of inline special blocks
(discuss it in https://orgmode.org/list/87a6b8pbhg.fsf@posteo.net).

Special blocks already support custom :type, but lack convenient Elisp
interface like org-link-set-parameters. (let's keep this part in
https://list.orgmode.org/87edzqv4ha.fsf@localhost/T/#m6b95119faa65645fd1c32b0e17b6440f73ecd3af)

I'd like to keep discussion on the specifics of customizeable Org syntax
in the relevant threads.

In here, I'd like to hear feedback on possible additional
incompatibilities between texinfo and Org. Is there anything (not
covered by the above list) that is present in texinfo, but impossible in
Org?

For starters, Org does not have a full support for generating index
(apart from ox-texinfo and ox-publish). Though see
https://github.com/tecosaur/org-glossary

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15 23:15                                                           ` Richard Stallman
@ 2022-06-17  6:43                                                             ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  6:43 UTC (permalink / raw)
  To: rms; +Cc: eliz, theophilusx, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

>   > It would certainly help with development. For example, I did not look
>   > close into allowed markup nesting. File with tricky examples will be
>   > helpful as a benchmark for any attempted implementation.
>
> I don't know for certain that every possible nesting "does the right
> thing".  I do know that @var{} is used inside many other constructs.
> By contrast, @dfn{} would not be nested inside or around other
> contructs very much.  @key can be nested inside @kbd, and it behaves
> a little differently when nested.

Thanks! I replied in a different thread, that includes Org ML in CC.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Org mode and Emacs (was: Convert README.org to plain text README while installing package)
  2022-06-15 23:15                                                     ` Org mode and Emacs (was: Convert README.org to plain text README while installing package) Richard Stallman
@ 2022-06-17  6:52                                                       ` Ihor Radchenko
  0 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  6:52 UTC (permalink / raw)
  To: rms; +Cc: theophilusx, eliz, monnier, acm, emacs-devel

Richard Stallman <rms@gnu.org> writes:

>   > However, we have special blocks for this purpose. They are extendable,
>   > as I descibed above. Though some improvements in Org core might be
>   > needed if we have to use this extensibility more frequently.
>
> Maybe that is sufficient.  It would be ok to use, for manuals, a
> submode which extend the usual Org facilities.

Submode is what I am inclined to. All these @var and @dfn constructs
only make sense for software documentation. Org is aiming much more
general writeup.

> But it would be good for that submode to define shorter command names
> for these.

Could you please elaborate? Which parts of the Org syntax you find too
long?

Best,
Ihor




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Larger GC thresholds for non-interactive Emacs (was: Org mode and Emacs)
  2022-06-16 16:59                                                                       ` Org mode and Emacs Stefan Monnier
@ 2022-06-17  7:18                                                                         ` Ihor Radchenko
  2022-06-17 13:49                                                                           ` Larger GC thresholds for non-interactive Emacs Stefan Monnier
  2022-06-21 13:04                                                                           ` Larger GC thresholds for non-interactive Emacs Lars Ingebrigtsen
  0 siblings, 2 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-17  7:18 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Mattias Engdegård, Eli Zaretskii, Tim Cross, rms,
	Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>>
>> I am wondering if there is a "universal" value suitable for one-off Emacs
>> batch invocations.
>
> I doubt that's the case, but of course we should try and use values that
> work "overall best" on average.  Maybe we should revisit the current
> knobs and their default values.
>
> Currently the two important knobs we have are:
>
>     gc-cons-threshold
>     gc-cons-percentage
>
> Maybe it's time for some benchmarks with various values for these knobs
> to see if the values we're using now are "good enough" or might need to
> be changed.

Note that I did not initially ask to change the GC defaults for normal
Emacs session.

What I am trying to suggest is:
1. Have some conservative GC settings in interactive Emacs session (as
   we do now; I do _not_ suggest changing it)
2. Increase GC thresholds in non-interactive --batch sessions, which
   tend to not run for a long time.

I am a bit afraid to touch the topic of changing the interactive GC
defaults. People tend to have contradicting opinions on this.

I'd like to ask one thing from the very beginning here: Please spin a
separate thread if you have objections on actual GC defaults applicable
to normal Emacs sessions.

> We could also try and be bit more discerning.  E.g. currently when the
> program is in a phase where it builds a lot of new data-structures, we
> run the GC everytime some amount of new memory has been allocated, but
> presumably almost none of those objects have died yet (we're still
> building the overall datastructure) so the GC will be a pure waste of
> time.  OTOH in other phases the code allocates objects which are used
> only temporarily, in which case it's beneficial to run the GC somewhat
> frequently to avoid growing our heap unnecessarily (and suffering
> fragmentation as a result).

AFAIK, there is no way to determine such stage automatically. It's
usually up to the program. For example, org-element-parse-buffer
(function that creates buffer AST in Org) does the following:

(let (... (gc-cons-threshold #x40000000))
      ...
      (org-element--parse-elements ...))

A possible useful thing Emacs could help with would be a macro dedicated
to let-binds like the above. Something like:

(with-reduced-gc
 ;; Do staff)

 with-reduced-gc could take care about determining the inner specifics
 on what alternative gc-cons-threshold value should be used (possibly
 depending on the system memory information).

> It's hard to know beforehand whether a GC will be useful or not, tho.
> But maybe we can find good heuristics.  E.g. have something like
> a `gc-cons-percentage` which depends on how much garbage we collected in
> the last GC: if a GC doesn't collect any garbage (or very little of it),
> it's a sign that we're in a phase where running the GC is not very
> useful so we should wait a bit more until the next GC, whereas if the GC
> collected a fair bit of garbage, it's a sign that we're in a phase where
> running the GC is beneficial and we can run it a bit more often.

FYI, I have played with this approach making use of
https://gitlab.com/koral/gcmh

In practice, relying on recent GC history is often misleading.

GC is normally not very frequent - tens of seconds to minutes. And the
practical Emacs memory usage patterns are often irregular: long periods
of "idleness"/low intensity usage vs. short bursts of high memory
allocations when some "heavy" commands are called.

The adjustment you propose will suffer from the following:
1. There is long "idle" period and GC gets less frequent
2. User runs "heavy" command, GC frequency not yet adjusted
3. Command allocates a lots of memory - GC is finally triggered and has
   to go through a lot of garbage; It often happens _after_ the command
   finishes.
4. GC re-adjusts the frequency, but Emacs is again "idle" with no
   "heavy" commands being used.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17  7:18                                                                         ` Larger GC thresholds for non-interactive Emacs (was: Org mode and Emacs) Ihor Radchenko
@ 2022-06-17 13:49                                                                           ` Stefan Monnier
  2022-06-17 15:11                                                                             ` Lars Ingebrigtsen
                                                                                               ` (2 more replies)
  2022-06-21 13:04                                                                           ` Larger GC thresholds for non-interactive Emacs Lars Ingebrigtsen
  1 sibling, 3 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-17 13:49 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Mattias Engdegård, Eli Zaretskii, Tim Cross, rms,
	Alan Mackenzie, emacs-devel

>> Maybe it's time for some benchmarks with various values for these knobs
>> to see if the values we're using now are "good enough" or might need to
>> be changed.
>
> Note that I did not initially ask to change the GC defaults for normal
> Emacs session.

You're absolutely right, but we've had various reports of late that
indicate that we're probably spending too much time in the GC.
So I think we're due for some re-evaluation.

> What I am trying to suggest is:
> 1. Have some conservative GC settings in interactive Emacs session (as
>    we do now; I do _not_ suggest changing it)
> 2. Increase GC thresholds in non-interactive --batch sessions, which
>    tend to not run for a long time.

Agreed.  Then again, we occasionally also use long-running commands
interactively so it might be worthwhile to tune the settings for
interactive use as well.

> I am a bit afraid to touch the topic of changing the interactive GC
> defaults.  People tend to have contradicting opinions on this.

:-)

The issue is that it's easy to see "oh god, we're spending so much time
in the GC, let's disable it" but much harder to see the "long term"
impact of increasing the thresholds (which tend to increase the
fragmentation and the total heap size both of which tend to slow down
execution but the second additionally makes every GC cycle take longer,
again increasing the pressure from naive onlookers to disable the GC).

> I'd like to ask one thing from the very beginning here: Please spin a
> separate thread if you have objections on actual GC defaults applicable
> to normal Emacs sessions.

No, I think focusing on batch use is a good idea.  Also because it makes
it much easier to actually measure the impact.

> A possible useful thing Emacs could help with would be a macro dedicated
> to let-binds like the above. Something like:
>
> (with-reduced-gc
>  ;; Do staff)

Sounds about right, tho I think the name of the macro should be related
to what the code does rather than to what the author thinks the GC should
do about it.  Something like `this-code-does-not-generate-garbage`.

>  with-reduced-gc could take care about determining the inner specifics
>  on what alternative gc-cons-threshold value should be used (possibly
>  depending on the system memory information).

Sounds good.

>> It's hard to know beforehand whether a GC will be useful or not, tho.
>> But maybe we can find good heuristics.  E.g. have something like
>> a `gc-cons-percentage` which depends on how much garbage we collected in
>> the last GC: if a GC doesn't collect any garbage (or very little of it),
>> it's a sign that we're in a phase where running the GC is not very
>> useful so we should wait a bit more until the next GC, whereas if the GC
>> collected a fair bit of garbage, it's a sign that we're in a phase where
>> running the GC is beneficial and we can run it a bit more often.
>
> FYI, I have played with this approach making use of
> https://gitlab.com/koral/gcmh

But this one focuses on interactive use.  The kind of heuristic I'm
proposing above would only make sense within a single (long-running)
command, or in batch mode, I think.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 13:49                                                                           ` Larger GC thresholds for non-interactive Emacs Stefan Monnier
@ 2022-06-17 15:11                                                                             ` Lars Ingebrigtsen
  2022-06-17 15:30                                                                               ` Lars Ingebrigtsen
  2022-06-17 17:48                                                                               ` Stefan Monnier
  2022-06-18  5:28                                                                             ` Ihor Radchenko
  2022-07-01  2:34                                                                             ` Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs) Ihor Radchenko
  2 siblings, 2 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-17 15:11 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

I tried this very advanced patch:

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index d28ec0be16..1bd0793097 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -5366,6 +5366,7 @@ batch-byte-compile
   ;; behavior.
   (setq attempt-stack-overflow-recovery nil
         attempt-orderly-shutdown-on-fatal-signal nil)
+  (setq gc-cons-threshold (* 100 gc-cons-threshold))
   (let ((error nil))
     (while command-line-args-left
       (if (file-directory-p (expand-file-name (car command-line-args-left)))

A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀

Most of this time is spent doing other things than building lisp files,
though, so the impact on byte compilation is huge.

rm lisp/*/*.elc
time make -j32

goes from 29s to 14s.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 15:11                                                                             ` Lars Ingebrigtsen
@ 2022-06-17 15:30                                                                               ` Lars Ingebrigtsen
  2022-06-17 16:05                                                                                 ` Stefan Monnier
  2022-06-17 17:48                                                                               ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-17 15:30 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

Lars Ingebrigtsen <larsi@gnus.org> writes:

> A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀

I wonder whether we should just have a command line option for this --
i.e., --no-garbage-collection to be used in combination with --batch for
jobs that aren't long-running (i.e., will exit before growing huge).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 15:30                                                                               ` Lars Ingebrigtsen
@ 2022-06-17 16:05                                                                                 ` Stefan Monnier
  2022-06-17 16:11                                                                                   ` Lars Ingebrigtsen
  2022-06-18  5:35                                                                                   ` Ihor Radchenko
  0 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-17 16:05 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

>> A "make -j32 bootstrap" went from 1m25s to 1m6s.  😀
> I wonder whether we should just have a command line option for this --
> i.e., --no-garbage-collection to be used in combination with --batch for
> jobs that aren't long-running (i.e., will exit before growing huge).

"no-garbage-collection" would be either a bug (we'd crash or be killed[*]
on long-enough-running jobs) or a lie (we'd still perform GC sometimes).

Also I hope we can first try to simply provide better defaults before
requiring manual intervention from the user.  I haven't seen any
evidence yet that the current defaults are good for batch jobs.

BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
does not adapt to the heap size, so maybe we'd be better off
playing with `gc-cons-percentage`.


        Stefan


[*] Like the Org manual's build on elpa.gnu.org yesterday because its
gc-threshold was set to most-positive-fixnum and `elpa.gnu.org` only has
1GB of RAM.




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 16:05                                                                                 ` Stefan Monnier
@ 2022-06-17 16:11                                                                                   ` Lars Ingebrigtsen
  2022-06-18  5:35                                                                                   ` Ihor Radchenko
  1 sibling, 0 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-17 16:11 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> "no-garbage-collection" would be either a bug (we'd crash or be killed[*]
> on long-enough-running jobs) or a lie (we'd still perform GC sometimes).

It's not a bug to do what the user says.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 15:11                                                                             ` Lars Ingebrigtsen
  2022-06-17 15:30                                                                               ` Lars Ingebrigtsen
@ 2022-06-17 17:48                                                                               ` Stefan Monnier
  2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
  2022-06-18  5:58                                                                                 ` Eli Zaretskii
  1 sibling, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-17 17:48 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

> rm lisp/*/*.elc
> time make -j32

IIRC this test can give surprising results because it's strongly
affected by the time when the byte-compiler files get compiled which is
somewhat arbitrary.

> goes from 29s to 14s.

Wow!

Here are the results of a similar test where I tried to avoid the above
problem by also removing src/boostrap-emacs (and I also rm'd
lisp/loaddefs.el for good measure tho it was probably not a great idea),
and I used

    BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":

make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total
make -j24 FOO=0.1    1338.36s user 60.60s system 1252% cpu 1:51.65 total
make -j24 FOO=0.1    1339.53s user 60.08s system 1252% cpu 1:51.76 total
make -j24 FOO=0.2    1266.54s user 59.75s system 1264% cpu 1:44.92 total
make -j24 FOO=0.2    1269.17s user 60.18s system 1256% cpu 1:45.83 total
make -j24 FOO=0.2    1266.10s user 60.42s system 1258% cpu 1:45.43 total
make -j24 FOO=0.3    1200.06s user 60.24s system 1251% cpu 1:40.73 total
make -j24 FOO=0.3    1198.80s user 60.06s system 1254% cpu 1:40.39 total
make -j24 FOO=0.3    1202.49s user 59.60s system 1252% cpu 1:40.80 total
make -j24 FOO=0.4    1134.26s user 59.75s system 1233% cpu 1:36.81 total
make -j24 FOO=0.4    1133.11s user 60.07s system 1226% cpu 1:37.25 total
make -j24 FOO=0.4    1132.18s user 59.94s system 1226% cpu 1:37.18 total
make -j24 FOO=0.5    1069.59s user 59.70s system 1211% cpu 1:33.22 total
make -j24 FOO=0.5    1070.84s user 60.06s system 1211% cpu 1:33.34 total
make -j24 FOO=0.5    1070.93s user 59.86s system 1210% cpu 1:33.43 total
make -j24 FOO=1.0     845.92s user 61.24s system 1126% cpu 1:20.55 total
make -j24 FOO=1.0     847.92s user 61.82s system 1130% cpu 1:20.46 total
make -j24 FOO=1.0     846.59s user 61.46s system 1133% cpu 1:20.10 total
make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total
make -j24 FOO=2.0     678.04s user 69.19s system 1042% cpu 1:11.64 total
make -j24 FOO=2.0     677.66s user 68.24s system 1041% cpu 1:11.59 total
make -j24 FOO=3.0     677.50s user 69.38s system 1047% cpu 1:11.28 total
make -j24 FOO=3.0     677.42s user 68.43s system 1047% cpu 1:11.19 total
make -j24 FOO=3.0     677.98s user 69.10s system 1044% cpu 1:11.50 total
make -j24 FOO=4.0     677.43s user 68.58s system 1051% cpu 1:10.96 total
make -j24 FOO=4.0     677.09s user 68.49s system 1056% cpu 1:10.57 total
make -j24 FOO=4.0     677.60s user 69.23s system 1049% cpu 1:11.17 total
make -j24 FOO=5.0     677.22s user 68.90s system 1043% cpu 1:11.52 total
make -j24 FOO=5.0     677.47s user 68.91s system 1044% cpu 1:11.49 total
make -j24 FOO=5.0     677.13s user 67.98s system 1049% cpu 1:10.98 total
make -j24 FOO=7.0     678.21s user 68.57s system 1044% cpu 1:11.52 total
make -j24 FOO=7.0     677.48s user 68.50s system 1044% cpu 1:11.42 total
make -j24 FOO=7.0     677.95s user 68.25s system 1050% cpu 1:11.00 total
make -j24 FOO=10.0    677.30s user 69.52s system 1045% cpu 1:11.45 total
make -j24 FOO=10.0    678.18s user 68.55s system 1042% cpu 1:11.62 total
make -j24 FOO=10.0    678.97s user 68.45s system 1043% cpu 1:11.64 total
make -j24 FOO=20.0    677.99s user 68.42s system 1041% cpu 1:11.67 total
make -j24 FOO=20.0    677.69s user 69.31s system 1046% cpu 1:11.36 total
make -j24 FOO=20.0    677.89s user 68.70s system 1045% cpu 1:11.39 total

Clearly for this test, the default (0.1) results in too low a threshold
for best performance and thus too much time spent in the GC.  We reach
a plateau around percentage=2.0.

This suggests that for batch jobs maybe we should bump up
`gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 17:48                                                                               ` Stefan Monnier
@ 2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
  2022-06-17 18:39                                                                                   ` Alan Mackenzie
                                                                                                     ` (2 more replies)
  2022-06-18  5:58                                                                                 ` Eli Zaretskii
  1 sibling, 3 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-17 18:20 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Here are the results of a similar test where I tried to avoid the above
> problem by also removing src/boostrap-emacs (and I also rm'd
> lisp/loaddefs.el for good measure tho it was probably not a great idea),
> and I used
>
>     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":
>
> make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total

[...]

> make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total

Nice!

> This suggests that for batch jobs maybe we should bump up
> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

Yup.  But do you mean in general?  I.e., -batch would set that variable
to 2.0?  Would there be any likely major repercussions -- i.e., jobs
that used to run fine would run out of memory?

As a quick test, I added a (setq gc-cons-percentage 2.0) to startup.el
here, and a

git clean -dfx
time make -j32

went from 1m29s to 1m4s.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
@ 2022-06-17 18:39                                                                                   ` Alan Mackenzie
  2022-06-17 22:21                                                                                     ` Stefan Monnier
  2022-06-17 22:53                                                                                   ` Stefan Monnier
  2022-06-18  2:32                                                                                   ` Stefan Monnier
  2 siblings, 1 reply; 376+ messages in thread
From: Alan Mackenzie @ 2022-06-17 18:39 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Stefan Monnier, Ihor Radchenko, Mattias Engdegård,
	Eli Zaretskii, Tim Cross, rms, emacs-devel

Hello, Lars.

On Fri, Jun 17, 2022 at 20:20:48 +0200, Lars Ingebrigtsen wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Here are the results of a similar test where I tried to avoid the above
> > problem by also removing src/boostrap-emacs (and I also rm'd
> > lisp/loaddefs.el for good measure tho it was probably not a great idea),
> > and I used

> >     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":

> > make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total

> [...]

> > make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total

> Nice!

> > This suggests that for batch jobs maybe we should bump up
> > `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?  Would there be any likely major repercussions -- i.e., jobs
> that used to run fine would run out of memory?

> As a quick test, I added a (setq gc-cons-percentage 2.0) to startup.el
> here, and a

> git clean -dfx
> time make -j32

> went from 1m29s to 1m4s.

Just as another independent test (as if any were needed), I put

    BYTE_COMPILE_EXTRA_FLAGS = --eval '(setq gc-cons-threshold 50000000)'

into lisp/Makefile.in, and my time for $ make -j17 bootstrap dropped
from 5m52s to 3m35s, a drop of 39%.  Watching the .elc's being built was
noticeably faster, even to the naked eye.  This was a build with native
compilation enabled.

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 18:39                                                                                   ` Alan Mackenzie
@ 2022-06-17 22:21                                                                                     ` Stefan Monnier
  2022-06-17 22:31                                                                                       ` Lars Ingebrigtsen
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-17 22:21 UTC (permalink / raw)
  To: Alan Mackenzie
  Cc: Lars Ingebrigtsen, Ihor Radchenko, Mattias Engdegård,
	Eli Zaretskii, Tim Cross, rms, emacs-devel

> Just as another independent test (as if any were needed), I put

Thanks anyway, but indeed, it would be good to see tests that do other
things than compilation.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 22:21                                                                                     ` Stefan Monnier
@ 2022-06-17 22:31                                                                                       ` Lars Ingebrigtsen
  0 siblings, 0 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-17 22:31 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Alan Mackenzie, Ihor Radchenko, Mattias Engdegård,
	Eli Zaretskii, Tim Cross, rms, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Thanks anyway, but indeed, it would be good to see tests that do other
> things than compilation.

Perhaps

rm lisp/leim/ja-dic/ja-dic.el*; time make

It goes from around 9 seconds to around 6 seconds (with a
cons-percentage of 4.0).  "make -j32 check" goes from 18s to 15s (with a
value of 2.0).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
  2022-06-17 18:39                                                                                   ` Alan Mackenzie
@ 2022-06-17 22:53                                                                                   ` Stefan Monnier
  2022-06-18  6:11                                                                                     ` Eli Zaretskii
  2022-06-18 12:45                                                                                     ` Lars Ingebrigtsen
  2022-06-18  2:32                                                                                   ` Stefan Monnier
  2 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-17 22:53 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

>> This suggests that for batch jobs maybe we should bump up
>> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.
> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?

Yup.

> Would there be any likely major repercussions

That's the question we should investigate as well.

> -- i.e., jobs that used to run fine would run out of memory?

I added the patch below to try and see a bit more what's going on:

    diff --git a/src/alloc.c b/src/alloc.c
    index 55e18ecd77e..d954e928eed 100644
    --- a/src/alloc.c
    +++ b/src/alloc.c
    @@ -6245,6 +6245,11 @@ garbage_collect (void)
       consing_until_gc = gc_threshold
         = consing_threshold (gc_cons_threshold, Vgc_cons_percentage, 0);

    +  fprintf (stderr, "GC-%d p=%.1f total=%.1fM thresold=%.1fM\n",
    +          getpid (), XFLOAT_DATA (Vgc_cons_percentage),
    +          total_bytes_of_live_objects () / 1048576.0,
    +          consing_until_gc / 1048576.0);
    +
       /* Unblock *after* re-setting `consing_until_gc` in case `unblock_input`
          signals an error (see bug#43389).  */
       unblock_input ();

and then I ran

    rm **/*.elc src/bootstrap-emacs lisp/loaddefs.el
    make -j24 BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage 2.0)'

The first thing I see is that with the number of GCs where p=0.1
outweighs those with p=2.0 by a factor 10, so clearly the number of GCs
is significantly reduced.  I also looked for the number of GCs performed
in a given process:

    % grep 'p=2.0' build-log | sed 's/ .*//' | sort | uniq -c
      1 GC-15817
      1 GC-15837
      1 GC-15841
      1 GC-15842
      1 GC-15985
      1 GC-15990
      1 GC-15991
      1 GC-15993
      1 GC-15995
      1 GC-15998
     18 GC-16009
     18 GC-16021
     18 GC-16023
     18 GC-16026
     18 GC-16028
     18 GC-16042
     18 GC-16045
     18 GC-16048
     18 GC-16056
     18 GC-16058
      1 GC-16062
      1 GC-16064
     18 GC-16067
     17 GC-16079

This says that only 24 processes performed a GC with p=2.0, so the
vast majority of the byte-compilation processes finished without
performing any GC at all (well, maybe not quite: see below).

It's also odd how most performed 0 GC, then 12 performed 1 GC, then
1 performed 17 GCs and 11 performed exactly 18 GCs.  What's so magical
about 18?

Then I looked at those that performed 18 GCs and they all look quite similar:

    % grep GC-16026 ./+make-2.0.log
    GC-16026 p=0.1 total=0.5M thresold=0.8M
    GC-16026 p=2.0 total=3.9M thresold=7.8M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M
    GC-16026 p=2.0 total=4.1M thresold=9.5M

The first GC seems to happen before we set percentage to 2.0 (so
apparently all compilation processes performed at least one GC before we
set percentage to 2.0 and then the majority of them performed no further
GC before exiting).  [ So if we set percentage to 2.0 a bit earlier than
what happens with `--eval` we may gain yet a bit more time.  ]

9.5M is actually exactly 10000000 bytes so I suspect these come from
either `cedet/semantic.el` or `international/mule-cmds.el` which both
set gc-cons-threshold to (max gc-cons-threshold 10000000).


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
  2022-06-17 18:39                                                                                   ` Alan Mackenzie
  2022-06-17 22:53                                                                                   ` Stefan Monnier
@ 2022-06-18  2:32                                                                                   ` Stefan Monnier
  2022-06-18 12:49                                                                                     ` Lars Ingebrigtsen
                                                                                                       ` (2 more replies)
  2 siblings, 3 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18  2:32 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

> Yup.  But do you mean in general?  I.e., -batch would set that variable
> to 2.0?  Would there be any likely major repercussions -- i.e., jobs
> that used to run fine would run out of memory?

I've looked a bit further at some of the repercussions.  The most
obvious immediate one is that a program that needs 10MB of live data
will now use up 30MB of heap (10MB of live data at the last GC plus upto
20MB of data allocated since the last GC), so it will increase the
process sizes in a non-trivial way.

I'd lean towards 1.0 instead of 2.0 for that kind of reason.

I also took a look at related data.  E.g. comparing p=1.0 to the old
p-0.1 on the process that performs the highest number of GCs (188 cycles
with p=1.0 and 650 cycles with p=0.1) during an Emacs build.  Here are
the corresponding last few GC cycles:

    % grep GC-26227 ./+make-0.1.log | tail -n 10              
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.6M free=2.0M thresold=1.9M
    GC-26227 p=0.1 total=18.7M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.7M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.9M thresold=1.9M
    GC-26227 p=0.1 total=18.8M free=1.8M thresold=1.9M
    GC-26227 p=0.1 total=145.7M free=32.6M thresold=14.6M
    GC-26227 p=0.1 total=132.8M free=60.7M thresold=13.3M
    % grep GC-898 ./+make-1.0.log | tail -n 10              
    GC-898 p=1.0 total=18.5M free=7.5M thresold=18.5M
    GC-898 p=1.0 total=18.6M free=7.4M thresold=18.6M
    GC-898 p=1.0 total=18.6M free=7.4M thresold=18.6M
    GC-898 p=1.0 total=18.7M free=7.4M thresold=18.7M
    GC-898 p=1.0 total=18.7M free=7.3M thresold=18.7M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=18.8M free=7.3M thresold=18.8M
    GC-898 p=1.0 total=145.7M free=32.6M thresold=145.7M
    % 

Obviously this process ends up with a very large one-step allocation
which is arguably interesting in itself (I suspect there's some
GC-inhibition going on there), but the more interesting point
is that with p=0.1 we get an amount of free space after GC
(i.e. blocks we can't release, because of fragmentation) that's about as
large as the next threshold, i.e. about 10%, whereas with p=1.0 this
amount of unreleasable free space is significantly higher.  Most likely
this is not wasted space: a lot of it will be re-used for new
allocations, but still with 19MB of live data, we end up with 7MB of
space we can't release back.

I think p=1.0 (and the corresponding implication that we use up about
twice as much memory as the minimum we need) might be an acceptable
tradeoff (for batch use), but I don't think I'd be comfortable going
beyond that.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package)
  2022-06-17  6:40                                                           ` Eli Zaretskii
@ 2022-06-18  4:40                                                             ` Ihor Radchenko
  2022-06-18  7:10                                                               ` Eli Zaretskii
  2022-06-18  5:10                                                             ` Convert README.org to plain text README while installing package Po Lu
  2022-06-20  6:05                                                             ` Kévin Le Gouguec
  2 siblings, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-18  4:40 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> I do not think that Org will support major user changes in Org syntax
>> any time soon or in future. At least, there is no intention to guarantee
>> such support.
>
> Neither of the other markup modes is being proposed for viewing and
> editing documents that were not originally edited under those modes.
> By contrast, there's a fraction of Emacs contributors and developers
> who repeatedly suggest to use Org for documents that were not
> originally written in Org.  A notable example (not the only one) is
> recent discussions of turning on Org when visiting NEWS files.

Do I understand correctly that users would like to have some Org
features (like fontification) in text that partially follows Org markup,
but not fully?

If so, Org mode cannot be used there as a major mode. non-Org parts of
syntax will cause undefined behavior. Which is to be expected as
major mode (every major mode) uses certain assumptions about the text in
buffer. Even text-mode will behave weirdly if binary data is mixed with
plain text. Org major mode expects text in buffers to be using Org
markup.

On the other hand, some parts of Org functionality are available in a
form of minor modes or individual commands. As usual, minor modes are
less demanding on the text in buffer.

AFAIK, we have the following functionality exposed to non-Org buffers:
- org-open-at-point-global/org-insert-link-global commands
- orgtbl-mode to edit tables, which use Org table syntax outside Org mode
- (ELPA) orgalist mode to edit lists written using Org list syntax outside Org mode
- (MELPA) org-msg mode to edit and send messages using Org markup (it
  exports to html mime parts)
- outline minor mode share a lot of functionality with Org without
  relying on Org markup. AFAIK, some Org features have been ported back
  to outline mode as well.

This dedicated functionality is designed to be used outside Org markup
buffers.

People who request Org mode to be used in non-Org buffers probably have
a specific subset of useful Org features in mind. I'd ask them which
features they want to and then consider exposing them out from Org mode
into minor mode (if not yet covered by the above).

> If you think these ideas are problematic from the POV of Org
> developers, please voice this opinion whenever such proposals are
> brought up.  Those proposals, and in general the proposals to use Org
> widely in unrelated contexts, is what I had in mind all the time in
> this discussion.  Perhaps now you can better understand some of my
> comments and responses.

I do understand you comments now. However, I do not follow emacs-devel
closely. So, it would be more efficient to CC Org ML in such
discussions.

> For example, what is your opinion of using Org markup in email
> messages?  There are a lot of examples of that, both here and on the
> bug tracker.  People use Org markup and Org-style code blocks quite a
> lot, and reading that is always jarring to me.  For some reason,
> people assume that I read my email in Org mode or some derivative of
> Org.

I also noticed that. AFAIU it is sometimes the text part sent by org-msg
and sometimes people just using Org syntax, because why not.

I personally do not see a problem with using Org markup to indicate code
blocks. At least, it is not much different from markdown some other
people are using in emails. Even without Org mode, such markup is
perfectly readable. AFAIK, there is no convention on indicating
"special" parts of plain text email (e.g. code snippets) in mailing
lists.

The convention for indicating "markup" in emails is associating the
correct mime-type to the message part. If it is html, the mail agent
should render html. If it is Org, Org can be rendered. If it is plain
text, the behavior is undefined in principle - people sending plain text
should make the text readable as plain text in text-mode (Org can be
made readable usually, but same can be said for e.g. Markdown).

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-17  6:40                                                           ` Eli Zaretskii
  2022-06-18  4:40                                                             ` Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package) Ihor Radchenko
@ 2022-06-18  5:10                                                             ` Po Lu
  2022-06-18 11:28                                                               ` Lars Ingebrigtsen
  2022-06-20  6:05                                                             ` Kévin Le Gouguec
  2 siblings, 1 reply; 376+ messages in thread
From: Po Lu @ 2022-06-18  5:10 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: Ihor Radchenko, theophilusx, acm, emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

> For example, what is your opinion of using Org markup in email
> messages?  There are a lot of examples of that, both here and on the
> bug tracker.  People use Org markup and Org-style code blocks quite a
> lot, and reading that is always jarring to me.  For some reason,
> people assume that I read my email in Org mode or some derivative of
> Org.

+1.  I hope people will stop doing that.  It is the reason I can't use
my work email to talk to those people, since the filter thinks the Org
syntax is a virus.

It's also visually jarring.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 13:49                                                                           ` Larger GC thresholds for non-interactive Emacs Stefan Monnier
  2022-06-17 15:11                                                                             ` Lars Ingebrigtsen
@ 2022-06-18  5:28                                                                             ` Ihor Radchenko
  2022-07-01  2:34                                                                             ` Lisp-level macro to avoid excessive GC in memory-allocating code (was: Larger GC thresholds for non-interactive Emacs) Ihor Radchenko
  2 siblings, 0 replies; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-18  5:28 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Mattias Engdegård, Eli Zaretskii, Tim Cross, rms,
	Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> FYI, I have played with this approach making use of
>> https://gitlab.com/koral/gcmh
>
> But this one focuses on interactive use.  The kind of heuristic I'm
> proposing above would only make sense within a single (long-running)
> command, or in batch mode, I think.

Got it.
But are there actually long-running batch mode commands over there?
The byte-compilation benchmarks in the other replies revealed that the
total number of GCs is not very high over the course of single batch
invocation.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 16:05                                                                                 ` Stefan Monnier
  2022-06-17 16:11                                                                                   ` Lars Ingebrigtsen
@ 2022-06-18  5:35                                                                                   ` Ihor Radchenko
  2022-06-18 13:16                                                                                     ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Ihor Radchenko @ 2022-06-18  5:35 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Lars Ingebrigtsen, Mattias Engdegård, Eli Zaretskii,
	Tim Cross, rms, Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
> does not adapt to the heap size, so maybe we'd be better off
> playing with `gc-cons-percentage`.

I am not sure if `gc-cons-percentage' is reliable.
When we have a small heap (at the beginning of the batch command run),
the GC will be triggered frequently. Later, after initial memory
allocations for data structures is complete, the same
`gc-cons-percentage' will make GC fire less frequently and will have
nothing to do with actually allocated/deallocated memory objects in the
subsequent parts of the batch process.

Consider the following example:

(defun init () (allocate-huge-hash-map-object))
(defun run () (dolist (...) (let (this and that) (query the huge hash-map))))
(init)
(run)

The (init) part will trigger frequent GCs yielding no of very small
amount of garbage being collected.

The (run) part may use more temporary allocations. Yet, GC will be
triggered much less frequently simply because the atomic (let ...)
allocations are small compared to the initially allocated hash-map
object.

I believe that gc-cons-threshold will still be useful as _an upper
bound_ to trigger GC. gc-cons-percentage is probably more efficient when
it triggers below gc-cons-threshold.

Best,
Ihor



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 17:48                                                                               ` Stefan Monnier
  2022-06-17 18:20                                                                                 ` Lars Ingebrigtsen
@ 2022-06-18  5:58                                                                                 ` Eli Zaretskii
  2022-06-18 13:46                                                                                   ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-18  5:58 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: larsi, yantar92, mattiase, theophilusx, rms, acm, emacs-devel

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>  emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 17 Jun 2022 13:48:46 -0400
> 
>     BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq gc-cons-percentage $FOO)'":
> 
> make -j24 FOO=0.1    1338.52s user 60.95s system 1256% cpu 1:51.42 total
> make -j24 FOO=0.1    1338.36s user 60.60s system 1252% cpu 1:51.65 total
> make -j24 FOO=0.1    1339.53s user 60.08s system 1252% cpu 1:51.76 total
> make -j24 FOO=0.2    1266.54s user 59.75s system 1264% cpu 1:44.92 total
> make -j24 FOO=0.2    1269.17s user 60.18s system 1256% cpu 1:45.83 total
> make -j24 FOO=0.2    1266.10s user 60.42s system 1258% cpu 1:45.43 total
> make -j24 FOO=0.3    1200.06s user 60.24s system 1251% cpu 1:40.73 total
> make -j24 FOO=0.3    1198.80s user 60.06s system 1254% cpu 1:40.39 total
> make -j24 FOO=0.3    1202.49s user 59.60s system 1252% cpu 1:40.80 total
> make -j24 FOO=0.4    1134.26s user 59.75s system 1233% cpu 1:36.81 total
> make -j24 FOO=0.4    1133.11s user 60.07s system 1226% cpu 1:37.25 total
> make -j24 FOO=0.4    1132.18s user 59.94s system 1226% cpu 1:37.18 total
> make -j24 FOO=0.5    1069.59s user 59.70s system 1211% cpu 1:33.22 total
> make -j24 FOO=0.5    1070.84s user 60.06s system 1211% cpu 1:33.34 total
> make -j24 FOO=0.5    1070.93s user 59.86s system 1210% cpu 1:33.43 total
> make -j24 FOO=1.0     845.92s user 61.24s system 1126% cpu 1:20.55 total
> make -j24 FOO=1.0     847.92s user 61.82s system 1130% cpu 1:20.46 total
> make -j24 FOO=1.0     846.59s user 61.46s system 1133% cpu 1:20.10 total
> make -j24 FOO=2.0     678.14s user 68.64s system 1044% cpu 1:11.52 total
> make -j24 FOO=2.0     678.04s user 69.19s system 1042% cpu 1:11.64 total
> make -j24 FOO=2.0     677.66s user 68.24s system 1041% cpu 1:11.59 total
> make -j24 FOO=3.0     677.50s user 69.38s system 1047% cpu 1:11.28 total
> make -j24 FOO=3.0     677.42s user 68.43s system 1047% cpu 1:11.19 total
> make -j24 FOO=3.0     677.98s user 69.10s system 1044% cpu 1:11.50 total
> make -j24 FOO=4.0     677.43s user 68.58s system 1051% cpu 1:10.96 total
> make -j24 FOO=4.0     677.09s user 68.49s system 1056% cpu 1:10.57 total
> make -j24 FOO=4.0     677.60s user 69.23s system 1049% cpu 1:11.17 total
> make -j24 FOO=5.0     677.22s user 68.90s system 1043% cpu 1:11.52 total
> make -j24 FOO=5.0     677.47s user 68.91s system 1044% cpu 1:11.49 total
> make -j24 FOO=5.0     677.13s user 67.98s system 1049% cpu 1:10.98 total
> make -j24 FOO=7.0     678.21s user 68.57s system 1044% cpu 1:11.52 total
> make -j24 FOO=7.0     677.48s user 68.50s system 1044% cpu 1:11.42 total
> make -j24 FOO=7.0     677.95s user 68.25s system 1050% cpu 1:11.00 total
> make -j24 FOO=10.0    677.30s user 69.52s system 1045% cpu 1:11.45 total
> make -j24 FOO=10.0    678.18s user 68.55s system 1042% cpu 1:11.62 total
> make -j24 FOO=10.0    678.97s user 68.45s system 1043% cpu 1:11.64 total
> make -j24 FOO=20.0    677.99s user 68.42s system 1041% cpu 1:11.67 total
> make -j24 FOO=20.0    677.69s user 69.31s system 1046% cpu 1:11.36 total
> make -j24 FOO=20.0    677.89s user 68.70s system 1045% cpu 1:11.39 total
> 
> Clearly for this test, the default (0.1) results in too low a threshold
> for best performance and thus too much time spent in the GC.  We reach
> a plateau around percentage=2.0.
> 
> This suggests that for batch jobs maybe we should bump up
> `gc-cons-percentage` from 0.1 to something like 1.0 or 2.0.

It is important to track the memory footprint of the Emacs process in
each run (like, show the maximum VM of all the compilations, for
example), because that's the other side of the tradeoff.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 22:53                                                                                   ` Stefan Monnier
@ 2022-06-18  6:11                                                                                     ` Eli Zaretskii
  2022-06-18 12:45                                                                                     ` Lars Ingebrigtsen
  1 sibling, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-18  6:11 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: larsi, yantar92, mattiase, theophilusx, rms, acm, emacs-devel

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Fri, 17 Jun 2022 18:53:17 -0400
> 
> It's also odd how most performed 0 GC, then 12 performed 1 GC, then
> 1 performed 17 GCs and 11 performed exactly 18 GCs.  What's so magical
> about 18?

Divine intervention?  18 means "alive" (חי) in gematria.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package)
  2022-06-18  4:40                                                             ` Limitations on using Org mode in buffers mixing Org markup with non-Org markup (was: Convert README.org to plain text README while installing package) Ihor Radchenko
@ 2022-06-18  7:10                                                               ` Eli Zaretskii
  2022-06-19  5:15                                                                 ` Ihor Radchenko
  0 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-18  7:10 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: theophilusx, acm, emacs-devel

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: theophilusx@gmail.com,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sat, 18 Jun 2022 12:40:10 +0800
> 
> > Neither of the other markup modes is being proposed for viewing and
> > editing documents that were not originally edited under those modes.
> > By contrast, there's a fraction of Emacs contributors and developers
> > who repeatedly suggest to use Org for documents that were not
> > originally written in Org.  A notable example (not the only one) is
> > recent discussions of turning on Org when visiting NEWS files.
> 
> Do I understand correctly that users would like to have some Org
> features (like fontification) in text that partially follows Org markup,
> but not fully?

It's more than that.  People suggest to start using Org for files that
were not written under Org mode, and the reason is not just
fontifications, but also Org outline-related and other commands.  IIUC
what you mean by "Org markup", it basically isn't used at all in those
files (but would probably "seep in" given enough time, if we indeed
begin using Org there).

> If so, Org mode cannot be used there as a major mode. non-Org parts of
> syntax will cause undefined behavior. Which is to be expected as
> major mode (every major mode) uses certain assumptions about the text in
> buffer. Even text-mode will behave weirdly if binary data is mixed with
> plain text. Org major mode expects text in buffers to be using Org
> markup.

Yup.

> On the other hand, some parts of Org functionality are available in a
> form of minor modes or individual commands. As usual, minor modes are
> less demanding on the text in buffer.
> 
> AFAIK, we have the following functionality exposed to non-Org buffers:
> - org-open-at-point-global/org-insert-link-global commands
> - orgtbl-mode to edit tables, which use Org table syntax outside Org mode
> - (ELPA) orgalist mode to edit lists written using Org list syntax outside Org mode
> - (MELPA) org-msg mode to edit and send messages using Org markup (it
>   exports to html mime parts)
> - outline minor mode share a lot of functionality with Org without
>   relying on Org markup. AFAIK, some Org features have been ported back
>   to outline mode as well.

Indeed, files that currently use Outline mode are the ones where the
above proposals are voiced most frequently.

The other minor modes are not necessarily relevant; at least I don't
recall those parts of Org's functionality being used as the reasons
for using Org where it wasn't before.

> People who request Org mode to be used in non-Org buffers probably have
> a specific subset of useful Org features in mind. I'd ask them which
> features they want to and then consider exposing them out from Org mode
> into minor mode (if not yet covered by the above).

Thanks, will keep this in mind next time such an issue comes up.

> > If you think these ideas are problematic from the POV of Org
> > developers, please voice this opinion whenever such proposals are
> > brought up.  Those proposals, and in general the proposals to use Org
> > widely in unrelated contexts, is what I had in mind all the time in
> > this discussion.  Perhaps now you can better understand some of my
> > comments and responses.
> 
> I do understand you comments now. However, I do not follow emacs-devel
> closely. So, it would be more efficient to CC Org ML in such
> discussions.

Fair enough.

> The convention for indicating "markup" in emails is associating the
> correct mime-type to the message part. If it is html, the mail agent
> should render html. If it is Org, Org can be rendered. If it is plain
> text, the behavior is undefined in principle - people sending plain text
> should make the text readable as plain text in text-mode (Org can be
> made readable usually, but same can be said for e.g. Markdown).

The difference between HTML and Org is that all MUAs support the
former, whereas Org can only be supported by Emacs-based MUA (and is
not supported by all Emacs-based MUA: e.g., Rmail doesn't currently
support it), and then only if the mime-type has been set correctly.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-18  5:10                                                             ` Convert README.org to plain text README while installing package Po Lu
@ 2022-06-18 11:28                                                               ` Lars Ingebrigtsen
  2022-06-18 13:33                                                                 ` Stefan Monnier
  0 siblings, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-18 11:28 UTC (permalink / raw)
  To: Po Lu; +Cc: Eli Zaretskii, Ihor Radchenko, theophilusx, acm, emacs-devel

Po Lu <luangruo@yahoo.com> writes:

> +1.  I hope people will stop doing that.  It is the reason I can't use
> my work email to talk to those people, since the filter thinks the Org
> syntax is a virus.
>
> It's also visually jarring.

Most Emacs mail readers understand the syntax and format it in a
readable and pretty way, so I hope people won't stop doing that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-17 22:53                                                                                   ` Stefan Monnier
  2022-06-18  6:11                                                                                     ` Eli Zaretskii
@ 2022-06-18 12:45                                                                                     ` Lars Ingebrigtsen
  2022-06-18 13:26                                                                                       ` Stefan Monnier
  1 sibling, 1 reply; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-18 12:45 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The first GC seems to happen before we set percentage to 2.0 (so
> apparently all compilation processes performed at least one GC before we
> set percentage to 2.0 and then the majority of them performed no further
> GC before exiting).

Hm, interesting...

> [ So if we set percentage to 2.0 a bit earlier than what happens with
> `--eval` we may gain yet a bit more time.  ]

This should probably be early enough, I guess?

diff --git a/src/emacs.c b/src/emacs.c
index 43b9901e08..33ad9c6b8d 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1620,6 +1620,7 @@ main (int argc, char **argv)
   if (argmatch (argv, argc, "-batch", "--batch", 5, NULL, &skip_args)
       || only_version)
     {
+      Vgc_cons_percentage = make_float (2.0);
       noninteractive = 1;
       Vundo_outer_limit = Qnil;
     }


> 9.5M is actually exactly 10000000 bytes so I suspect these come from
> either `cedet/semantic.el` or `international/mule-cmds.el` which both
> set gc-cons-threshold to (max gc-cons-threshold 10000000).

Ah, I see.  Perhaps those should increase that even more if it still
leads to 18 GCs?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18  2:32                                                                                   ` Stefan Monnier
@ 2022-06-18 12:49                                                                                     ` Lars Ingebrigtsen
  2022-06-18 13:06                                                                                       ` Stefan Monnier
                                                                                                         ` (2 more replies)
  2022-06-19  7:25                                                                                     ` Ihor Radchenko
  2022-06-19  9:23                                                                                     ` Ihor Radchenko
  2 siblings, 3 replies; 376+ messages in thread
From: Lars Ingebrigtsen @ 2022-06-18 12:49 UTC (permalink / raw)
  To: Stefan Monnier
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Obviously this process ends up with a very large one-step allocation
> which is arguably interesting in itself (I suspect there's some
> GC-inhibition going on there), but the more interesting point
> is that with p=0.1 we get an amount of free space after GC
> (i.e. blocks we can't release, because of fragmentation) that's about as
> large as the next threshold, i.e. about 10%, whereas with p=1.0 this
> amount of unreleasable free space is significantly higher.

As far as I know, we never actually release back cons space to the OS
anyway -- even if we could.  (Because glibc's malloc implementation
doesn't do that unless we call that trim function, which we don't.)  So
I'm not sure that makes much difference -- especially in a batch process.

> I think p=1.0 (and the corresponding implication that we use up about
> twice as much memory as the minimum we need) might be an acceptable
> tradeoff (for batch use), but I don't think I'd be comfortable going
> beyond that.

Right.  We can still have the Emacs makefiles increase the threshold for
build purposes even if we settle on a lower general level for --batch.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18 12:49                                                                                     ` Lars Ingebrigtsen
@ 2022-06-18 13:06                                                                                       ` Stefan Monnier
  2022-06-19 11:03                                                                                         ` Lars Ingebrigtsen
  2022-06-18 13:20                                                                                       ` Eli Zaretskii
  2022-06-20 12:42                                                                                       ` Lynn Winebarger
  2 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18 13:06 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

>> Obviously this process ends up with a very large one-step allocation
>> which is arguably interesting in itself (I suspect there's some
>> GC-inhibition going on there), but the more interesting point
>> is that with p=0.1 we get an amount of free space after GC
>> (i.e. blocks we can't release, because of fragmentation) that's about as
>> large as the next threshold, i.e. about 10%, whereas with p=1.0 this
>> amount of unreleasable free space is significantly higher.
> As far as I know, we never actually release back cons space to the OS
> anyway -- even if we could.

But we do release to the malloc library, which means it can be reused
for other things (instead of being constrained to only contain cons
cells, for example).   Whether that library in turn releases it back to
the OS is out of our control (and apparently current glibc never does
so).

>> I think p=1.0 (and the corresponding implication that we use up about
>> twice as much memory as the minimum we need) might be an acceptable
>> tradeoff (for batch use), but I don't think I'd be comfortable going
>> beyond that.
>
> Right.  We can still have the Emacs makefiles increase the threshold for
> build purposes even if we settle on a lower general level for --batch.

We could, I guess, tho the performance difference between p=1.0 and
p=2.0 isn't very significant.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18  5:35                                                                                   ` Ihor Radchenko
@ 2022-06-18 13:16                                                                                     ` Stefan Monnier
  0 siblings, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18 13:16 UTC (permalink / raw)
  To: Ihor Radchenko
  Cc: Lars Ingebrigtsen, Mattias Engdegård, Eli Zaretskii,
	Tim Cross, rms, Alan Mackenzie, emacs-devel

Ihor Radchenko [2022-06-18 13:35:04] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> BTW, I seen everyone's focused on `gc-cons-threshold` but that variable
>> does not adapt to the heap size, so maybe we'd be better off
>> playing with `gc-cons-percentage`.
> I am not sure if `gc-cons-percentage' is reliable.

From where I stand it's the only one that's reliable.

It basically controls the percentage of *time* we spend in the GC: if we
assume that the time to perform a GC is proportional to the heap size,
and that the allocation rate (in MB/s) is more or less constant, then
`gc-cons-percentage' directly controls the percentage of time spent in
the GC.

Of course, those two assumptions don't hold exactly, but they're fairly
good approximations of reality.

Tweaking `gc-cons-threshold` on the other hand is very coarse and
brittle: if the value is significantly smaller than the heap, then we'll
spend too much time in the GC and if it's significantly larger then we
waste a lot of memory on garbage.  There simply can't be any "one good
value" for `gc-cons-threshold`.

> When we have a small heap (at the beginning of the batch command run),
> the GC will be triggered frequently.

Yes, but the GC cycles will finish quickly.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18 12:49                                                                                     ` Lars Ingebrigtsen
  2022-06-18 13:06                                                                                       ` Stefan Monnier
@ 2022-06-18 13:20                                                                                       ` Eli Zaretskii
  2022-06-19 11:02                                                                                         ` Lars Ingebrigtsen
  2022-06-20 12:42                                                                                       ` Lynn Winebarger
  2 siblings, 1 reply; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-18 13:20 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: monnier, yantar92, mattiase, theophilusx, rms, acm, emacs-devel

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Ihor Radchenko <yantar92@gmail.com>,  Mattias Engdegård
>  <mattiase@acm.org>,  Eli Zaretskii <eliz@gnu.org>,  Tim Cross
>  <theophilusx@gmail.com>,  rms@gnu.org,  Alan Mackenzie <acm@muc.de>,
>   emacs-devel <emacs-devel@gnu.org>
> Date: Sat, 18 Jun 2022 14:49:49 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > Obviously this process ends up with a very large one-step allocation
> > which is arguably interesting in itself (I suspect there's some
> > GC-inhibition going on there), but the more interesting point
> > is that with p=0.1 we get an amount of free space after GC
> > (i.e. blocks we can't release, because of fragmentation) that's about as
> > large as the next threshold, i.e. about 10%, whereas with p=1.0 this
> > amount of unreleasable free space is significantly higher.
> 
> As far as I know, we never actually release back cons space to the OS
> anyway -- even if we could.  (Because glibc's malloc implementation
> doesn't do that unless we call that trim function, which we don't.)

AFAIK, that's not really true.  We call 'free' and glibc does release
to the OS when it can.  And don't forget that Emacs calls 'malloc' a
lot not just for conses and other Lisp data.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18 12:45                                                                                     ` Lars Ingebrigtsen
@ 2022-06-18 13:26                                                                                       ` Stefan Monnier
  0 siblings, 0 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18 13:26 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Ihor Radchenko, Mattias Engdegård, Eli Zaretskii, Tim Cross,
	rms, Alan Mackenzie, emacs-devel

>> 9.5M is actually exactly 10000000 bytes so I suspect these come from
>> either `cedet/semantic.el` or `international/mule-cmds.el` which both
>> set gc-cons-threshold to (max gc-cons-threshold 10000000).
>
> Ah, I see.  Perhaps those should increase that even more if it still
> leads to 18 GCs?

As I just explained in another message, I think messing with
`gc-cons-threshold` is a bad idea because it depends on the heap size.
We see this here where we end up thinking that maybe we should update
this magical number `10000000` with some other magical number.

The values of `gc-cons-percentage` aren't nearly as magical and they
automatically adjust to changes in heap sizes.  We can see that the
proposed `gc-cons-percentage` of 2.0 would result in a heap size almost
equal to that magical `10000000`, so rather than increase this number we
should likely just remove those hacks.

As for reducing the number of those 18 GCs, the data we have doesn't
tell us that it would bring any benefit.  Instead the data says we already
use up more than 3 times as much memory as what's truly needed (13.6MB
vs 4.1MB), which suggests to me that `10000000` is already rather high.


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-18 11:28                                                               ` Lars Ingebrigtsen
@ 2022-06-18 13:33                                                                 ` Stefan Monnier
  2022-06-18 15:50                                                                   ` tomas
  2022-06-19 11:08                                                                   ` Lars Ingebrigtsen
  0 siblings, 2 replies; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18 13:33 UTC (permalink / raw)
  To: Lars Ingebrigtsen
  Cc: Po Lu, Eli Zaretskii, Ihor Radchenko, theophilusx, acm, emacs-devel

> Most Emacs mail readers understand the syntax and format it in a
> readable and pretty way, so I hope people won't stop doing that.

Really?  I don't mind the Org syntax in email, FWIW, but I can't
remember ever seeing a MUA that recognizes it and displays it in
a pretty way.  Does Gnus does that?  How/when?


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18  5:58                                                                                 ` Eli Zaretskii
@ 2022-06-18 13:46                                                                                   ` Stefan Monnier
  2022-06-18 14:13                                                                                     ` Eli Zaretskii
  0 siblings, 1 reply; 376+ messages in thread
From: Stefan Monnier @ 2022-06-18 13:46 UTC (permalink / raw)
  To: Eli Zaretskii
  Cc: larsi, yantar92, mattiase, theophilusx, rms, acm, emacs-devel

> It is important to track the memory footprint of the Emacs process in
> each run (like, show the maximum VM of all the compilations, for
> example), because that's the other side of the tradeoff.

I measured the amount of free data in a later measurement, which is not
quite the same but gives some of that info.

Do you happen to know how to get the current process's RSS or VM size
from C so I can try and add this data to my log?


        Stefan




^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Larger GC thresholds for non-interactive Emacs
  2022-06-18 13:46                                                                                   ` Stefan Monnier
@ 2022-06-18 14:13                                                                                     ` Eli Zaretskii
  0 siblings, 0 replies; 376+ messages in thread
From: Eli Zaretskii @ 2022-06-18 14:13 UTC (permalink / raw)
  To: emacs-devel, Stefan Monnier
  Cc: larsi, yantar92, mattiase, theophilusx, rms, acm

On June 18, 2022 4:46:14 PM GMT+03:00, Stefan Monnier <monnier@iro.umontreal.ca> wrote
> 
> Do you happen to know how to get the current process's RSS or VM size
> from C so I can try and add this data to my log?


We do that in process-attributes, so you could just use that.



^ permalink raw reply	[flat|nested] 376+ messages in thread

* Re: Convert README.org to plain text README while installing package
  2022-06-18 13:33                                                                 ` Stefan Monnier
@ 2022-06-18 15:50                                                                   ` tomas
  2022-06-18 16:00                                                                     ` Visuwesh
  2022-06-19 11:08                                                                   ` Lars Ingebrigtsen
  1 sibling, 1 reply; 376+ messages in thread
From: tomas @ 2022-06-18 15:50 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

On Sat, Jun 18, 2022 at 09:33:16AM -0400, Stefan Monnier wrote:
> > Most Emacs mail readers understand the syntax and format it in a
> > readable and pretty way, so I hope people won't stop doing that.
> 
> Really?  I don't mind the Org syntax in email, FWIW, but I can't
> remember ever seeing a MUA that recognizes it and displays it in
> a pretty way.  Does Gnus does that?  How/when?

Same with me: I actually use mutt and my reader/writer is vim (yes,
really: I do most of my stuff with Emacs, but mail is one of the
last surviving realms vor vim). I even catch myself formatting
things in an Orgish way with vim (no special plugin: I know there
are).

But I can totally understand that some people don't like it. What
to do?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[