Re: Org mode and Emacs

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think the way forward is to define a "basic org-mode".  This one could
> be used at many more places where there's currently an occasional desire
> to use Org that's resisted because of the above problems.

I think that people that use Org mode wouldn't be very interested in
that, and people that don't use Org can just continue editing those
files with `M-x find-file-literally', really.  Doing so is fine.

But I don't understand the discussion this thread warped out of.  When
we display a README from Package, we shouldn't be showing the raw text
of README.org or README.md or README.html, lightly fontified -- we
should be showing a rendering of it.  People that read the README aren't
supposed to edit it, after all.

Which would make the 40K key bindings that Org apparently has completely
irrelevant for non-Org users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs

[Stefan Monnier]

> I think that people that use Org mode wouldn't be very interested in
> that, and people that don't use Org can just continue editing those
> files with `M-x find-file-literally', really.  Doing so is fine.

I was thinking of files like etc/NEWS where having "a bit more than
outline-mode" would be welcome.

> But I don't understand the discussion this thread warped out of.  When
> we display a README from Package, we shouldn't be showing the raw text
> of README.org or README.md

Why not?  If it can be prettified on the fly by font-lock?

Separating "viewing" from "editing" is un-Emacsish, because it puts
a barrier between the "consumers" and the "producers" whereas Emacs
likes to try and get end-users exposed to the source as mush as possible
so it takes little effort for them to change role.


        Stefan

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Alan Mackenzie <acm@muc.de>,  Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Tue, 07 Jun 2022 14:48:31 -0400
> 
> > But I don't understand the discussion this thread warped out of.  When
> > we display a README from Package, we shouldn't be showing the raw text
> > of README.org or README.md
> 
> Why not?

Because it's butt-ugly!

Re: Org mode and Emacs

[Stefan Monnier]

>> > But I don't understand the discussion this thread warped out of.  When
>> > we display a README from Package, we shouldn't be showing the raw text
>> > of README.org or README.md
>> Why not?
> Because it's butt-ugly!

I don't think it has to be.


        Stefan

Re: Org mode and Emacs

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But I don't understand the discussion this thread warped out of.  When
>> we display a README from Package, we shouldn't be showing the raw text
>> of README.org or README.md
>
> Why not?  If it can be prettified on the fly by font-lock?
>
> Separating "viewing" from "editing" is un-Emacsish, because it puts
> a barrier between the "consumers" and the "producers" whereas Emacs
> likes to try and get end-users exposed to the source as mush as possible
> so it takes little effort for them to change role.

That's true, but on the other hand, we don't show texinfo files to
people for a good reason.  That's about as "plain text" as normal Org
files, or as some Markdown files get.  (Most Markdown files are kinda
pleasant, but there's plenty of scope for unreadability -- it drops down
into HTML for tables, for instance.)

Or take doc strings as an example -- it's always been a markup language,
too (\\ and \\[] and \\<>), but it's moving more clearly in that
direction -- because we want to make the documentation we present the
user pretty and easy to read.  And we have one-key commands to take us
to the source code, which we could have in the Package README rendering,
too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Org mode and Emacs

[Visuwesh]

[புதன் ஜூன் 08, 2022] Tim Cross wrote:

> [...]
> - Increasing org's use of built-in Emacs capabilities rather than using
>   org specific implementations. For example, adopting transient instead
>   of an org implemented module which replicates similar functionality. 

Where can I read more about this?  I see it being mentioned a few times
in the org-mode mailing list, and in the matrix room.  Is the plan to
completely remove the org-mks interface and replace it with transient
without having an option to use the former?  I find org-mks perfectly
fine to use and would be sad if it was replaced with transient.
Transient needs time to get used to, and the default settings is quite
un-Emacsy; I'm not too excited about configuring yet another package
that has a hard-to-understand manual TBH.

Re: Org mode and Emacs

[Tim Cross]

Visuwesh <visuweshm@gmail.com> writes:

> [புதன் ஜூன் 08, 2022] Tim Cross wrote:
>
>> [...]
>> - Increasing org's use of built-in Emacs capabilities rather than using
>>   org specific implementations. For example, adopting transient instead
>>   of an org implemented module which replicates similar functionality. 
>
> Where can I read more about this?  I see it being mentioned a few times
> in the org-mode mailing list, and in the matrix room.  Is the plan to
> completely remove the org-mks interface and replace it with transient
> without having an option to use the former?  I find org-mks perfectly
> fine to use and would be sad if it was replaced with transient.
> Transient needs time to get used to, and the default settings is quite
> un-Emacsy; I'm not too excited about configuring yet another package
> that has a hard-to-understand manual TBH.


The specifics of what is planned are still being worked out. Initially,
the likely initial candidate for change will be to the export menu.
Unless you have extensive low level customisation, it should not be a
change which has significant impact on users. In fact, maintaining
backwards compatibility and consistency for end users is important to
the org developers. It is also quite possible that after an initial
investigation, it may be decided transient is not a good option for org
mode or perhaps it will be a good option once additonal functionality or
enhancements are added. Right now, all that has been agreed is that it
would be worthwhile looking at it to see if it can be of benefit in
helping to reduce org maintenance overheads and/or increase org's
consistency with other emacs packages. 

What will determine what remains and what choices are available will
depend on what involvement people have in the development and what will
be maintainable. There are a number of areas in org mode where
functionality has been implemented that is 80+% equivalent to
functionality which exists or has been added to core emacs. Having this
duplication of effort is adding to the burden of maintenance, which is
already significant. In general, org maintainers keep an eye on what is
being added/expanded in Emacs core and when things are added, like
transient mode, they are assessed to see if adopting that functionality
would reduce the org maintenance load and improve org's consistency
withi the rest of Emacs. At this stage, transient has been added as
something to look at in the backlog. When this task makes if off the
backlog, the typical process would be for it to be discussed on the org
devel list, for initial implementations to be done either in its own
branch (if considered a significantly large enough change) or on the
development branch otherwise and people will be asked to try it out. 

So, if your interested in this area, the first thing would be to get on
the org devel mail list. Anyone who is particularly keen to see such
things added might also initiate discussions and development in their
own branch, which could then be added in as a PR. However, like Emacs,
any significant development work on org mode also requires FSF copyright
assignment. 

At this specific point in time, the main focus with org has been on
stability, performance improvements, especially for large org files with
many babel blocks and sections and improvements to the org syntax and provision of an
org parser,  

In particular, I think the work being done by Ihor on folding and
the org parser will have huge benefits. In addition to making the code
base easier to maintain and improving performance, it will help reduce
org's current dependency on complicated and difficult to maintain
regular expressions in the font-locking layer. This should improve
performance and reduce errors or unexpected results that sometimes occur
because of regexp errors etc and reduce time spent by maintainers in
tracking down such errors and maintenance of those complex regexps.   

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-08 21:22:07] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> AFAICT the problem seen from Emacs, is that Org is large (even for
>> a basic uses, which occasionally leads to high load times) and that it
>> doesn't follow all the usual Emacs conventions, such as
>> overriding/remapping standard key-bindings (the resulting behaviors may
>> sometimes be similar to the standard one, but even when that's the case
>> the redefinition can easily bite the user).
> I am not sure if I follow the argument. Major modes are allowed to
> change defaults. For example, special modes often change truncate-lines
> value. Org mode is also tweaking defaults (yes, many of them). I do not
> see any problem here in general.
> If you have some specific cases when Org mode alters Emacs defaults in a
> way that bites the user, please give concrete examples. Otherwise, your
> criticism is not very constructive.

[ Alan gave one or two concrete examples of things that bit him.  ]

This is not a criticism, just a description of how Org is perceived from
the side of "old-time Emacs users who aren't Org users".
The key in what you wrote above is the "yes, many of them", which means
that even tho those tweaks are minor they sum up to something that
old-time users will almost inevitably bump into.

It's not a problem for Org itself, and there are good reasons for each
one of those tweaks, I'm sure, but it does create opposition to using
Org "in core", such as in etc/NEWS.

> This is reasonable. RMS also asked for this years back
> https://orgmode.org/list/E1kIPh1-0001Lu-Rg@fencepost.gnu.org
> Since we cannot start Org from scratch, factoring out individual modules
> is taking a lot of time and having the hostile attitudes expressed in
> some of the emails in this thread is not exactly encouraging.

I know it's a lot of work and I'm glad you (plural) are taking it on,
and I'm not very happy about some of the things that have been said in
that thread, which is why I tried to start this thread.


        Stefan

Re: Org mode and Emacs

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> If you have some specific cases when Org mode alters Emacs defaults in a
>> way that bites the user, please give concrete examples. Otherwise, your
>> criticism is not very constructive.
>
> [ Alan gave one or two concrete examples of things that bit him.  ]

Alan gave two concrete examples:

>>> I put org.org into a buffer.  I saw lots of lines ending in ..., which is
>>> fair enough.  I try C-c C-s to show a subtree, and instead get a calendar
>>> in a new window.  Brilliant!  Did I mention I hate context dependent
>>> commands?  So I have to type M-x outline-show-subtree, which works.  I
>>> read the screenful of text, then type C-S-<down> to scroll it up.
>>> Instead of my standard scrolling command, I get an error message about
>>> "Not at a clock log".  Org mode has stolen the key sequences
>>> C-S-<up>/<down>, amongst many others.  So how am I meant to scroll text?

where he
1. Complains about major mode shadowing key binding reserved for major
   modes, according to D.2 Key Binding Conventions section of the Elisp
   manual:
>      • Sequences consisting of ‘C-c’ followed by a control character or a
>     digit are reserved for major modes.

2. C-S-<down>, which is a valid point. However, arrow commands is one of
   the few core concepts used in Org major mode.
   I am not sure if re-binding C-S-<down> is so much of a sin on Org
   mode side. Emacs manual 49.3 Customizing Key Bindings says:

>         Since most modes define their own key bindings, activating a mode
>      might override your custom key bindings.  A small number of keys are
>      reserved for user-defined bindings, and should not be used by modes, so
>      key bindings using those keys are safer in this regard.  The reserved
>      key sequences are those consisting of ‘C-c’ followed by a letter (either
>      upper or lower case), and function keys <F5> through <F9> without
>      modifiers (*note Modifier Keys::).

   As a compromise, Org may provide some kind of read-only mode just to
   navigate the document. The arrow bindings can be disabled then. They
   are for editing.

>>> I see many of these bindings as context dependent, with the name of the
>>> function named after the key sequence, not the functionality - e.g.
>>> org-shiftcontroldown.  That is not the way the rest of Emacs is
>>> constructed.  I hate context dependence (except when it is properly
>>> implemented, e.g. by major modes).

    Only a handful of bindings are context dependent. In particular,
    control/shift/meta-arrow keys, <TAB>, and C-c C-c. They are most
    frequently used when editing and navigating Org buffers.

    It would help if "properly implemented" were defined more precisely
    in the above statement.

> This is not a criticism, just a description of how Org is perceived from
> the side of "old-time Emacs users who aren't Org users".
> The key in what you wrote above is the "yes, many of them", which means
> that even tho those tweaks are minor they sum up to something that
> old-time users will almost inevitably bump into.
>
> It's not a problem for Org itself, and there are good reasons for each
> one of those tweaks, I'm sure, but it does create opposition to using
> Org "in core", such as in etc/NEWS.

I understand. However, such criticism is not helpful. Only concrete
examples can lead to actual Org improvements in this area.

Best,
Ihor

Re: Org mode and Emacs

[Po Lu]

Richard Stallman <rms@gnu.org> writes:

> If this succeeds, it would be an important advance for the GNU system.
> We would replace Texinfo with a much cleaner system, easier to use and
> more maintainable.  Not solely for Emacs, but for ALL our
> documentation!

Org mode is not easier to use than Texinfo... at least, I didn't figure
out how to use it.

IIRC it's also much slower to create similarly sized .info files from
Org source than it is with makeinfo, and it also requires an entire
manual to be contained in a single file.

Texinfo also comes with the added benefit of not requiring Emacs to edit
or to translate into other formats.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> IIRC it's also much slower to create similarly sized .info files from
> Org source than it is with makeinfo

Could you please quantify much slower?
Do you have concrete example of a .org file which can demonstrate the
problem?

Also, I think that I have to clarify about texinfo export from Org.
Org does not currently generate .info files directly. .org file is first
transformed into .texi and then uses makeinfo to translate the generated
.texi into .info.

If there is an interest in Org being used instead of texinfo, it should
be based on the extensibility of Org rather than just making Org
replicate what texinfo does. However, I am not sure if all parties in
this discussion have a clear idea about concrete advantages of Org
compared with texinfo. I am not faimiliar with texinfo. Others may not
be as faimilar with Org.

I believe that if we want to be serious about using Org for
documentation more widely, we should first identify which particular
aspects of Org are beneficial for documentation and which particular
aspects are available in other documentation generators and must be then
provided by Org (if not yet available).

> ... , and it also requires an entire
> manual to be contained in a single file.

This is not true. Org mode supports #+include: directives to incorporate
multiple smaller files into one larger file.

> Texinfo also comes with the added benefit of not requiring Emacs to edit
> or to translate into other formats.

You can edit Org files outside Emacs. Say, in vim.
The point about exporting to other formats is valid.

This last point also raises a question. Can Elisp interpreter and
libraries be factored out of Emacs to create a way to execute Elisp
programs without installing all the interactive parts of Emacs?

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>> or to translate into other formats.
>
> You can edit Org files outside Emacs. Say, in vim.
> The point about exporting to other formats is valid.
>
> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Isn't Tim Cross(?) working on something like that -- ie. a parser for
the Org language.  Once we have a solid parser, we can build a standard
(set of?) backend(s) for much of Orgmode.

-- 
David Masterson

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 383 bytes --]

Note that org's export to texinfo is quite good; as an example, I wrote
the "Emacspeak Turning Twenty" article in org-mode and later exported it
to texinfo among other formats. Org's export to TeX,LaTeX and PDF is
also very good; likely superior to its export to Texinfo.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@gmail.com> writes:

> Could you please quantify much slower?

Making the Org manual takes about 27 seconds on my system:

  GEN      org.texi
  GEN      ../../info/org.info

real	0m27.210s
user	0m25.567s
sys	0m0.783s

Which is longer than it takes for the entire (much more massive) Emacs
Lisp reference manual:

  GEN      ../../info/elisp.info

real	0m24.561s
user	0m23.660s
sys	0m0.515s

And generating the .info file from the Org manual texinfo source takes
only:

  GEN      ../../info/org.info

real	0m5.381s
user	0m5.109s
sys	0m0.141s

So it's clear that the org to texinfo export takes most of the time.

> I believe that if we want to be serious about using Org for
> documentation more widely, we should first identify which particular
> aspects of Org are beneficial for documentation and which particular
> aspects are available in other documentation generators and must be then
> provided by Org (if not yet available).

I don't see why we should be serious about using Org for our
documentation, when most people already know texinfo and are quite happy
with it.

> This is not true. Org mode supports #+include: directives to incorporate
> multiple smaller files into one larger file.

That's nice, but why isn't the Org manual itself structured that way?

> You can edit Org files outside Emacs. Say, in vim.

But does it work very well?

> This last point also raises a question. Can Elisp interpreter and
> libraries be factored out of Emacs to create a way to execute Elisp
> programs without installing all the interactive parts of Emacs?

Probably not without a lot of work.

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

Org already have a parser. Written in Elisp. Export is built on top of
the parser.

Best,
Ihor

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Texinfo also comes with the added benefit of not requiring Emacs to edit
>>> or to translate into other formats.
>>
>> You can edit Org files outside Emacs. Say, in vim.
>> The point about exporting to other formats is valid.
>>
>> This last point also raises a question. Can Elisp interpreter and
>> libraries be factored out of Emacs to create a way to execute Elisp
>> programs without installing all the interactive parts of Emacs?
>
> Isn't Tim Cross(?) working on something like that -- ie. a parser for
> the Org language.  Once we have a solid parser, we can build a standard
> (set of?) backend(s) for much of Orgmode.

No. Ihor is the one working on the parser. 

I think Ihor's question about isolating the Elisp interpreter is about
options to make org mode work outside of Emacs. There are frequent
questions to the org list about making org available to other
editors/environments. However, the big problem is that much of the
really powerful featurs of org mode are intrinsically tied to elisp
functionality - for exmaple all the babel and backend export support.
While it is possible to write a parser in any language which would
enable basic org markup/formatting, that does not solve the problem of
executable blocks, babel/noweb and interaction with back end exporters
etc. 

This would be one example of where something like an elisp LSP module
would be useful. However, the idea of an elisp LSP module was
discouraged by a couple of people on this list over concerns that such a
module would enable non-free platforms to take advantage of elisp - a
fear which I think is overstated and which I think one person referred
to as "Jumping at shadows" 

The overall conclusion was that such a module would be considerable
amount of work. Some suggested you could use emacsclient to create
something which could be used in such a manner. However, I suspect that
if you go to the effort of installing and configuring emacs, you would
just use emacs. 

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This last point also raises a question. Can Elisp interpreter and
>>> libraries be factored out of Emacs to create a way to execute Elisp
>>> programs without installing all the interactive parts of Emacs?
>>
>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>> the Org language.  Once we have a solid parser, we can build a standard
>> (set of?) backend(s) for much of Orgmode.
>
> Org already have a parser. Written in Elisp. Export is built on top of
> the parser.

But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
long run, it would be better to define a "parseable" language as the
standard basis for Org.  From that, people can develop (parts of) Org on
other platforms (Vim, Beorg, Orgzly) and test/prove that they are
compatible with version X of the language.  I think Organice was doing
this, but I haven't looked at it deeply.

Oh, but I see your point about "export". By backend, I was assuming a
true parser would generate a standard "internal" language which could be
fed into simpler backends to actually do the work.  The front-end parser
and back-ends could be translated/rebuilt as needed on new platorms
(iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
the idea.

-- 
David Masterson

Re: Org mode and Emacs

[chad]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On Sat, Jun 11, 2022 at 10:52 PM Po Lu <luangruo@yahoo.com> wrote:

> I don't see why we should be serious about using Org for our
> documentation, when most people already know texinfo and are quite happy
> with it.
>

I think a reasonable examination of the emacs-devel archives as well as the
common practices of most of the people publishing emacs lisp packages today
would lead to a very different conclusion. There are several threads about
maintenance concerns around makeinfo/texinfo, and many discussions about
replacing texinfo with, for example, HTML.There are periodic threads where
people claim that they won't try to add their project to GNU because the
burden of learning and using texinfo is too high, although those have died
down in volume since it became more practical to translate other formats to
texinfo.

None of this is to say that Org is the "right" format for Emacs
documentation. I just don't think it's correct to say "most people already
know texinfo and are quite happy with it".

~Chad

[-- Attachment #2: Type: text/html, Size: 1430 bytes --]

Re: Org mode and Emacs

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>>> This last point also raises a question. Can Elisp interpreter and
>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>> programs without installing all the interactive parts of Emacs?
>>>
>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>> the Org language.  Once we have a solid parser, we can build a standard
>>> (set of?) backend(s) for much of Orgmode.
>>
>> Org already have a parser. Written in Elisp. Export is built on top of
>> the parser.
>
> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
> long run, it would be better to define a "parseable" language as the
> standard basis for Org.  From that, people can develop (parts of) Org on
> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
> compatible with version X of the language.  I think Organice was doing
> this, but I haven't looked at it deeply.
>

This is all part of the aims and process. However, the first step is to
develop a robust elisp based parser. This helps to ensure the org syntax
is consistent and helps identifies ambiguities which need to be fixed as
well as provides a reference implementation which developers can use to
develop parsers in other languages.

> Oh, but I see your point about "export". By backend, I was assuming a
> true parser would generate a standard "internal" language which could be
> fed into simpler backends to actually do the work.  The front-end parser
> and back-ends could be translated/rebuilt as needed on new platorms
> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
> the idea.

Personally, I think it is unlikely we will ever see an org mode for
other platforms which is equivalent to Emacs. However, the work which is
being done to create a clean parser in elisp and use that as the basis
for an API to manipulate org data, generate font-locking and formatting
and provide a clean API for exporters to use will make it much easier
for people to develop external org support at varying levels. 

Those doing the main work, like Ihor, are very aware of the desire to
facilitate external implementations. It is one reason that a fairly
conservative attitude to change is adopted by the project and often, one
consideration is what imapct a change would have on external org
compatible projects. Sometimes, this is challenging as on one level, we
want ot advance and improve org mode, but on the other, we want it to be
as stable as possible to reduce adverse impact on external projects. So
while the main target is and will always be Emacs, an eye is kept on
efforts of other external projects and an effort is made to support them
when possible and within Emacs and FSF principals/guidelines. 

So far, the key contributors have been doing a very good job. 

Re: Org mode and Emacs

[Po Lu]

chad <yandros@gmail.com> writes:

> I think a reasonable examination of the emacs-devel archives as well
> as the common practices of most of the people publishing emacs lisp
> packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo,
> and many discussions about replacing texinfo with, for example,
> HTML.

Published Emacs Lisp packages in the wild typically come with no
documentation at all, aside from doc strings and a single README file.

Org mode happens to be popular for the latter, but I don't know why.
The learning curve is high and the results seem to be unjustified.  An
easier format such as HTML or Markdown can be used instead.

> There are periodic threads where people claim that they won't try to
> add their project to GNU because the burden of learning and using
> texinfo is too high, although those have died down in volume since it
> became more practical to translate other formats to texinfo.

Why would Org mode be any different?  Unlike HTML, it is completely
specific to Emacs, which actually makes it worse than Texinfo, because
Texinfo is at least widely used throughout the entire GNU project.

And anyway, I make the following offer: if someone doesn't want to
contribute documentation (or code) because he doesn't know Texinfo, he
can write the documentation in plain text, and I will translate it to
Texinfo for him.  IIRC the GDB developers made a similar offer, and for
the many years that it existed nobody ever took advantage of it, which
shows that Texinfo is not a serious barrier to contributors.

Further more, using Org mode for documentation will make Emacs lose many
people who are actually writing documentation, right now.  At least I
have no interest in ever learning Org mode, and there seem to be no
volunteers who will translate Texinfo (or plain text) to Org mode for me
if we begin to use it for documentation in the future.

Re: Org mode and Emacs

[Po Lu]

Tim Cross <theophilusx@gmail.com> writes:

>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.

> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to

It is not the goal of Emacs to support tyrant devices such as the iPhone
running proprietary operating systems such as iOS.

If people are going to refactor Org mode, I hope they will not do it
with support for iPhones in mind.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>
> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

I am not sure why you only picked up iPhones from the list suggested by
David. The aim is not supporting iPhones. The aim is supporting
other GNU and Free Software projects that are making use of Org, which
implies conservative approach to breaking changes, especially in syntax.

Best,
Ihor

Re: Org mode and Emacs

[Tim Cross]

Po Lu <luangruo@yahoo.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>>> long run, it would be better to define a "parseable" language as the
>>> standard basis for Org.  From that, people can develop (parts of) Org on
>>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>>> compatible with version X of the language.  I think Organice was doing
>>> this, but I haven't looked at it deeply.
>
>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>
> It is not the goal of Emacs to support tyrant devices such as the iPhone
> running proprietary operating systems such as iOS.
>

> If people are going to refactor Org mode, I hope they will not do it
> with support for iPhones in mind.

There has never been suggestion that what we are doing is to facilitate
development on non-free platforms. However, if you develop good clean
code and clear algorithms which others are able to use as a reference to
understand the syntax and semantics of a system, you cannot control how
they will use that information. In fact, it can be argued that making
such information readily available and accessible is a large part of the
freedoms being promoted by the FSF.

This is completely different from, for example, providing an interface
which can be used by a non-free platform. 

Besides, if org is as bad and difficult to learn as you have indicated
in other posts, nobody will bother implementing it on those platforms
you seem so threatened by anyway!

Re: Org mode and Emacs

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Ihor Radchenko <yantar92@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>>> This last point also raises a question. Can Elisp interpreter and
>>>>> libraries be factored out of Emacs to create a way to execute Elisp
>>>>> programs without installing all the interactive parts of Emacs?
>>>>
>>>> Isn't Tim Cross(?) working on something like that -- ie. a parser for
>>>> the Org language.  Once we have a solid parser, we can build a standard
>>>> (set of?) backend(s) for much of Orgmode.
>>>
>>> Org already have a parser. Written in Elisp. Export is built on top of
>>> the parser.
>>
>> But Elisp is not portable to a non-Emacs system (say, iPhone).  In the
>> long run, it would be better to define a "parseable" language as the
>> standard basis for Org.  From that, people can develop (parts of) Org on
>> other platforms (Vim, Beorg, Orgzly) and test/prove that they are
>> compatible with version X of the language.  I think Organice was doing
>> this, but I haven't looked at it deeply.
>>
>
> This is all part of the aims and process. However, the first step is to
> develop a robust elisp based parser. This helps to ensure the org syntax
> is consistent and helps identifies ambiguities which need to be fixed as
> well as provides a reference implementation which developers can use to
> develop parsers in other languages.

Semantic/Bovine ??

Agree

>> Oh, but I see your point about "export". By backend, I was assuming a
>> true parser would generate a standard "internal" language which could be
>> fed into simpler backends to actually do the work.  The front-end parser
>> and back-ends could be translated/rebuilt as needed on new platorms
>> (iOS, Android, MS-Windows, etc.).  More is needed than this, but that's
>> the idea.
>
> Personally, I think it is unlikely we will ever see an org mode for
> other platforms which is equivalent to Emacs. However, the work which is
> being done to create a clean parser in elisp and use that as the basis
> for an API to manipulate org data, generate font-locking and formatting
> and provide a clean API for exporters to use will make it much easier
> for people to develop external org support at varying levels.

Agree

> Those doing the main work, like Ihor, are very aware of the desire to
> facilitate external implementations. It is one reason that a fairly
> conservative attitude to change is adopted by the project and often, one
> consideration is what imapct a change would have on external org
> compatible projects. Sometimes, this is challenging as on one level, we
> want ot advance and improve org mode, but on the other, we want it to be
> as stable as possible to reduce adverse impact on external projects. So
> while the main target is and will always be Emacs, an eye is kept on
> efforts of other external projects and an effort is made to support them
> when possible and within Emacs and FSF principals/guidelines. 
>
> So far, the key contributors have been doing a very good job. 

That's good.

-- 
David Masterson

Re: Org mode and Emacs

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sat, 11 Jun 2022 23:54:39 -0400
> Cc: Ihor Radchenko <yantar92@gmail.com>, Richard Stallman <rms@gnu.org>,
>  Tim Cross <theophilusx@gmail.com>, 
>  Stefan Monnier <monnier@iro.umontreal.ca>, Alan Mackenzie <acm@muc.de>, 
>  EMACS development team <emacs-devel@gnu.org>
> 
>  I don't see why we should be serious about using Org for our
>  documentation, when most people already know texinfo and are quite happy
>  with it.
> 
> I think a reasonable examination of the emacs-devel archives as well as the common practices of most of
> the people publishing emacs lisp packages today would lead to a very different conclusion. There are
> several threads about maintenance concerns around makeinfo/texinfo, and many discussions about
> replacing texinfo with, for example, HTML.There are periodic threads where people claim that they won't try
> to add their project to GNU because the burden of learning and using texinfo is too high, although those have
> died down in volume since it became more practical to translate other formats to texinfo.

The real issue behind these claims is that developers seldom like
writing good documentation.  So learning to use a sophisticated markup
system such as Texinfo is overkill for them.  On top of that, if the
package is not an Emacs package, there's a problem that Texinfo
editing is not really well supported outside of Emacs, which is
another reason for complaining by people who don't use Emacs for
developing their package.

Good HTML is no easier to write than good Texinfo, but the difference
is that there are tools out there other than Emacs which can be used
to create HTML, especially if the manual is relatively simple (as many
Free Software manuals are).

Bottom line: the complaints are real, but I'm not sure they help in
this discussion, because there's no known alternative to Texinfo for
creating good documentation of the kind that we are used to in Emacs.
GCC folks are switching to Sphinx, but that has its own problems, and
without good support for it in Emacs (and no, rst.el is not enough, as
it doesn't provide enough markup commands, AFAICT) it isn't (yet) a
viable alternative.

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>> This is all part of the aims and process. However, the first step is to
>> develop a robust elisp based parser. This helps to ensure the org syntax
>> is consistent and helps identifies ambiguities which need to be fixed as
>> well as provides a reference implementation which developers can use to
>> develop parsers in other languages.
>
> Semantic/Bovine ??

Org is not context-free.
Also, Org maintaners previously rejected the idea of implementing Org
parser not in Elisp. Mainly because it would limit the ability to
maintain and contribute to Org - one would need to learn another
programming language to alter anything in Org syntax.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Richard Stallman <rms@gnu.org>,  Tim Cross <theophilusx@gmail.com>,
>  monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Sun, 12 Jun 2022 10:15:59 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > IIRC it's also much slower to create similarly sized .info files from
> > Org source than it is with makeinfo
> 
> Could you please quantify much slower?
> Do you have concrete example of a .org file which can demonstrate the
> problem?

With an unoptimized build of Emacs 29, generation of org.texi from
org.org takes 2.5 minutes here.  An optimized build of Emacs 28 does
that in 30 sec on one system and 1 minute on another.  Generation of
org.info then takes between 3 and 4 seconds.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Published Emacs Lisp packages in the wild typically come with no
> documentation at all, aside from doc strings and a single README file.
>
> Org mode happens to be popular for the latter, but I don't know why.
> The learning curve is high and the results seem to be unjustified.  An
> easier format such as HTML or Markdown can be used instead.

Which indicates that others may not agree with your assertion about the
learning curve.
For simple README
https://gitlab.com/publicvoit/orgdown/-/blob/master/doc/Orgdown1-Syntax-Examples.org
is more than enough.

Best,
Ihor

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Ihor Radchenko <yantar92@gmail.com> Cc:
    >> theophilusx@gmail.com, rms@gnu.org, monnier@iro.umontreal.ca,
    >> acm@muc.de, emacs-devel@gnu.org Date: Sun, 12 Jun 2022 18:38:45
    >> +0800
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > I really wonder how come no one on the Org list paid any
    >> attention to > the 10-fold to 40-fold slowdown in the time it
    >> takes to build the > manual, as result of that change.  But
    >> that's water under the bridge.
    >> 
    >> We rarely have bugs related to manual builds. I recall two in
    >> many years.
    >> 
    >> Usually documentation is built automatically on ELPA and by our
    >> publishing scripts on orgmode.org.

    > So basically no one builds Org, including the manual, on their own
    > system?  Even not the Org developers?

I build org-mode from git and have noticed the slowness in building the
documentation.

Weekly building from git is my way of avoiding having to deal
immediately with a myriad of changes in past Versions of org-mode.

Best wishes.

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>> This is all part of the aims and process. However, the first step is to
>>> develop a robust elisp based parser. This helps to ensure the org syntax
>>> is consistent and helps identifies ambiguities which need to be fixed as
>>> well as provides a reference implementation which developers can use to
>>> develop parsers in other languages.
>>
>> Semantic/Bovine ??
>
> Org is not context-free.

But could it be moved in that direction? (ie. Organice)

> Also, Org maintaners previously rejected the idea of implementing Org
> parser not in Elisp. Mainly because it would limit the ability to
> maintain and contribute to Org - one would need to learn another
> programming language to alter anything in Org syntax.

Hmmm. That would make it difficult to keep the language "parseable" by a
different parser.  Elisp would not provide the checks for (say) keeping
the language context-free.

-- 
David Masterson

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since we cannot start Org from scratch, factoring out individual modules
  > > is taking a lot of time and having the hostile attitudes expressed in
  > > some of the emails in this thread is not exactly encouraging.

  > I know it's a lot of work and I'm glad you (plural) are taking it on,

I feel the same.  To move some of the features out of Org, so that
they are available in all Emacs modes, will make enhance their
availabiliy to all Emacs users.

Also, it could eventually make Org itself simpler -- and less daunting
to try out and learn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > And anyway, I make the following offer: if someone doesn't want to
  > contribute documentation (or code) because he doesn't know Texinfo, he
  > can write the documentation in plain text, and I will translate it to
  > Texinfo for him.

Thank you for making this offer.  I hope that many contributors will
take you up on it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Published Emacs Lisp packages in the wild typically come with no
  > documentation at all, aside from doc strings and a single README file.

Depending on the quality of the README file, this could be imperfect
but adequate, or it could be lousy.  Please don't accept it as a standard
to judge other things by.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Werner LEMBERG]

> Why replace Texinfo?  Because it has some tricky syntax which it
> inherits directly from TeX.  And some syntax which is cumbersome,
> again imposed directly by TeX.

Well, the thing is backward compatibility – for the sake of the GNU
project.  If Texinfo was allowed to drop that (in particular, the many
unfortunate restrictions of the `.info` file format), everything would
be much easier.

> Nowadays nobody can maintain the TeX-based part which is used to
> generate printed manuals.

How do you come to this conclusion?  `texinfo.tex` is actively
maintained, and bug reports (which I submitted a lot) are usually
fixed within days.

Of course, there are some areas where more development would be
desirable (in particular, a better font interface based on ideas from
LaTeX).  On the other hand, the next Texinfo version will contain a
converter from Texinfo source to LaTeX source, which should help
overcome (at least some of) the font problems.

> (In the Info version of the file, the title of the node for @code is
> actually formatted wrong.  It has single quotes around `@code' in
> the title itself.  Texinfo does have some problems that need fixing,
> and it is hard to fix anything in it.)

I strongly suggest that you write to bug-texinfo, discussing this
issue and your other concerns.


    Werner

Re: Org mode and Emacs

[Werner LEMBERG]

> Had discussed with Gavin Smith about latex a couple of years ago,
> who explained that texinfo would have to be rewritten completely if
> we are to have a latex engine for it.  Lot of work and unlikely to
> happen.

By the way, has someone informed the Texinfo maintainers that people
are whetting knives here on the list, discussing quite fundamental
changes about the future of the project?


    Werner

Re: Org mode and Emacs

[Eli Zaretskii]

On June 13, 2022 8:22:57 AM GMT+03:00, Werner LEMBERG <wl@gnu.org> wrote:
> 
> > Had discussed with Gavin Smith about latex a couple of years ago,
> > who explained that texinfo would have to be rewritten completely if
> > we are to have a latex engine for it.  Lot of work and unlikely to
> > happen.
> 
> By the way, has someone informed the Texinfo maintainers that people
> are whetting knives here on the list, discussing quite fundamental
> changes about the future of the project?

Please don't over-dramatize.  No one is "wetting any knives" here, we are just talking.  If you followed the discussion, supplanting Texinfo is nowhere in sight yet.  This is Emacs, not GCC.

And yes, this discussion should move to the Texinfo list, as has been suggested already.

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> Semantic/Bovine ??
>>
>> Org is not context-free.
>
> But could it be moved in that direction? (ie. Organice)

I don't think so. It is motivated by the fundamental Org syntax design,
AFAIU. (mostly by first match wins design). We are not going to change
fundamentals of the Org syntax. It will break backward compatibility.

>> Also, Org maintaners previously rejected the idea of implementing Org
>> parser not in Elisp. Mainly because it would limit the ability to
>> maintain and contribute to Org - one would need to learn another
>> programming language to alter anything in Org syntax.
>
> Hmmm. That would make it difficult to keep the language "parseable" by a
> different parser.  Elisp would not provide the checks for (say) keeping
> the language context-free.

At this point, we are trying to "freeze" Org syntax as much as possible.
So, major changes are not expected. Different parsers should not suffer
from future changes (if they do, we should not make those changes to
start with).

As for keeping checks, we do have a set of parser tests using ERT. So,
major breakage will be prevented. On top of this, we plan to make the
parser tests more friendly to third-party tools:
https://orgmode.org/list/87fsqzi4tw.fsf@localhost

Best,
Ihor

Re: Org mode and Emacs

[Robert Pluim]

>>>>> On Tue, 14 Jun 2022 21:18:58 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> #+begin_export latex
    Ihor> This text will only appear when exported to \LaTeX, not to any other format.
    Ihor> #+end_export


    Ihor> They may be not great experience what one needs to do something like

    Ihor> #+begin_export texinfo
    Ihor> @var{test} will only appear in .texi output.
    Ihor> #+end_export

    Ihor> #+begin_export latex
    Ihor> \emph{test} will only appear in .tex output.
    Ihor> #+end_export

    Ihor> many times in the same .org document.

    Ihor> However, we have special blocks for this purpose. They are extendable,
    Ihor> as I descibed above. Though some improvements in Org core might be
    Ihor> needed if we have to use this extensibility more frequently.

My first thought here was Org macros, but as far as I can tell they canʼt
be multi-line, which would make them a bit cumbersome to use. Tell us
more about special blocks, the documentation on them is a bit sparse.

Robert
-- 

Re: Org mode and Emacs

[David Engster]

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> We rarely have bugs related to manual builds. I recall two in many
>>> years.
>>> 
>>> Usually documentation is built automatically on ELPA and by our
>>> publishing scripts on orgmode.org.
>>
>> So basically no one builds Org, including the manual, on their own
>> system?  Even not the Org developers?
>
> Org releases are not frequent for most users and occasional build time
> should not matter too much. Nobody complained on the list.

Well, for the record, I did complain, but that was over 11 years ago. :-)

https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

I still think Org is way too slow to be a viable documentation system
for a project like Emacs.

-David

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>>> So basically no one builds Org, including the manual, on their own
>>> system?  Even not the Org developers?
>>
>> Org releases are not frequent for most users and occasional build time
>> should not matter too much. Nobody complained on the list.
>
> Well, for the record, I did complain, but that was over 11 years ago. :-)
>
> https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01356.html

Point taken.

> I still think Org is way too slow to be a viable documentation system
> for a project like Emacs.

Things have changed since 2014.
On my system, using Emacs 28 and latest Org, exporting org-manual +
org-guide takes 18.8 sec. This time includes genering the .texi files
and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

org export takes: 15.4 sec, which is 4.5 x texinfo time.
Considering that Org export is more sofisticated compared to texinfo and
that Org export is written in Elisp, I see this performance as
acceptable for a documentation system.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Wed, 15 Jun 2022 15:36:16 +0800
> 
> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
> 
> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

I don't know about "acceptable", sorry.  It is definitely "endurable",
but 18.8 seconds is annoyingly long for an optimized build of Emacs to
do anything this simple.

Can you find any other single command that's part of building Emacs
which takes a comparable amount of time in an optimized build?

Re: Org mode and Emacs

[David Engster]

> On my system, using Emacs 28 and latest Org, exporting org-manual +
> org-guide takes 18.8 sec. This time includes genering the .texi files
> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.

On my system, generating org.texi from org.org takes 21s, makeinfo
org.texi takes 2.7s. But it's nice to see that it has become much faster
compared to 2014.

> org export takes: 15.4 sec, which is 4.5 x texinfo time.
> Considering that Org export is more sofisticated compared to texinfo and
> that Org export is written in Elisp, I see this performance as
> acceptable for a documentation system.

Why does the implementation language matter whether a documentation
system is acceptable? And while sophisticated, Org is actually still
missing features (like a proper index) to be a suitable replacement.

Org is about 1/16th of the whole Emacs documentation, so we're looking
at over 5min if everything was written in Org, give or take. People were
already up in arms when the switch from Texinfo v4 to v5 was done (which
switched to a Perl implementation). When documentation generation takes
a long time, writing it becomes more painful, as you cannot quickly
check the resulting output. And I think you underestimate how important
a quick build process is. Apart from developer annoyance, you need less
resources for CI, for instance.

-David

Re: Org mode and Emacs

[Pankaj Jangid]

Ihor Radchenko <yantar92@gmail.com> writes:

>> This is not needed for the emacs-28 branch,
>> so I meant master.
>
> I am not too familiar with how Emacs handles
> Org updates. AFAIK, Emacs master only tracks
> our stable tagged releases. Not the working
> branches (both bugfix and main). I do not
> know the motivation behind it and I do not
> know if it is a good idea to change it. Do
> you think that Emacs master should instead
> track our Org dev branch?
>

Emacs master is the development branch of
Emacs. I have always wanted Org dev branch to
be part of that. But may be that Org’s dev
branch is not as stable as Emacs’. Hence it is
kept that way.

I have complained about this branch confusion
once on the emacs-devel mailing list. Only to
learn that there is a separate mailing list for
org development related things.

Even for reporting bugs there was a different
mailing list for Org. I am not sure what is the
situation now. I use whichever version comes
with master branch of Emacs. And I report bugs
using ‘M-x report-emacs-bug’ only.

Re: Org mode and Emacs

[Visuwesh]

[வியாழன் ஜூன் 16, 2022 08:49] Pankaj Jangid wrote:

> Even for reporting bugs there was a different
> mailing list for Org. I am not sure what is the
> situation now. I use whichever version comes
> with master branch of Emacs. And I report bugs
> using ‘M-x report-emacs-bug’ only.

The situation is still the same and you're supposed to report org-mode
bugs with M-x org-submit-bug-report.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>> 
>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> I don't know about "acceptable", sorry.  It is definitely "endurable",
> but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> do anything this simple.

Ok. I pushed things a bit further. The latest Org main takes ~8sec to
generate Org manual. This _includes_ makeinfo call. Now, Org export
takes the same time with texinfo - ~4sec on my system.

Further improvements are not trivial, given my familiarity with Org
babel and Org export code.

> Can you find any other single command that's part of building Emacs
> which takes a comparable amount of time in an optimized build?

Optimized Emacs build takes >30 minutes on my system. Mainly because of
native compilation. 40, 20, or 10 seconds spent generating Org
documentation are negligible for me.

Best,
Ihor

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: deng@randomsample.de,  theophilusx@gmail.com,  rms@gnu.org,
>   monnier@iro.umontreal.ca,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 16 Jun 2022 13:36:35 +0800
> 
> > I don't know about "acceptable", sorry.  It is definitely "endurable",
> > but 18.8 seconds is annoyingly long for an optimized build of Emacs to
> > do anything this simple.
> 
> Ok. I pushed things a bit further. The latest Org main takes ~8sec to
> generate Org manual. This _includes_ makeinfo call. Now, Org export
> takes the same time with texinfo - ~4sec on my system.

Thanks, this is awesome.  Hope to see this speedup in the Emacs
repository soon.

> > Can you find any other single command that's part of building Emacs
> > which takes a comparable amount of time in an optimized build?
> 
> Optimized Emacs build takes >30 minutes on my system. Mainly because of
> native compilation.

The slowness of native-compilation is a long-standing bug report, so
I'd rather not use that as a reference for acceptable speed.  I meant
to compare with other commands run by the build.

> 40, 20, or 10 seconds spent generating Org documentation are
> negligible for me.

They aren't negligible, even in a native-compilation build, because
they happen at the end, when all the rest was already done, and no
large N in "-j N" of the Make command can help with that last long
wait.

Re: Org mode and Emacs

[Ihor Radchenko]

David Engster <deng@randomsample.de> writes:

>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>> org-guide takes 18.8 sec. This time includes genering the .texi files
>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>
> On my system, generating org.texi from org.org takes 21s, makeinfo
> org.texi takes 2.7s. But it's nice to see that it has become much faster
> compared to 2014.

Can you test the latest Org main?

>> org export takes: 15.4 sec, which is 4.5 x texinfo time.
>> Considering that Org export is more sofisticated compared to texinfo and
>> that Org export is written in Elisp, I see this performance as
>> acceptable for a documentation system.
>
> Why does the implementation language matter whether a documentation
> system is acceptable? And while sophisticated, Org is actually still
> missing features (like a proper index) to be a suitable replacement.

Well. It was a rather hand-waving argument: Elisp is not as fast of many
other programming languages. So software written in Elisp may be slow
despite the best effort. etc etc

In any case, that argument is futile now. I managed to get Org export
work as fast as texinfo.

As for proper index, we do support index for texinfo export
specifically and for publishing websites. A more general-purpose index
support is in the works. See https://github.com/tecosaur/org-glossary

> Org is about 1/16th of the whole Emacs documentation, so we're looking
> at over 5min if everything was written in Org, give or take. People were
> already up in arms when the switch from Texinfo v4 to v5 was done (which
> switched to a Perl implementation). When documentation generation takes
> a long time, writing it becomes more painful, as you cannot quickly
> check the resulting output. And I think you underestimate how important
> a quick build process is. Apart from developer annoyance, you need less
> resources for CI, for instance.

Fair point.

Best,
Ihor

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Optimized Emacs build takes >30 minutes on my system. Mainly because of
>> native compilation.
>
> The slowness of native-compilation is a long-standing bug report, so
> I'd rather not use that as a reference for acceptable speed.  I meant
> to compare with other commands run by the build.

The other relatively slow part of the built process without native-comp
is Scraping autoloads.

Re: Org mode and Emacs

[David Engster]

> David Engster <deng@randomsample.de> writes:
>
>>> On my system, using Emacs 28 and latest Org, exporting org-manual +
>>> org-guide takes 18.8 sec. This time includes genering the .texi files
>>> and running texinfo. From that 18.8 sec, texinfo takes 3.4 sec to run.
>>
>> On my system, generating org.texi from org.org takes 21s, makeinfo
>> org.texi takes 2.7s. But it's nice to see that it has become much faster
>> compared to 2014.
>
> Can you test the latest Org main?

I tested with latest Emacs master and with increased gc-cons-threshold
it takes ~5 seconds to generate org.texi. Great job!

-David

Re: Org mode and Emacs

[Stefan Monnier]

Ihor Radchenko [2022-06-16 20:58:07] wrote:
> Mattias Engdegård <mattiase@acm.org> writes:
>>> I limited
>>> gc-cons-threshold in doc/misc/Makefile.in to 50,000,000, not
>>> most-positive-fixnum.
>>
>> Some timings for the export to .texi (old machine, optimised build, bytecode only):
>>
>> |    gc-cons-threshold | time |
>> |----------------------+------|
>> |               800000 | 14.1 | (default value)
>> |             25000000 |  6.2 |
>> |             50000000 |  5.7 |
>> |            100000000 |  5.7 |
>> | most-positive-fixnum |  5.1 |
>>
>> thus Eli's choice is very good, and we really should do something about that GC of ours.
>
> I am wondering if there is a "universal" value suitable for one-off Emacs
> batch invocations.

I doubt that's the case, but of course we should try and use values that
work "overall best" on average.  Maybe we should revisit the current
knobs and their default values.

Currently the two important knobs we have are:

    gc-cons-threshold
    gc-cons-percentage

Maybe it's time for some benchmarks with various values for these knobs
to see if the values we're using now are "good enough" or might need to
be changed.

We could also try and be bit more discerning.  E.g. currently when the
program is in a phase where it builds a lot of new data-structures, we
run the GC everytime some amount of new memory has been allocated, but
presumably almost none of those objects have died yet (we're still
building the overall datastructure) so the GC will be a pure waste of
time.  OTOH in other phases the code allocates objects which are used
only temporarily, in which case it's beneficial to run the GC somewhat
frequently to avoid growing our heap unnecessarily (and suffering
fragmentation as a result).

It's hard to know beforehand whether a GC will be useful or not, tho.
But maybe we can find good heuristics.  E.g. have something like
a `gc-cons-percentage` which depends on how much garbage we collected in
the last GC: if a GC doesn't collect any garbage (or very little of it),
it's a sign that we're in a phase where running the GC is not very
useful so we should wait a bit more until the next GC, whereas if the GC
collected a fair bit of garbage, it's a sign that we're in a phase where
running the GC is beneficial and we can run it a bit more often.


        Stefan

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> Semantic/Bovine ??
>>>
>>> Org is not context-free.
>>
>> But could it be moved in that direction? (ie. Organice)
>
> I don't think so. It is motivated by the fundamental Org syntax design,
> AFAIU. (mostly by first match wins design). We are not going to change
> fundamentals of the Org syntax. It will break backward compatibility.

Could Org be moved toward a "well-defined" grammar that could be
separated from the Emacs implementation to allow other systems (iOS,
Android, Windows) to implement (at least part of) a "standard" Org?
Could the backward compatibility be covered by an Emacs library where
necessary? 

>>> Also, Org maintaners previously rejected the idea of implementing Org
>>> parser not in Elisp. Mainly because it would limit the ability to
>>> maintain and contribute to Org - one would need to learn another
>>> programming language to alter anything in Org syntax.
>>
>> Hmmm. That would make it difficult to keep the language "parseable" by a
>> different parser.  Elisp would not provide the checks for (say) keeping
>> the language context-free.
>
> At this point, we are trying to "freeze" Org syntax as much as possible.
> So, major changes are not expected. Different parsers should not suffer
> from future changes (if they do, we should not make those changes to
> start with).
>
> As for keeping checks, we do have a set of parser tests using ERT. So,
> major breakage will be prevented. On top of this, we plan to make the
> parser tests more friendly to third-party tools:
> https://orgmode.org/list/87fsqzi4tw.fsf@localhost

This sounds good.

-- 
David Masterson

Re: Org mode and Emacs

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

>>> But could it be moved in that direction? (ie. Organice)
>>
>> I don't think so. It is motivated by the fundamental Org syntax design,
>> AFAIU. (mostly by first match wins design). We are not going to change
>> fundamentals of the Org syntax. It will break backward compatibility.
>
> Could Org be moved toward a "well-defined" grammar that could be
> separated from the Emacs implementation to allow other systems (iOS,
> Android, Windows) to implement (at least part of) a "standard" Org?
> Could the backward compatibility be covered by an Emacs library where
> necessary? 

Yes. See https://orgmode.org/worg/dev/org-syntax.html
I hope it is well-defined enough for you.
I don't think that we need major changes to allow implementation in
other systems. There is already a number of existing third-party parsers
for Org:

https://github.com/200ok-ch/org-parser
https://github.com/tgbugs/laundry
https://github.com/milisims/tree-sitter-org
https://github.com/tecosaur/Org.jl

Best,
Ihor

Re: Org mode and Emacs

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>>>> But could it be moved in that direction? (ie. Organice)
>>>
>>> I don't think so. It is motivated by the fundamental Org syntax design,
>>> AFAIU. (mostly by first match wins design). We are not going to change
>>> fundamentals of the Org syntax. It will break backward compatibility.
>>
>> Could Org be moved toward a "well-defined" grammar that could be
>> separated from the Emacs implementation to allow other systems (iOS,
>> Android, Windows) to implement (at least part of) a "standard" Org?
>> Could the backward compatibility be covered by an Emacs library where
>> necessary? 
>
> Yes. See https://orgmode.org/worg/dev/org-syntax.html
> I hope it is well-defined enough for you.
> I don't think that we need major changes to allow implementation in
> other systems. There is already a number of existing third-party parsers
> for Org:
>
> https://github.com/200ok-ch/org-parser
> https://github.com/tgbugs/laundry
> https://github.com/milisims/tree-sitter-org
> https://github.com/tecosaur/Org.jl

Excellent!

-- 
David Masterson

Re: Org mode and Emacs

[Bastien]

Here is the email I sent when I accepted to switch to using .org as
the native format for Org documentation:

https://list.orgmode.org/orgmode/87371gfas7.fsf@bzg.fr/

As you can read, I wanted to make the switch as an experiment to see
if we were really solving a problem here.

I believe we didn't get more contributions to the manual by switching
to .org, so I'd be in favor of switching back to using .texi as the
native format for Org's documentation.  (Not for 9.6, obviously, more
probably for 10.0 -- I'll discuss this with other Org maintainers.)

The other topics in this thread (make Org's Texinfo exporter provide
good .texi manuals, make Org more modular, etc.) are interesting, but
they are really separate questions IMHO.

-- 
 Bastien

Re: Org mode and Emacs

[Payas Relekar]

Hi Bastien,

Bastien <bzg@gnu.org> writes:

> Here is the email I sent when I accepted to switch to using .org as
> the native format for Org documentation:
>
> https://list.orgmode.org/orgmode/87371gfas7.fsf@bzg.fr/
>
> As you can read, I wanted to make the switch as an experiment to see
> if we were really solving a problem here.
>
> I believe we didn't get more contributions to the manual by switching
> to .org, so I'd be in favor of switching back to using .texi as the
> native format for Org's documentation.  (Not for 9.6, obviously, more
> probably for 10.0 -- I'll discuss this with other Org maintainers.)

From your mail, below were the motivators for change:

- Let's stabilize editing standards around the org.org file.

- Let's test org capabilities against a giant .org file.

- Let's make `C-x 4 a' do something useful in an .org section.

- Let's write more non-emacs parsers and exporters.

- Let's see if we have more contributions to the manual and if
  we really solved a problem here.

While you're best to judge the number of contributions, #1 and #2, or
the dogfooding opportunities provided by the switch are immense.

One doesn't occasionally run into org documents the size of org.org. It
has already resulted in gc enhancements as it was slowing down Emacs
build and was optimized. I'll say that alone is a benefit worth keeping.

There is also more progress being made on non-emacs parsers[0][1], and
perhaps we can reach out if they find org.org useful. Generally
speaking, these projects are even smaller (number of contributors-wise)
than org-mode, but a shared burden is always nicer.

[0]: https://github.com/nvim-orgmode/orgmode
[1]: https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf

> The other topics in this thread (make Org's Texinfo exporter provide
> good .texi manuals, make Org more modular, etc.) are interesting, but
> they are really separate questions IMHO.

Thanks,
Payas
--

Re: Org mode and Emacs

[Bastien]

Hi Payas,

Payas Relekar <relekarpayas@gmail.com> writes:

> From your mail, below were the motivators for change:

Re-reading it, the list was a mix of (1) things to be done for the
switch to be sensible (and that will be boosted if the switch happens)
and (2) possible nice outcomes.

> - Let's stabilize editing standards around the org.org file.

(=> 1) Something that we did before the switch.

> - Let's test org capabilities against a giant .org file.

(=> 2) Yes, this can lead to enhancements in Org like it did recently,
but I don't think this is a good reason enough to justify the switch.

> - Let's make `C-x 4 a' do something useful in an .org section.

(=> 1) Also something now available.

> - Let's write more non-emacs parsers and exporters.

(=> 2) This was an illusion: I don't think projects like Pandoc use
the org-manual.org file to test whether they are good parsers and
exporters.  For a good reason: nobody really needs to use Pandoc for
parsing/exporting the Org manual.

> - Let's see if we have more contributions to the manual and if
>   we really solved a problem here.

(=> 2) This didn't happen.

> While you're best to judge the number of contributions, #1 and #2, or
> the dogfooding opportunities provided by the switch are immense.
>
> One doesn't occasionally run into org documents the size of org.org. It
> has already resulted in gc enhancements as it was slowing down Emacs
> build and was optimized. I'll say that alone is a benefit worth keeping.

I'm not convinced: slowing down Emacs build to create opportunities
for enhancing the Org parser and exporter does not strike me as a good
reason.

Respecting the GNU standards about manuals ("The preferred document
format for the GNU system is the Texinfo formatting language.") and
the recent discussions provide good reasons for switching back to
.texi, if all maintainers agree.

> There is also more progress being made on non-emacs parsers[0][1], and
> perhaps we can reach out if they find org.org useful. Generally
> speaking, these projects are even smaller (number of contributors-wise)
> than org-mode, but a shared burden is always nicer.
>
> [0]: https://github.com/nvim-orgmode/orgmode
> [1]: https://github.com/200ok-ch/org-parser/blob/master/resources/org.ebnf

Of course, feel free to contact them, but why would they want to try
solving the challenge for exporting org-manual.org?  They can create
any .org file with the complexity they desire if they need it.  Also,
independently from this discussion, _we_ should certainly provide an
example document containing alls things a parser/exporter should be
able to handle.

All best,

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

>> While you're best to judge the number of contributions, #1 and #2, or
>> the dogfooding opportunities provided by the switch are immense.
>>
>> One doesn't occasionally run into org documents the size of org.org. It
>> has already resulted in gc enhancements as it was slowing down Emacs
>> build and was optimized. I'll say that alone is a benefit worth keeping.
>
> I'm not convinced: slowing down Emacs build to create opportunities
> for enhancing the Org parser and exporter does not strike me as a good
> reason.

As the outcome of that thread, I managed to reduce org->texinfo
conversion down to 4sec. https://yhetil.org/emacs-devel/87czf9np98.fsf@localhost
I do not think that the overall impact on the build process is something
we need to worry about in this context.

> Respecting the GNU standards about manuals ("The preferred document
> format for the GNU system is the Texinfo formatting language.") and
> the recent discussions provide good reasons for switching back to
> .texi, if all maintainers agree.

Do note that RMS expressed interest in changing this standard to Org,
given that Org provides all the necessary tooling to produce
high-quality manual.
https://yhetil.org/emacs-devel/E1nzQh5-0001OB-22@fencepost.gnu.org

I also spawned a thread about this in Org ML.
https://list.orgmode.org/87k09frdsv.fsf@localhost/

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> As the outcome of that thread, I managed to reduce org->texinfo
> conversion down to 4sec. https://yhetil.org/emacs-devel/87czf9np98.fsf@localhost
> I do not think that the overall impact on the build process is something
> we need to worry about in this context.

Thanks a lot for working on this, this is a great achievement.

To be fair, the impact on the build process was not really my main
concern.  My concern is that Org does not respect the GNU standards.

>> Respecting the GNU standards about manuals ("The preferred document
>> format for the GNU system is the Texinfo formatting language.") and
>> the recent discussions provide good reasons for switching back to
>> .texi, if all maintainers agree.
>
> Do note that RMS expressed interest in changing this standard to Org,
> given that Org provides all the necessary tooling to produce
> high-quality manual.
> https://yhetil.org/emacs-devel/E1nzQh5-0001OB-22@fencepost.gnu.org

I'm aware of this possibility, but there are many "if" :)

Also, we can work toward this possibility without using .org for the
Org manual.

What do you think of my proposal to produce an example document,
demonstrating the Org syntax and its possibilities?  

It would surely help us test our own code (parser and exporters) and
help others write parsers/exporters (outside of Emacs).

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

> To be fair, the impact on the build process was not really my main
> concern.  My concern is that Org does not respect the GNU standards.

We do. Org does provide manual in Texinfo format. The fact that our
texinfo version is auto-generated is irrelevant wrt GNU Standards.

> What do you think of my proposal to produce an example document,
> demonstrating the Org syntax and its possibilities?  
>
> It would surely help us test our own code (parser and exporters) and
> help others write parsers/exporters (outside of Emacs).

See https://orgmode.org/list/87fsqzi4tw.fsf@localhost

P.S. Note that I am trying to keep discussion relevant to this
emacs-devel thread. More serious discussion about switching back from
.org to .texinfo should probably happen at Org ML.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Bastien <bzg@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 25 Sep 2022 08:35:54 +0200
> 
> Respecting the GNU standards about manuals ("The preferred document
> format for the GNU system is the Texinfo formatting language.") and
> the recent discussions provide good reasons for switching back to
> .texi, if all maintainers agree.

I definitely agree (it makes it easier for me to contribute to the Org
manual).  But eventually it is the decision of the Org project.

Re: Org mode and Emacs

[Bastien Guerry]

Bastien <bzg@gnu.org> writes:

> What do you think of my proposal to produce an example document,
> demonstrating the Org syntax and its possibilities?  

PS: I see that you raised a similar idea on the Org-mode mailing list,
I will follow-up there.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Tim Cross]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Bastien <bzg@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 25 Sep 2022 08:35:54 +0200
>> 
>> Respecting the GNU standards about manuals ("The preferred document
>> format for the GNU system is the Texinfo formatting language.") and
>> the recent discussions provide good reasons for switching back to
>> .texi, if all maintainers agree.
>
> I definitely agree (it makes it easier for me to contribute to the Org
> manual).  But eventually it is the decision of the Org project.

and I would be exactly the opposite. I know and use org-mode. I don't
know and have never used texinfo despite over 25 years of Emacs
use. Having to learn another formatting solution just to contribute to
the formatting solution I use isn't going to be encouraging. 

The question I wonder about is where are we most likely to get the
majority of our contributions from, those who use org mode and know it
or those who don't and for those who use org-mode, how many will know texinfo? 

I know one of the original justifications was to see if it improved
contributions towards documentation and it appears this has not been the
case. However, I do wonder about that - I have certainly seen numerous
manual patches on the list and I wonder how many of those would not
occur if the patcher also needed to know texinfo? There may also be
other impediments which slows down contributions that are unrelated to
the documentation format (I still find determining if something is a
known issue or not and the state of progress to resolving it difficult
to track - not a criticism of the core maintainers, who I believe do an
incredible job. Real problem is the challenge of realising a better
process given the very very few core contributors available - basically
a resourcing challenge).  

At the end of the day, I think the dog food argument is
important. Having the manual in org format has seen a number of
improvements and does provide a good and most importantly large and used
example. Having a sample document which developers could use to verify
parsers etc would be a good addition, but the problem with such
documents is they tend not to be maintained and are not actively
used. There is huge value in having a large and reasonably complex
document which is being actively updated/enhanced and which is used in
the real world to produce documents in various formats which are also
actively read and used. It tends to be in active use of generated
documents we find more subtle issues, things which tend to be
missed in cursory scans of test documents.

Re: Org mode and Emacs

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

Bastien <bzg@gnu.org> writes:

>> - Let's see if we have more contributions to the manual and if
>>   we really solved a problem here.
>
> (=> 2) This didn't happen.

I played around with the git log data and your claim appears to be
incorrect. The number of commits affecting manual increases
substantially according to my analysis. And it is not just initial spike
of fixes after introducing the new format.

See the attached histogram.


[-- Attachment #2: manual-commits-stats.png --]
[-- Type: image/png, Size: 27488 bytes --]

[-- Attachment #3: Type: text/plain, Size: 916 bytes --]


Methodology:

The org manual has been introduced around d330eed7c (2017-12)

#+begin_src bash
git log main --grep manual --format=%ci > manual-commits.dat
cat manual-commits.dat | awk '{print $1}' | cut -d- -f1,2 > manual-commits-month.dat 
#+end_src

#+begin_src gnuplot
set timefmt '%Y-%m'
set xdata time
set title "Number of commits according to \"git log --grep manual\"" font ",20"
set tics out
set label 1 ".texi → .org" at "2017-12",155 center font ",18" textcolor rgb'red'
set arrow from "2017-12",0 to "2017-12",150 nohead front lw 10 lc rgb'red'
set style fill solid 0.5 border
set yrange [0:160]
plot 'manual-commits-month.dat' u 1:(1.0) bins=40 w boxes t''
#+end_src

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@gmail.com> writes:

> I played around with the git log data and your claim appears to be
> incorrect. The number of commits affecting manual increases
> substantially according to my analysis. And it is not just initial spike
> of fixes after introducing the new format.
>
> See the attached histogram.
>
>
>
>
> Methodology:
>
> The org manual has been introduced around d330eed7c (2017-12)
>
> #+begin_src bash
> git log main --grep manual --format=%ci > manual-commits.dat
> cat manual-commits.dat | awk '{print $1}' | cut -d- -f1,2 > manual-commits-month.dat 
> #+end_src
>
> #+begin_src gnuplot
> set timefmt '%Y-%m'
> set xdata time
> set title "Number of commits according to \"git log --grep manual\"" font ",20"
> set tics out
> set label 1 ".texi → .org" at "2017-12",155 center font ",18" textcolor rgb'red'
> set arrow from "2017-12",0 to "2017-12",150 nohead front lw 10 lc rgb'red'
> set style fill solid 0.5 border
> set yrange [0:160]
> plot 'manual-commits-month.dat' u 1:(1.0) bins=40 w boxes t''
> #+end_src

Maybe that points to Org being more maintainence-heavy than Texinfo?

Anyway, I've made the following offer and will make it again: if anyone
who does not know Texinfo wants to contribute documentation to Emacs, he
can write it in plain text, and I will manually convert it to Texinfo.

Re: Org mode and Emacs

[Emanuel Berg]

Po Lu wrote:

> Anyway, I've made the following offer and will make it
> again: if anyone who does not know Texinfo wants to
> contribute documentation to Emacs, he can write it in plain
> text, and I will manually convert it to Texinfo.

There are tools for that ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: Org mode and Emacs

[Bastien]

Hi Tim,

Tim Cross <theophilusx@gmail.com> writes:

> The question I wonder about is where are we most likely to get the
> majority of our contributions from, those who use org mode and know it
> or those who don't and for those who use org-mode, how many will
> know texinfo?

Recruiting contributors for Org is also a way to recruit contributors
for the GNU project in general, which uses Texinfo as its standard
format for manuals.

For occasional fixes, I don't think the difference between the .texi
and .org format makes that much of a difference.

For substantial contributions, it probably does: but contributors of
these important changes are probably those for which this difference
can easily be overcome -- and *should* be overcome, because they are
also potential contributors for the GNU project.

> (I still find determining if something is a
> known issue or not and the state of progress to resolving it difficult
> to track

(FWIW I agree, that's the motivation behind my work on Woof!.)

> Real problem is the challenge of realising a better
> process given the very very few core contributors available - basically
> a resourcing challenge).

What we don't see so far is the contributors we lose because we use
.org as the format for the manual: Eli is one and there are probably
others.

> At the end of the day, I think the dog food argument is
> important. Having the manual in org format has seen a number of
> improvements and does provide a good and most importantly large and used
> example. Having a sample document which developers could use to verify
> parsers etc would be a good addition, but the problem with such
> documents is they tend not to be maintained and are not actively
> used. There is huge value in having a large and reasonably complex
> document which is being actively updated/enhanced and which is used in
> the real world to produce documents in various formats which are also
> actively read and used. It tends to be in active use of generated
> documents we find more subtle issues, things which tend to be
> missed in cursory scans of test documents.

Full disclosure: the dog food argument never convinced me.  Dog
fooding /per se/ never makes any sense, unless you motivate it with
another good reason.

I suspect our (lispian?) brains is fascinated by recursive stuff (a
rose is a rose is a rose) but this is something we should resist.

Anyway, I won't insist on this anymore, the decision will be that
of all Org core maintainers, of course.

-- 
 Bastien

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> I played around with the git log data and your claim appears to be
> incorrect. 

Thanks for providing these numbers, I stand corrected then.

But I'm not *that* impressed as I still doubt we can infer a causal
connection between the texi->org switch and the increasing number of
commits (I've read the detailed logs.)

If you, Nicolas, Tim and other core contributors say that switching
back to .texi will definitely slow down your contributions to Org
(these has to be considered and said in all honesty, which is hard)
that I would consider a strong argument against the switch back to
.texi---but that's our collective decision anyway, so I'm fine with
whatever everyone decides (not now, around Org 10).

-- 
 Bastien

Re: Org mode and Emacs

[Bastien Guerry]

Po Lu <luangruo@yahoo.com> writes:

> Anyway, I've made the following offer and will make it again: if anyone
> who does not know Texinfo wants to contribute documentation to Emacs, he
> can write it in plain text, and I will manually convert it to Texinfo.

That's a very good move, one that we should make for Org too.

I'll propose something on our guide for contributors.

-- 
 Bastien

Re: Org mode and Emacs

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Bastien <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> I played around with the git log data and your claim appears to be
>> incorrect. 
>
> Thanks for providing these numbers, I stand corrected then.

Actually, the number are not as accurate as I thought.
The old manual was in doc/org.texi and had no word "manual" in it.

Grepping the file names yields (different colours represent .texi and .org)


[-- Attachment #2: manual-commits-v2.png --]
[-- Type: image/png, Size: 16592 bytes --]

[-- Attachment #3: Type: text/plain, Size: 117 bytes --]


which is still an improvement because the total number of commits
is just a fraction of the early Org development.


[-- Attachment #4: total-commits-stats.png --]
[-- Type: image/png, Size: 27303 bytes --]

[-- Attachment #5: Type: text/plain, Size: 1036 bytes --]


> But I'm not *that* impressed as I still doubt we can infer a causal
> connection between the texi->org switch and the increasing number of
> commits (I've read the detailed logs.)

I am sure that you are a lot more familiar with the historic data.

> If you, Nicolas, Tim and other core contributors say that switching
> back to .texi will definitely slow down your contributions to Org
> (these has to be considered and said in all honesty, which is hard)
> that I would consider a strong argument against the switch back to
> .texi---but that's our collective decision anyway, so I'm fine with
> whatever everyone decides (not now, around Org 10).

Learning .texi will certainly be an extra burden for me. Not impossible,
but feels awkward that I'd need it for Org.

In any case, emacs-devel is not the place to decide this.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

> Hi Tim,
>
> Tim Cross <theophilusx@gmail.com> writes:
>
>> The question I wonder about is where are we most likely to get the
>> majority of our contributions from, those who use org mode and know it
>> or those who don't and for those who use org-mode, how many will
>> know texinfo?
>
> Recruiting contributors for Org is also a way to recruit contributors
> for the GNU project in general, which uses Texinfo as its standard
> format for manuals.

I am now more concerned about Org contributors. I am not sure if we have
a luxury to push our contributors into GNU project by forcing them to
learn texinfo in addition to Org (note that I do not say that we should
not encourage people to contribute to GNU; just that we should not
discourage people from becoming Org contributors in the first place).

>> Real problem is the challenge of realising a better
>> process given the very very few core contributors available - basically
>> a resourcing challenge).
>
> What we don't see so far is the contributors we lose because we use
> .org as the format for the manual: Eli is one and there are probably
> others.

I am wondering how many Emacs developers contributed to Org in the past.
In particular, before we switched away from .texi manual.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Maybe that points to Org being more maintainence-heavy than Texinfo?

While I am not familiar with Texinfo, I can only recall a single issue
with Org manual that is related to it being written in Org. I would not
call it maintainence-heavy.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Bastien]

Ihor Radchenko <yantar92@gmail.com> writes:

> I am now more concerned about Org contributors. I am not sure if we have
> a luxury to push our contributors into GNU project by forcing them to
> learn texinfo in addition to Org (note that I do not say that we should
> not encourage people to contribute to GNU; just that we should not
> discourage people from becoming Org contributors in the first place).

I will write to you off-list this week to try to find a way to define
our strategy here (beyond the sole topic of the manual).

I suggest we have an online event after the EmacsConf (Dec. 11th would
be good) to include the community in our decisions about setting up a
strategy for Org future maintenance and contributions.

> I am wondering how many Emacs developers contributed to Org in the past.
> In particular, before we switched away from .texi manual.

My impression is that these contributions have been small in size of
text modified but very useful and regular, at least as much useful as
the occasional ones received from Org casual contributors.

-- 
 Bastien

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Tim Cross <theophilusx@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 14:35:00 +0800
> 
> I am wondering how many Emacs developers contributed to Org in the past.
> In particular, before we switched away from .texi manual.

You can easily see that with the following command:

  git log -- doc/misc/org.texi

Re: Org mode and Emacs

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2022-09-26 09:35]:
> I am now more concerned about Org contributors. I am not sure if we have
> a luxury to push our contributors into GNU project by forcing them to
> learn texinfo in addition to Org (note that I do not say that we should
> not encourage people to contribute to GNU; just that we should not
> discourage people from becoming Org contributors in the first
> place).

You may have contributors write it in Org and then convert Org to
texinfo by using pandoc:

pandoc -f org -t texinfo your-file.org


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Org mode and Emacs

[Ihor Radchenko]

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2022-09-26 09:35]:
>> I am now more concerned about Org contributors. I am not sure if we have
>> a luxury to push our contributors into GNU project by forcing them to
>> learn texinfo in addition to Org (note that I do not say that we should
>> not encourage people to contribute to GNU; just that we should not
>> discourage people from becoming Org contributors in the first
>> place).
>
> You may have contributors write it in Org and then convert Org to
> texinfo by using pandoc:
>
> pandoc -f org -t texinfo your-file.org

This is almost what we do now. Org contributors write in Org, and then we
use Org itself to convert to texinfo. Org files serve as the source
files.

What we are discussing here is whether Org should switch from Org
sources back to texinfo sources used in Emacs.

1. Switching to texinfo will improve Emacs coherence and potentially
   increase the pool of Emacs contributors familiar with texinfo (by
   pulling some of the Org contributors)
2. On the other hand, switching from Org to texinfo sources on Org side
   will require Org contributors to learn texinfo, which they do not
   really need now.
3. Yet, switching from Org to texinfo will allow Emacs contributors to
   send patches in familiar texinfo format to Org.

From the perspective of Emacs, using texinfo sources is a good thing.
From the perspective of Org, it is not that obvious.
Hence this discussion.

Note that we are now discussing this on emacs-devel and do not yet have
inputs from many Org users/contributors. However, we (Org devs) can get
inputs from Emacs devs, which is part of the information we need to see
what we want to do about Org vs. Texinfo sources.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Robert Pluim]

>>>>> On Mon, 26 Sep 2022 17:54:33 +0800, Ihor Radchenko <yantar92@gmail.com> said:

    Ihor> 1. Switching to texinfo will improve Emacs coherence and potentially
    Ihor>    increase the pool of Emacs contributors familiar with texinfo (by
    Ihor>    pulling some of the Org contributors)
    Ihor> 2. On the other hand, switching from Org to texinfo sources on Org side
    Ihor>    will require Org contributors to learn texinfo, which they do not
    Ihor>    really need now.
    Ihor> 3. Yet, switching from Org to texinfo will allow Emacs contributors to
    Ihor>    send patches in familiar texinfo format to Org.

    Ihor> From the perspective of Emacs, using texinfo sources is a good thing.
    Ihor> From the perspective of Org, it is not that obvious.
    Ihor> Hence this discussion.

As someone who has written the odd bit of Emacs documentation and
(less) org-mode documentation, I find the cognitive load of texinfo or
org to be about the same [1][2]; whichever one I wrote most recently is
relatively easy to use, and the other I need to relearn. I suspect the
same is true for most casual submitters of documentation patches.

Robert

Footnotes:
[1]  In a lot of cases itʼs "scroll around to see what the rest of the
     document is doing and just copy that", which is perhaps
     inefficent of my time, but works.

[2]  The load would be less if I didnʼt *also* have to write Markdown
     for work, but Emacs help a lot with that.  😀

-- 

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Speaking as the Chief GNUisance, rssponsible for GNU Project
standards, I would be happy to adopt an upgraded Org format as a new
standard source format for GNU manuals, _provided_ Org format has been
extended with the capability to express all the constructions and
distinctions that Texinfo can express, generate all the output formats
Texinfo can generate, and use TeX to make beautiful printed output.

Texinfo can generate these output formats: Info files, HTML, ASCII
text, and DVI and PDF files via TeX.

Texinfo provides numerous subtle distinctions that show up clearly in
each of these output formats.  Compare, for example, @var, @dfn and
@emph; compare @code, @samp, @file, @command, @option, @kbd, and @key.

I am sure people can extend Org software to handle these semantic
distinctions and generate these output formats.  Since it has been
done once, it can be done again.  But the work is not trivial.

The work has to start by designing what the extended Org format will look
like.  That part is the crucial part; once it has been specified,
people can work independently to implement various parts of handling
that format.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1206 bytes --]

I think there are two (perhaps more) aspects to writing good
documentation and accepting / making contributions:

1. The basics -- write a paragraph or two describing some feature.
Orthogonal to that and here is where each format bites in different
ways:

2. Integrating that contribution into the main corpus of the work --- at
   the write place, with the right surrounding context, generating the
   right table-of-contents and menus (a running nightmare in Texinfo if
   you dont set things up right at the beginning) and so on.

I think (2) might likely contribute negatively to  (1) happening more
often.

So rather than arguing about formats (and it's been pointed out already
that there are tools for automatic conversion) we should perhaps:

A. Create a simple content repository where contributors can check-in
   their small to medium contributions in the format of their choice
   (limited to a few open formats) along with the meta-data 
B. A small group of more experienced volunteers take on the task of
   incorporating those contributions into the main work.


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Po Lu <luangruo@yahoo.com>,  Ihor Radchenko <yantar92@gmail.com>,  Payas
>  Relekar <relekarpayas@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 26 Sep 2022 06:58:31 -0700
> 
> I think there are two (perhaps more) aspects to writing good
> documentation and accepting / making contributions:
> 
> 1. The basics -- write a paragraph or two describing some feature.
> Orthogonal to that and here is where each format bites in different
> ways:
> 
> 2. Integrating that contribution into the main corpus of the work --- at
>    the write place, with the right surrounding context, generating the
>    right table-of-contents and menus (a running nightmare in Texinfo if
>    you dont set things up right at the beginning) and so on.
> 
> I think (2) might likely contribute negatively to  (1) happening more
> often.

I think the division you suggest is somewhat artificial.  Unlike man
pages, Info manuals are not a collection of loosely-couples articles;
instead, the sections in a good Info manual are always connected and
context-aware.  Thus, writing the documentation for the manual should
_always_ start by studying the "neighborhood" in which that new stuff
will live.  IOW, putting the new text in context is not an
after-thought, it should be considered prior to writing.

I don't know what you allude to as "running nightmare in Texinfo".
Adding a new node to a manual is since long ago a very simple process:
just add it and make sure the @menu of the parent node (if there is
one) is updated to have the new node in its correct place.  That's it.

> So rather than arguing about formats (and it's been pointed out already
> that there are tools for automatic conversion) we should perhaps:
> 
> A. Create a simple content repository where contributors can check-in
>    their small to medium contributions in the format of their choice
>    (limited to a few open formats) along with the meta-data 
> B. A small group of more experienced volunteers take on the task of
>    incorporating those contributions into the main work.

Most contributors to Emacs already write pretty good Texinfo; adding
finishing touches to that and fixing a few style or markup mistakes is
already being done in the background by those who spot those blunders.
So I think we already have the above in place.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is almost what we do now. Org contributors write in Org, and then we
  > use Org itself to convert to texinfo. Org files serve as the source
  > files.

In order for that to produce proper Texinfo source as output, there
would need to be, in Org format, ways to express the many distinctions
that Texinfo source needs to make.  Yesterday I posted a list of
Texinfo constructs and suggested adding to Org mode ways to
distinguish them.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> In order for that to produce proper Texinfo source as output, there
> would need to be, in Org format, ways to express the many distinctions
> that Texinfo source needs to make.  Yesterday I posted a list of
> Texinfo constructs and suggested adding to Org mode ways to
> distinguish them.

I have just forwarded that email to Org ML.
https://list.orgmode.org/87bkqx4jyg.fsf@localhost/T/#u

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: Org mode and Emacs

[Ihor Radchenko]

Bastien <bzg@gnu.org> writes:

>> I am wondering how many Emacs developers contributed to Org in the past.
>> In particular, before we switched away from .texi manual.
>
> My impression is that these contributions have been small in size of
> text modified but very useful and regular, at least as much useful as
> the occasional ones received from Org casual contributors.

Hmm. I tried (in Emacs git repo), skipping contributions from Org
maintainers:

git shortlog -- doc/misc/org.texi (see below for shortened output)

The overwhelming majority of the commits are (1) typos; (2) texinfo
markup changes (not needed as long as we follow the conventions in
ox-texinfo); (3) copyright year updates; (4) some common wording
conventions, like using "GNU" or https in urls.

This was since 2007.

git shortlog -- doc/misc/org.org is since Feb 2021 (3 years)

Eli Zaretskii (1):
      <copyright>

Glenn Morris (3):
      <moving from texi to org source>
      <copyright>

Hanno Perrey (1):
      ; * doc/misc/org.org: fix capture context example

Stefan Kangas (14):
      <typos>
      Avoid treating number as an enum in the org manual
      Drop support for the dead third-party w3 package
      <merges>
      Recommend NonGNU ELPA over MELPA
      Delete broken link to Network Theory Ltd.
      ; Prefer HTTPS to HTTP in many URLs

Total 19 commits from Emacs contributors.

For the same 3 year period org.texi saw
git shortlog --since 2018 -- doc/misc/org.texi
21 commits.

As expected, we are no longer seeing texinfo markup fixes, except a
single work around a bug in Emacs Info command. All other types of the
contributions from Emacs devs are not stopping.

I conclude that .org version of the manual did not cause reduced
contributions from Emacs devs compared to .texi version. And I believe
that automated export is useful to avoid dealing with texinfo issues in
addition to our own .org issues we would have to address anyway.

-------------------

git shortlog -- doc/misc/org.texi

Alan Mackenzie (1):
      Expunge "allow" + infinitive without direct object from source and doc.

Andreas Schwab (2):
      <2 commits fixing texinfo markup>

Chong Yidong (3):
      * org.texi (Org Plot): Fix tags (Bug#3507).
      * org.texi (Workflow states, Agenda commands): Fix tags (Bug#3508).
      <1 merge>      

Eli Zaretskii (4):
      <3 commits fixing texinfo markup>
      <1 commit fixing typo>

Glenn Morris (57):
      <several commits moving files around Emacs repo>
      Update Back-Cover Text as per maintain.info.
      Change to GFDL 1.2.  Refer to license in Emacs manual.
      <typos>
      <copyright year updates>
      Peter Tury  <tury.peter at gmail.com>  (tiny change)
      Peter Tury  <tury.peter at gmail.com>  (tiny change)
      Nuke arch-tags.
      Standardize possessive apostrophe usage in manuals, docs, and comments Ref: http://lists.gnu.org/archive/html/emacs-devel/2012-02/msg00649.html
      <texinfo markup fixes>
      Standardize case of "Front-Cover Texts" in texi file permissions notices.
      <merges>
      Replace doc references to load-hooks
      Distribute the real source for some doc/misc manuals (bug#45143)

Juanma Barranquero (7):
      <typos>

Karl Berry (1):
      <texinfo markup fixes>

Lars Ingebrigtsen (1):
      <texinfo markup fixes>

Mario Lang (2):
      <typos>

Martin Rudalics (1):
      <texinfo markup>

Mauro Aranda (1):
      <texinfo markup>

Michael Albinus (6):
      Document remote file name syntax change
      <texinfo markup>
      * doc/misc/org.texi (Installation): Fix clone commands.

Miles Bader (2):
      <merges>

Paul Eggert (54):
     <spelling>
     <copyright years>
     <texinfo markup>
      Simplify use of current-time and friends.
      Fix some 24-hour time stamps in documentation.
      Modernize usage of 'macOS' in doc and comments
      Prefer HTTPS to FTP and HTTP in documentation
      <merges>
      Update some URLs

Radon Rosborough (1):
      Add early init file, stop package-initialize insertion

Stefan Kangas (2):
      ; Prefer https to http in more URLs
      Avoid recommending Google

Stefan Monnier (3):
      <merges>
      * src/print.c (syms_of_print) <print_quoted>: Set default to true

Stephen Eglen (1):
      <typos>

Ville Skyttä (1):
      <typos>


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Ihor Radchenko <yantar92@gmail.com>, Tim Cross <theophilusx@gmail.com>,
>  emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 17:09:42 +0000
> 
> I conclude that .org version of the manual did not cause reduced
> contributions from Emacs devs compared to .texi version. And I believe
> that automated export is useful to avoid dealing with texinfo issues in
> addition to our own .org issues we would have to address anyway.

You are comparing negligibly small numbers of contributions, so any
conclusions from such comparison are questionable at best.

My personal conclusion is that Emacs developers don't contribute too
much to Org's documentation because Emacs developers who are used to
work on documentation seldom if ever use Org for any significant work.
So they don't have significant changes to contribute, except fixing
typos and markup.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> You are comparing negligibly small numbers of contributions, so any
> conclusions from such comparison are questionable at best.

> My personal conclusion is that Emacs developers don't contribute too
> much to Org's documentation because Emacs developers who are used to
> work on documentation seldom if ever use Org for any significant work.
> So they don't have significant changes to contribute, except fixing
> typos and markup.

I think that I need to clarify about the purpose of my "analysis" (I
know that data is not very conclusive, but that all we got).

I was looking into the idea that we can get more help from Emacs
contributors if we switch the format back from .org to .texi.
(Mostly, out of curiosity; and because it was hanging in my TODO list)

According to the above, I do not see that we can achieve such goal:

1. Absolute number of contributions to Org manual was never large (not
   by Org-only contrubutors)

2. Relative numbers, although hard to compare, also do not indicate that
   the switch would be helpful in any way.

Having said that, RMS officially asked Org team to work towards allowing
Org to become the new GNU documentation standard.
https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
So, the above idea was rather theoretical. Unless we disregard RMS
request, Org manual will be a good reference template to explore
real-life caveats of using .org as the true documentation source.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 18:49:53 +0000
> 
> I was looking into the idea that we can get more help from Emacs
> contributors if we switch the format back from .org to .texi.

I'd welcome such a change, FWIW.

> Having said that, RMS officially asked Org team to work towards allowing
> Org to become the new GNU documentation standard.
> https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
> So, the above idea was rather theoretical. Unless we disregard RMS
> request, Org manual will be a good reference template to explore
> real-life caveats of using .org as the true documentation source.

I don't know why Richard made that request.  From my POV, switching
from Texinfo to _any_ significantly different markup has the
significant disadvantage that people who routinely improve our manuals
will have to learn a new language, and that will likely lower their
motivation.

For better and for worse, Texinfo is our documentation system.  It is
a good system, and is actively developed (version 7.1 will be released
soon), so abandoning it for something that is less familiar to those
who take care of the documentation is a net loss, IMO.  We have enough
real problems on our hands to afford inventing new ones.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I was looking into the idea that we can get more help from Emacs
>> contributors if we switch the format back from .org to .texi.
>
> I'd welcome such a change, FWIW.

But how would Org benefit from it?
For good reasons or not, we have already switched to .org format.
And switching back now would require extra effort that needs to be justified.
(Even without taking into account RMS' request).

>> Having said that, RMS officially asked Org team to work towards allowing
>> Org to become the new GNU documentation standard.
>> https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/
> ...
> I don't know why Richard made that request.  From my POV, switching
> from Texinfo to _any_ significantly different markup has the
> significant disadvantage that people who routinely improve our manuals
> will have to learn a new language, and that will likely lower their
> motivation.
>
> For better and for worse, Texinfo is our documentation system.  It is
> a good system, and is actively developed (version 7.1 will be released
> soon), so abandoning it for something that is less familiar to those
> who take care of the documentation is a net loss, IMO.  We have enough
> real problems on our hands to afford inventing new ones.

Feel free to reply to RMS' email. I am sure that he had good reasons to
make the official request. I am not in position to argue on his behalf.

If you prefer, the same email is also on emacs-devel:
https://yhetil.org/emacs-devel/E1ocmvz-0002iB-2M@fencepost.gnu.org/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Fri, 18 Aug 2023 19:31:31 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I was looking into the idea that we can get more help from Emacs
> >> contributors if we switch the format back from .org to .texi.
> >
> > I'd welcome such a change, FWIW.
> 
> But how would Org benefit from it?

I explained that below this part in my email.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I'd welcome such a change, FWIW.
>> 
>> But how would Org benefit from it?
>
> I explained that below this part in my email.

I thought that you were referring to RMS' request.

>  I don't know why Richard made that request.  From my POV, switching
>  from Texinfo to _any_ significantly different markup has the
>  significant disadvantage that people who routinely improve our manuals
>  will have to learn a new language, and that will likely lower their
>  motivation.

>  For better and for worse, Texinfo is our documentation system.  It is
>  a good system, and is actively developed (version 7.1 will be released
>  soon), so abandoning it for something that is less familiar to those
>  who take care of the documentation is a net loss, IMO.  We have enough
>  real problems on our hands to afford inventing new ones.

This may be true in general, but not for Org development.
People contributing to the Org manual are expected to understand Org
markup. So, it is Texinfo that becomes the new language to learn and
that creates an extra contribution barrier.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 09:04:38 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > I'd welcome such a change, FWIW.
> >> 
> >> But how would Org benefit from it?
> >
> > I explained that below this part in my email.
> 
> I thought that you were referring to RMS' request.

I was.  But Richard reads this list, so he will see what I wrote and
respond if he wants to.

> >  I don't know why Richard made that request.  From my POV, switching
> >  from Texinfo to _any_ significantly different markup has the
> >  significant disadvantage that people who routinely improve our manuals
> >  will have to learn a new language, and that will likely lower their
> >  motivation.
> 
> >  For better and for worse, Texinfo is our documentation system.  It is
> >  a good system, and is actively developed (version 7.1 will be released
> >  soon), so abandoning it for something that is less familiar to those
> >  who take care of the documentation is a net loss, IMO.  We have enough
> >  real problems on our hands to afford inventing new ones.
> 
> This may be true in general, but not for Org development.
> People contributing to the Org manual are expected to understand Org
> markup. So, it is Texinfo that becomes the new language to learn and
> that creates an extra contribution barrier.

If the idea is that the main contributors to the Org documentation are
Org users, and the rest of Emacs developers don't really count for
this purpose, then why did you count the contributions of the latter
to begin with? they should be immaterial for you, AFAIU.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> This may be true in general, but not for Org development.
>> People contributing to the Org manual are expected to understand Org
>> markup. So, it is Texinfo that becomes the new language to learn and
>> that creates an extra contribution barrier.
>
> If the idea is that the main contributors to the Org documentation are
> Org users, and the rest of Emacs developers don't really count for
> this purpose, then why did you count the contributions of the latter
> to begin with? they should be immaterial for you, AFAIU.

They are not immaterial. The work contributed to Org manual by Emacs
devs, even if small, is certainly appreciated. However, decision to
switch back to .texi is a major one and will affect all the
contributors, not just Emacs devs. Thus, I am trying to weight the
overall impact on the _Org_ development and maintenance.

I looked into git logs specifically to see the contribution pattern of
Emacs developers and whether it was influenced by switching from .texi
to .org. I only found that Texinfo markup-related contributions
disappeared (naturally). Other contributions remained at their levels:
typo fixes and enforcing Emacs-wide terminology and wording standards.

Neither typo fixes nor enforcing certain terminology uses are limited by
the fact that Org manual is not in .texi format. (Correct me if I am
wrong).

On the other hand, people contributing non-trivial patches for Org
manual should naturally be familiar with Org mode. So, using Org markup
should not be a handicap for them, while forcing texinfo may sometimes
be.

That's why, not disregarding Emacs dev contributions, I do not see what
positive change switching back to .texi would bring to Org development. 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> This may be true in general, but not for Org development.
>>> People contributing to the Org manual are expected to understand Org
>>> markup. So, it is Texinfo that becomes the new language to learn and
>>> that creates an extra contribution barrier.
>>
>> If the idea is that the main contributors to the Org documentation are
>> Org users, and the rest of Emacs developers don't really count for
>> this purpose, then why did you count the contributions of the latter
>> to begin with? they should be immaterial for you, AFAIU.
>
> They are not immaterial. The work contributed to Org manual by Emacs
> devs, even if small, is certainly appreciated. However, decision to
> switch back to .texi is a major one and will affect all the
> contributors, not just Emacs devs. Thus, I am trying to weight the
                    ^^^^^^^^^^^^^^^

Is Org not part of Emacs?

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: bzg@gnu.org, yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 09:44:08 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If the idea is that the main contributors to the Org documentation are
> > Org users, and the rest of Emacs developers don't really count for
> > this purpose, then why did you count the contributions of the latter
> > to begin with? they should be immaterial for you, AFAIU.
> 
> They are not immaterial. The work contributed to Org manual by Emacs
> devs, even if small, is certainly appreciated. However, decision to
> switch back to .texi is a major one and will affect all the
> contributors, not just Emacs devs. Thus, I am trying to weight the
> overall impact on the _Org_ development and maintenance.
> 
> I looked into git logs specifically to see the contribution pattern of
> Emacs developers and whether it was influenced by switching from .texi
> to .org. I only found that Texinfo markup-related contributions
> disappeared (naturally). Other contributions remained at their levels:
> typo fixes and enforcing Emacs-wide terminology and wording standards.
> 
> Neither typo fixes nor enforcing certain terminology uses are limited by
> the fact that Org manual is not in .texi format. (Correct me if I am
> wrong).
> 
> On the other hand, people contributing non-trivial patches for Org
> manual should naturally be familiar with Org mode. So, using Org markup
> should not be a handicap for them, while forcing texinfo may sometimes
> be.
> 
> That's why, not disregarding Emacs dev contributions, I do not see what
> positive change switching back to .texi would bring to Org development. 

We are going in circles.  I already explained, or tried to explain,
what would be the positive change.  It's your call, obviously.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bzg@gnu.org,  yantar92@gmail.com,
>   theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 19 Aug 2023 18:19:45 +0800
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
> > They are not immaterial. The work contributed to Org manual by Emacs
> > devs, even if small, is certainly appreciated. However, decision to
> > switch back to .texi is a major one and will affect all the
> > contributors, not just Emacs devs. Thus, I am trying to weight the
>                     ^^^^^^^^^^^^^^^
> 
> Is Org not part of Emacs?

Its development is separate.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>> They are not immaterial. The work contributed to Org manual by Emacs
>> devs, even if small, is certainly appreciated. However, decision to
>> switch back to .texi is a major one and will affect all the
>> contributors, not just Emacs devs. Thus, I am trying to weight the
>                     ^^^^^^^^^^^^^^^
>
> Is Org not part of Emacs?
 
Org is maintained separately from Emacs, but distributed together with
Emacs. 

That said, by "Emacs devs" I mostly meant experienced Elisp contributors
who are not familiar with Org markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> That's why, not disregarding Emacs dev contributions, I do not see what
>> positive change switching back to .texi would bring to Org development. 
>
> We are going in circles.  I already explained, or tried to explain,
> what would be the positive change.  It's your call, obviously.

I understand that. However, I do not fully agree with the arguments you
used to support this statement. And I replied explaining why I do not
fully agree. Let me know if I missed something you wanted to point out.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > >  I don't know why Richard made that request.

I don't remember making that request.  I must have done so, but I
don't know why.

Can someone please show me that message?  Once i see its date,
I can find the pertinent messages in my saved old mail, and
think about this.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > >  I don't know why Richard made that request.
>
> I don't remember making that request.  I must have done so, but I
> don't know why.
>
> Can someone please show me that message?  Once i see its date,
> I can find the pertinent messages in my saved old mail, and
> think about this.

These were the two messages referenced in the thread:

https://yhetil.org/emacs-devel/E1ocmvz-0002iB-2M@fencepost.gnu.org/
https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Having said that, RMS officially asked Org team to work towards allowing
  > Org to become the new GNU documentation standard.
  > https://list.orgmode.org/orgmode/87bkqx4jyg.fsf@localhost/

The words quoted above can be interpreted in multiple ways, so when I
read it the first time, it didn't seem like anything I would have sad.
Now I see it does fit, when interpreted differently.

My suggestion was not that we adopt the current Org format for GNU
documentation.  The current Org format doesn't have all the features
that are needed -- it can't do the job that Texinfo does for us.

But I think the Org format could be _extended_ into a format that
would have all the features we need -- and then, with work, it could
become superior to Texinfo.

That would be an advance overall for documentation, and Org developers
will appreciate that Org plays a role in it.

The idea is based on a factual assumption that I think is true
but it couldn't hurt to check.  As I understand it, Org format is
fits into a family of formats, also including Markup and Sphinx,
which are similar in the basics.  Because of this, I think a lot
of people would find it easier to learn to use Org mode (including
the extensions that will be needed) than Texinfo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> The idea is based on a factual assumption that I think is true
> but it couldn't hurt to check.  As I understand it, Org format is
> fits into a family of formats, also including Markup and Sphinx,
> which are similar in the basics.

At least, Wikipedia classifies Org into "Lightweight Markup" family:
https://en.wikipedia.org/wiki/Lightweight_markup_language

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Ihor Radchenko <yantar92@posteo.net> writes:

    > Richard Stallman <rms@gnu.org> writes:
    >> The idea is based on a factual assumption that I think is true
    >> but it couldn't hurt to check.  As I understand it, Org format is
    >> fits into a family of formats, also including Markup and Sphinx,
    >> which are similar in the basics.

    > At least, Wikipedia classifies Org into "Lightweight Markup"
    > family: https://en.wikipedia.org/wiki/Lightweight_markup_language

"Lightweight Markup" appears to be the author's invention, with no
reference or citation given.

Colin Baxter.

Re: Org mode and Emacs

[Ihor Radchenko]

Colin Baxter <m43cap@yandex.com> writes:

>>>>>> Ihor Radchenko <yantar92@posteo.net> writes:
>
>     > Richard Stallman <rms@gnu.org> writes:
>     >> The idea is based on a factual assumption that I think is true
>     >> but it couldn't hurt to check.  As I understand it, Org format is
>     >> fits into a family of formats, also including Markup and Sphinx,
>     >> which are similar in the basics.
>
>     > At least, Wikipedia classifies Org into "Lightweight Markup"
>     > family: https://en.wikipedia.org/wiki/Lightweight_markup_language
>
> "Lightweight Markup" appears to be the author's invention, with no
> reference or citation given.

Why is it a problem? I have seen this term elsewhere as well.

In any case, the key point is not the terminology, but the fact that
other people independently grouped Org markup with multiple other
markups that are widely used.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Why is it a problem? I have seen this term elsewhere as well.
>
> In any case, the key point is not the terminology, but the fact that
> other people independently grouped Org markup with multiple other
> markups that are widely used.

I may not be the individual best qualified to speak apropos this
subject, but I can't imagine any reason learning Texinfo should be any
more challenging than learning a version of Org extended to sufficiently
address every one of Texinfo's applications.  Annulling any hypothetical
advantage -- should it exist -- is the significant body of GNU
developers who are already proficient in Texinfo, many of whom bear no
interest in Org.

Richard, what makes you believe that?  Can you provide any concrete
examples of Texinfo constructs that would be simpler in Org?

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Ihor Radchenko <yantar92@posteo.net> writes:

    > Colin Baxter <m43cap@yandex.com> writes:
    >>>>>>> Ihor Radchenko <yantar92@posteo.net> writes:
    >> 
    >> > Richard Stallman <rms@gnu.org> writes: >> The idea is based on
    >> a factual assumption that I think is true >> but it couldn't hurt
    >> to check.  As I understand it, Org format is >> fits into a
    >> family of formats, also including Markup and Sphinx, >> which are
    >> similar in the basics.
    >> 
    >> > At least, Wikipedia classifies Org into "Lightweight Markup" >
    >> family: https://en.wikipedia.org/wiki/Lightweight_markup_language
    >> 
    >> "Lightweight Markup" appears to be the author's invention, with
    >> no reference or citation given.

    > Why is it a problem? I have seen this term elsewhere as well.

Without a reference to consult, I do not understand what the author
means by the term "lightweight".

Colin Baxter.

Re: Org mode and Emacs

[Colin Baxter]

>>>>> Po Lu <luangruo@yahoo.com> writes:

    > Ihor Radchenko <yantar92@posteo.net> writes:
    >> Why is it a problem? I have seen this term elsewhere as well.
    >> 
    >> In any case, the key point is not the terminology, but the fact
    >> that other people independently grouped Org markup with multiple
    >> other markups that are widely used.

    > I may not be the individual best qualified to speak apropos this
    > subject, but I can't imagine any reason learning Texinfo should be
    > any more challenging than learning a version of Org extended to
    > sufficiently address every one of Texinfo's applications.
    > Annulling any hypothetical advantage -- should it exist -- is the
    > significant body of GNU developers who are already proficient in
    > Texinfo, many of whom bear no interest in Org.

    > Richard, what makes you believe that?  Can you provide any
    > concrete examples of Texinfo constructs that would be simpler in
    > Org?

Perhaps this ship sailed long ago but would removing org-mode from emacs
core go some way towards making the choice between using texinfo or
org-mode irrelevant? As an external package, org-mode's documentation
could then evolve independently, as its users think best.

Colin Baxter.

Re: Org mode and Emacs

[Ihor Radchenko]

Colin Baxter <m43cap@yandex.com> writes:

>     > Richard, what makes you believe that?  Can you provide any
>     > concrete examples of Texinfo constructs that would be simpler in
>     > Org?
>
> Perhaps this ship sailed long ago but would removing org-mode from emacs
> core go some way towards making the choice between using texinfo or
> org-mode irrelevant? As an external package, org-mode's documentation
> could then evolve independently, as its users think best.

IMHO, the problem with sources of Org manual is not that much of a big
deal. Eli favours all the manuals to be in .texi format because it is
standard. However, the current situation is certainly not _that_
unbearable to the level where we need to consider drastic measures like
removing the whole Org from Emacs. Not even close.

The far more important discussion here is about Org as potential
replacement to Texinfo (after Org is modified to support software
manual-specific markup). I am not an expert in Texinfo, so I cannot
judge if it is a good call or not.

On Texinfo side, there might be problems existing in Texinfo that will
not exist in Org, as I recall from previous discussions. But I cannot
tell any specifics.

On Org side, Org allows fine-tuned configuration of export for each
individual export target. And generating parts of the manuals
programmatically. AFAIK, this is not easily possible in Texinfo.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Tassilo Horn]

Colin Baxter <m43cap@yandex.com> writes:

>     >> "Lightweight Markup" appears to be the author's invention, with
>     >> no reference or citation given.
>
>     > Why is it a problem? I have seen this term elsewhere as well.
>
> Without a reference to consult, I do not understand what the author
> means by the term "lightweight".

I think the common definition of lightweight markup is that it's still
easily readable by humans.  For example, HTML is pretty much unreadable
without a browser rendering it.  In contrast, Org and Markdown can be
easily read as plain text without a special renderer or editor, e.g.,
with just "less".

Bye,
Tassilo

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Po Lu <luangruo@yahoo.com>, rms@gnu.org, eliz@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 17:47:43 +0000
> 
> On Org side, Org allows fine-tuned configuration of export for each
> individual export target. And generating parts of the manuals
> programmatically. AFAIK, this is not easily possible in Texinfo.

Yes, it is.  At least the separate configuration part.  See the
section "Customization Variables" and its sub-sections in the Texinfo
manual.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> On Org side, Org allows fine-tuned configuration of export for each
>> individual export target. And generating parts of the manuals
>> programmatically. AFAIK, this is not easily possible in Texinfo.
>
> Yes, it is.  At least the separate configuration part.  See the
> section "Customization Variables" and its sub-sections in the Texinfo
> manual.

I did not mean variables affecting all the output. I meant more
fine-grained staff, like direct html/latex/odt/... snippets only
embedded when exporting to html/latex/odt/...

For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
https://orgmode.org/manual/Quoting-HTML-tags.html

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: m43cap@yandex.com, luangruo@yahoo.com, rms@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 18:08:42 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> On Org side, Org allows fine-tuned configuration of export for each
> >> individual export target. And generating parts of the manuals
> >> programmatically. AFAIK, this is not easily possible in Texinfo.
> >
> > Yes, it is.  At least the separate configuration part.  See the
> > section "Customization Variables" and its sub-sections in the Texinfo
> > manual.
> 
> I did not mean variables affecting all the output. I meant more
> fine-grained staff, like direct html/latex/odt/... snippets only
> embedded when exporting to html/latex/odt/...
> 
> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
> https://orgmode.org/manual/Quoting-HTML-tags.html

Sorry, I don't understand what that means, even after looking at the
URLs.  "Direct snippets only embedded" I cannot parse, sorry.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I did not mean variables affecting all the output. I meant more
>> fine-grained staff, like direct html/latex/odt/... snippets only
>> embedded when exporting to html/latex/odt/...
>> 
>> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
>> https://orgmode.org/manual/Quoting-HTML-tags.html
>
> Sorry, I don't understand what that means, even after looking at the
> URLs.  "Direct snippets only embedded" I cannot parse, sorry.

For example, you can set a different color, for a single link, just when
exporting to html. Or put a page break at certain place just in odt
export. Or set image size differently in html and latex, for a single
image.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: m43cap@yandex.com, luangruo@yahoo.com, rms@gnu.org, bzg@gnu.org,
>  yantar92@gmail.com, theophilusx@gmail.com, emacs-devel@gnu.org
> Date: Wed, 23 Aug 2023 18:36:36 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I did not mean variables affecting all the output. I meant more
> >> fine-grained staff, like direct html/latex/odt/... snippets only
> >> embedded when exporting to html/latex/odt/...
> >> 
> >> For example, see https://orgmode.org/manual/Quoting-LaTeX-code.html or
> >> https://orgmode.org/manual/Quoting-HTML-tags.html
> >
> > Sorry, I don't understand what that means, even after looking at the
> > URLs.  "Direct snippets only embedded" I cannot parse, sorry.
> 
> For example, you can set a different color, for a single link, just when
> exporting to html. Or put a page break at certain place just in odt
> export. Or set image size differently in html and latex, for a single
> image.

Texinfo has @ifhtml..@end ifhtml blocks, where you can do stuff for
HTML output only.  And similar for all other formats; see "Conditional
Commands" in the Texinfo manual.  There's also @html..@end html, where
you can write raw HTML (and similar for other formats; see "Raw
Formatter Commands" in the Texinfo manual.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> For example, you can set a different color, for a single link, just when
>> exporting to html. Or put a page break at certain place just in odt
>> export. Or set image size differently in html and latex, for a single
>> image.
>
> Texinfo has @ifhtml..@end ifhtml blocks, where you can do stuff for
> HTML output only.  And similar for all other formats; see "Conditional
> Commands" in the Texinfo manual.  There's also @html..@end html, where
> you can write raw HTML (and similar for other formats; see "Raw
> Formatter Commands" in the Texinfo manual.

This is indeed the same functionality.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Bastien Guerry]

FWIW, I think it is good to improve the .org format and Org tools
so that they are as good as .texi and Texinfo tools when it comes
to writing and publishing technical documentation.

Because most of Org syntax is easy to learn, this may help more
developers enjoy writing good documentation.

I don't think we need to consider replacing Texinfo by Org as the
GNU documentat format, though.

I understand such a possibility can be a motivation boost for Org
developers, but it could also be a pitfall, inducing focus on the
wrong priorities.

Let's simply try to improve Org in general, and see if more GNU
maintainers want to use it as their native documentat format (the
example of Org's documentation shows it's already possible.)

2 cts,

-- 
 Bastien Guerry

Re: Org mode and Emacs

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 663 bytes --]

I agree. I think if we find gaps, we should first try to fix this in the
"export org to texi" code, and that will then reveal places where we
indeed need to evolve org as a format. In fact, if we discover that
org-to-texi cant express everything that texi can, we should first
consider adding styling options to org-to-texi flow, and only if that
fails, would we need to add new constructs to org.

Off the top of my head, the only things I can think of that org may not
presently cover is the ability to indicate index entries in the org
source


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Org mode and Emacs

[Ihor Radchenko]

"T.V Raman" <raman@google.com> writes:

> Off the top of my head, the only things I can think of that org may not
> presently cover is the ability to indicate index entries in the org
> source

For texi export, we allow #+cindex/#+findex/#+vindex.
Also, Timothy has been working on
https://github.com/tecosaur/org-glossary to bring it natively into Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[T.V Raman]

this is awesome to know -- glad we're already on it.



Ihor Radchenko writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > Off the top of my head, the only things I can think of that org may not
 > > presently cover is the ability to indicate index entries in the org
 > > source
 > 
 > For texi export, we allow #+cindex/#+findex/#+vindex.
 > Also, Timothy has been working on
 > https://github.com/tecosaur/org-glossary to bring it natively into Org.
 > 
 > -- 
 > Ihor Radchenko // yantar92,
 > Org mode contributor,
 > Learn more about Org mode at <https://orgmode.org/>.
 > Support Org development at <https://liberapay.com/org-mode>,
 > or support my work at <https://liberapay.com/yantar92>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Richard, what makes you believe that?  Can you provide any concrete
  > examples of Texinfo constructs that would be simpler in Org?

That's not what the issue is.

Org format is similar to Markdown and I'd expect that millions of
users use Markdown.  Those millions of people would be willing to
learn all the other details if those details were in a framework
like Markdown.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW, I think it is good to improve the .org format and Org tools
  > so that they are as good as .texi and Texinfo tools when it comes
  > to writing and publishing technical documentation.

The term "as good as" may suggest, incorrectly, that this is a matter
of comparing the two formats over some sense of _quality_.  But that's
not what this is about.  The improvements I've proposed for Org format
are a matter of _supporting the range of necessary constructs_.

Texinfo supports them all because I implemented them all in Texinfo.
As a result, you can express in Texinfo everything that has proved
necessary (so far, at least) in GNU documentation.

At present, that is not the case with Org format.  There are
situations in which Org format provides no way to express what needs
to be said.

  > Let's simply try to improve Org in general, and see if more GNU
  > maintainers want to use it as their native documentat format (the
  > example of Org's documentation shows it's already possible.)

We need to be careful here.  What does the existence of Org mode
documentation written in Org format actually show -- given that the
format doesn't support all the constructs that are needed in general?

It might show that the Org mode documentation doesn't make all the
textual distinctions properly -- that it fails to follow our style
guide.  If so, then it is "possible" but only with flawed output.

But not necesarily.  Perhaps it shows that the Org mode documentation
needs only a limited subset of those constructs, and those are all
implemened in Ogr format.  If so, that could mean that Org format is
fine for the Org mode manual in prticular, but is not adequate for the
whole range of our documentation.

Either way, to make Org format adequate for that whole range of
constructs, in all the output formats, will require working
specifically towards that goal.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Off the top of my head, the only things I can think of that org may not
  > presently cover is the ability to indicate index entries in the org
  > source

How does one express these constructs in Org format?

@code
@samo
@emph
@def
@var
@i
@url
@key
@kbd

  > In fact, if we discover that
  > org-to-texi cant express everything that texi can, we should first
  > consider adding styling options to org-to-texi flow, and only if that
  > fails, would we need to add new constructs to org.

I am skeptical that that would address these issues, but I can't be sure
since I don't know what that does.  Can you please show a hypotherical
example of solvin a problem this way?

For instance, how would you propose to use them to distinguish
between @var, @def and @emph?

They look alike in TeX output.  It is in OTHER formats that they
produce different output.

1. What would that approach look like in the document source?
I can't be sure 
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Bastien Guerry]

Richard Stallman <rms@gnu.org> writes:

> The term "as good as" may suggest, incorrectly, that this is a matter
> of comparing the two formats over some sense of _quality_.  But that's
> not what this is about.  The improvements I've proposed for Org format
> are a matter of _supporting the range of necessary constructs_.

I'm confident we can support the necessary constructs in Org.

>   > Let's simply try to improve Org in general, and see if more GNU
>   > maintainers want to use it as their native documentat format (the
>   > example of Org's documentation shows it's already possible.)
>
> We need to be careful here.  What does the existence of Org mode
> documentation written in Org format actually show -- given that the
> format doesn't support all the constructs that are needed in general?
>
> It might show that the Org mode documentation doesn't make all the
> textual distinctions properly -- that it fails to follow our style
> guide.  If so, then it is "possible" but only with flawed output.

If a .texi expert can report such flaws in the Org manual, we can then
fix them and, if needed, implement the necessary constructs.

> But not necesarily.  Perhaps it shows that the Org mode documentation
> needs only a limited subset of those constructs, and those are all
> implemened in Ogr format.  If so, that could mean that Org format is
> fine for the Org mode manual in prticular, but is not adequate for the
> whole range of our documentation.

I believe this is more plausible.

> Either way, to make Org format adequate for that whole range of
> constructs, in all the output formats, will require working
> specifically towards that goal.

Agreed, and this is what Org maintainers are working on.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Oops!  I wrote

  > @code
  > @samo
  > @emph
  > @def
  > @var
  > @i
  > @url
  > @key
  > @kbd

but it is not @def, it is @dfn.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Philip Kaludercic]

Bastien Guerry <bzg@gnu.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> The term "as good as" may suggest, incorrectly, that this is a matter
>> of comparing the two formats over some sense of _quality_.  But that's
>> not what this is about.  The improvements I've proposed for Org format
>> are a matter of _supporting the range of necessary constructs_.
>
> I'm confident we can support the necessary constructs in Org.
>
>>   > Let's simply try to improve Org in general, and see if more GNU
>>   > maintainers want to use it as their native documentat format (the
>>   > example of Org's documentation shows it's already possible.)
>>
>> We need to be careful here.  What does the existence of Org mode
>> documentation written in Org format actually show -- given that the
>> format doesn't support all the constructs that are needed in general?
>>
>> It might show that the Org mode documentation doesn't make all the
>> textual distinctions properly -- that it fails to follow our style
>> guide.  If so, then it is "possible" but only with flawed output.
>
> If a .texi expert can report such flaws in the Org manual, we can then
> fix them and, if needed, implement the necessary constructs.

I don't know if this is of any use, but the initial manual for Compat
(https://elpa.gnu.org/packages/compat.html) was written using Org and
ox-texinfo and I later switched to writing .texi directly.  This commit
here documents the switch that includes a partial rewrite.

https://git.sr.ht/~pkal/compat/commit/dd48603a136881a5321de4419be95ea873496172

Some things here might be difficult to map, such as the proper usage of
reference macros or the different kinds of markup from (texinfo) Indicating.

>
>> But not necesarily.  Perhaps it shows that the Org mode documentation
>> needs only a limited subset of those constructs, and those are all
>> implemened in Ogr format.  If so, that could mean that Org format is
>> fine for the Org mode manual in prticular, but is not adequate for the
>> whole range of our documentation.
>
> I believe this is more plausible.
>
>> Either way, to make Org format adequate for that whole range of
>> constructs, in all the output formats, will require working
>> specifically towards that goal.
>
> Agreed, and this is what Org maintainers are working on.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It might show that the Org mode documentation doesn't make all the
  > > textual distinctions properly -- that it fails to follow our style
  > > guide.  If so, then it is "possible" but only with flawed output.

  > If a .texi expert can report such flaws in the Org manual, we can then
  > fix them and, if needed, implement the necessary constructs.

Please look at the list I sent yesterday.  (See below.)  Can Org mode
handle each of them, and generate from it the correct output for each
output format?

  > @code
  > @samo
  > @emph
  > @dfn
  > @var
  > @i
  > @url
  > @key
  > @kbd

If it can, the next thing to do is to look through
the node in the Texinfo manual which documents these,
and see what other constructs are in that node,
and see if Org mode has equivalent constructs for
all of those Texinfo constructs.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 25 Aug 2023 22:04:13 -0400
> 
> Please look at the list I sent yesterday.  (See below.)  Can Org mode
> handle each of them, and generate from it the correct output for each
> output format?
> 
>   > @code
>   > @samo
>   > @emph
>   > @dfn
>   > @var
>   > @i
>   > @url
>   > @key
>   > @kbd

I think it can, but a typical Texinfo manual these days uses many more
commands, not just those.  So this list is insufficient to judge these
issues.  The actual list is much longer.

Re: Org mode and Emacs

[Ihor Radchenko]

Philip Kaludercic <philipk@posteo.net> writes:

>> If a .texi expert can report such flaws in the Org manual, we can then
>> fix them and, if needed, implement the necessary constructs.
>
> I don't know if this is of any use, but the initial manual for Compat
> (https://elpa.gnu.org/packages/compat.html) was written using Org and
> ox-texinfo and I later switched to writing .texi directly.  This commit
> here documents the switch that includes a partial rewrite.
>
> https://git.sr.ht/~pkal/compat/commit/dd48603a136881a5321de4419be95ea873496172
>
> Some things here might be difficult to map, such as the proper usage of
> reference macros or the different kinds of markup from (texinfo) Indicating.

ox-texinfo certainly misses some specialalized Texinfo syntax, like
@defn, @var, etc:

doc/Documentation_Standards.org:

   + Texinfo commands such as @var and @defoption are not used.  The
     preference for this type of thing is that the user browses the
     customize groups.  If you want or need to refer to, say, a
     variable then document it as "the variable
     @code{org-startup-folded}"

It is not export backend fault per se - Org markup simply does not
define specialized markups more granular than ~code~.
So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
direct texinfo export snippet @@texinfo:@kbd{@@C-c SPC@@texinfo:}@@

Though we do not provide similar macros for @var/@env/etc. We might.
Or we might allow custom inline special markup as I suggested in
https://orgmode.org/list/87bkqx4jyg.fsf@localhost

Another limitation, I can see in the commit is using @xref, which Org
does not do. ox-texinfo now transforms <info:#node> links into @ref
only.

On the other hand, Org provides a powerful citation syntax
[cite:see @citation-key], which may be also utilized if we support
info files as bibliography source. This will be a superset of what
@xref/@ref/@pxref does as cite/variant is an equivalent of \autocite in
LaTeX.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Jose E. Marchesi]

>> From: Richard Stallman <rms@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 25 Aug 2023 22:04:13 -0400
>> 
>> Please look at the list I sent yesterday.  (See below.)  Can Org mode
>> handle each of them, and generate from it the correct output for each
>> output format?
>> 
>>   > @code
>>   > @samo
>>   > @emph
>>   > @dfn
>>   > @var
>>   > @i
>>   > @url
>>   > @key
>>   > @kbd
>
> I think it can, but a typical Texinfo manual these days uses many more
> commands, not just those.  So this list is insufficient to judge these
> issues.  The actual list is much longer.

I think the point here is not just these particular marks, or some other
particular set.  It is that "markdown-style" formats, like org-mode, are
not very good supporting "inline" marking.

For inline marks, Texinfo uses a conventient generic and extensible
form: @NAME{CONTENT}.

"markdown-style" formats, on the contrary, generally support a closed
set of marks that use delimited based ad-hoc syntax, such as *foo*,
=foo=, "foo", `foo' and so on.

Block marks are usually supported well enough by these formats.  In the
case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

Re: Org mode and Emacs

[Ihor Radchenko]

"Jose E. Marchesi" <jemarch@gnu.org> writes:

> I think the point here is not just these particular marks, or some other
> particular set.  It is that "markdown-style" formats, like org-mode, are
> not very good supporting "inline" marking.
>
> For inline marks, Texinfo uses a conventient generic and extensible
> form: @NAME{CONTENT}.

That's what we plan to add to Org. Not just because of this discussion;
there are other reasons why we want such custom inline markup.

> Block marks are usually supported well enough by these formats.  In the
> case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

Yup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Philip Kaludercic]

Ihor Radchenko <yantar92@posteo.net> writes:

> "Jose E. Marchesi" <jemarch@gnu.org> writes:
>
>> I think the point here is not just these particular marks, or some other
>> particular set.  It is that "markdown-style" formats, like org-mode, are
>> not very good supporting "inline" marking.
>>
>> For inline marks, Texinfo uses a conventient generic and extensible
>> form: @NAME{CONTENT}.
>
> That's what we plan to add to Org. Not just because of this discussion;
> there are other reasons why we want such custom inline markup.

Do you have a reference where I could read up on how this is going?

>> Block marks are usually supported well enough by these formats.  In the
>> case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
>
> Yup.

Re: Org mode and Emacs

[Ihor Radchenko]

Philip Kaludercic <philipk@posteo.net> writes:

>>> For inline marks, Texinfo uses a conventient generic and extensible
>>> form: @NAME{CONTENT}.
>>
>> That's what we plan to add to Org. Not just because of this discussion;
>> there are other reasons why we want such custom inline markup.
>
> Do you have a reference where I could read up on how this is going?

https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Block marks are usually supported well enough by these formats.  In the
  > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.

That syntax would do the job, but they are more clumsy than the syntax
used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
Org mode syntax to be a step up, it should look as clean as Texinfo.

Might it be posisble to simplify the syntax that Org would use for
this?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For inline marks, Texinfo uses a conventient generic and extensible
  > form: @NAME{CONTENT}.

  > "markdown-style" formats, on the contrary, generally support a closed
  > set of marks that use delimited based ad-hoc syntax, such as *foo*,
  > =foo=, "foo", `foo' and so on.

The challenge is to extend Org format with an open-ended syntax that 
could handle many different inline markup opertions.

ISTR that Sphinx has an extension which is good for such markup.
Could that same syntax fit into Org mode too?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think it can, but a typical Texinfo manual these days uses many more
  > commands, not just those.  So this list is insufficient to judge these
  > issues.  The actual list is much longer.

You're right.  But I think that if Org can conveniently handle the
ones I listed, giving each its Texinfo behavior in each output format,
that means it has the capacity to handle any additional inline markup
features too.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > > Block marks are usually supported well enough by these formats.  In the
>   > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
>
> That syntax would do the job, but they are more clumsy than the syntax
> used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
> Org mode syntax to be a step up, it should look as clean as Texinfo.
>
> Might it be posisble to simplify the syntax that Org would use for
> this?

I am not convinced that it is necessary.
Doing so will either be backward-incompatible or introduce more syntax
constructs to Org, making the syntax more complicated.

And I do not find the current syntax "clumsy".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> ISTR that Sphinx has an extension which is good for such markup.
> Could that same syntax fit into Org mode too?

May you provide some link that describes the extension you are referring
to?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > ISTR that Sphinx has an extension which is good for such markup.
  > > Could that same syntax fit into Org mode too?

  > May you provide some link that describes the extension you are referring
  > to?

Sorry, I may have said that in an unclear way.  It is not an extension
in the sense of an add-on to Sphinx that you need to install
separately.  It is a standard part of Sphinx and described in the
manual,

It is an extension in the sense that they added it to the simple markup
language that they started with.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > > Block marks are usually supported well enough by these formats.  In the
  > >   > > case of org it would be #+BEGIN_WHATEVER end #+END_WHATEVER I suppose.
  > >
  > > That syntax would do the job, but they are more clumsy than the syntax
  > > used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
  > > Org mode syntax to be a step up, it should look as clean as Texinfo.
  > >
  > > Might it be posisble to simplify the syntax that Org would use for
  > > this?

  > I am not convinced that it is necessary.

I won't claim it is _necessary_, but it would sure make these constructs
more convenient.

  > Doing so will either be backward-incompatible or introduce more syntax
  > constructs to Org, making the syntax more complicated.

Maybe it is ok for it to be backward-incompatible.  I don't know for
certain, but it looks like these constructs were recently added.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > May you provide some link that describes the extension you are referring
>   > to?
>
> Sorry, I may have said that in an unclear way.  It is not an extension
> in the sense of an add-on to Sphinx that you need to install
> separately.  It is a standard part of Sphinx and described in the
> manual,
>
> It is an extension in the sense that they added it to the simple markup
> language that they started with.

Sorry, but I can only see a closed list of inline emphasis constructs in
https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

I may be misunderstanding you.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Yuri Khan]

On Mon, 28 Aug 2023 at 17:05, Ihor Radchenko <yantar92@posteo.net> wrote:

> Sorry, but I can only see a closed list of inline emphasis constructs in
> https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

The `interpreted text`:role: construct is extensible, by defining new
role codes.

Re: Org mode and Emacs

[Ihor Radchenko]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Mon, 28 Aug 2023 at 17:05, Ihor Radchenko <yantar92@posteo.net> wrote:
>
>> Sorry, but I can only see a closed list of inline emphasis constructs in
>> https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup
>
> The `interpreted text`:role: construct is extensible, by defining new
> role codes.

Thanks! It is similar to what we already plan for:
https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

Except that we also plan to allow assigning optional attributes to the
custom markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > > That syntax would do the job, but they are more clumsy than the syntax
>   > > used by Texinfo:  @WHATEVER and @end WHATEVER.  In order for a modified
>   > > Org mode syntax to be a step up, it should look as clean as Texinfo.
>   > >
>   > > Might it be posisble to simplify the syntax that Org would use for
>   > > this?
>
>   > I am not convinced that it is necessary.
>
> I won't claim it is _necessary_, but it would sure make these constructs
> more convenient.

Maybe. But also more confusing. In org, we can have

#+begin_src latex
...
#+end_src latex

or

#+begin_export latex
...
#+end_export

or

#+begin_example latex
...
#+end_example

or

#+begin: name
...
#+end:

If we add simplified syntax like

#+latex
...
#+end latex

or, less consistently with the rest of Org

@latex
...
@end latex

it is actually not very intuitive that such construct implies export,
but not something else.

In addition, we already have

#+latex: latex one-liner

as a simple form, limited to a single line that avoids

#+begin_export latex
latex one-liner
#+end_export

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> That's what we plan to add to Org. Not just because of this
> discussion;

Why?  Where can I find this discussion?

If the main reasons for supporting inline markup like @NAME{CONTENT}
(or similar) are the current fontification shortcomings, I'm skeptic.

The .org falls in the category of lightweight markup formats and I
believe it should stick close to this definition.

That is, natively support *bold*, /emphasis/, etc. without adding
heavy more constructs like @NAME{CONTENT} -- even optionnally.

If we need to support @defn{FUNCTION} for .texi export, what about
considering ~~FUNCTION~~ ?  Using two chars at the beg and end is a
natural extension of the current markup syntax and should spare us
some ambiguity that we have with one-char constructs.

> there are other reasons why we want such custom inline markup.

Thanks for pointing at these discussions.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/

OK, forget my previous request, I'll read from here.

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Bastien Guerry]

Richard Stallman <rms@gnu.org> writes:

>   > "markdown-style" formats, on the contrary, generally support a closed
>   > set of marks that use delimited based ad-hoc syntax, such as *foo*,
>   > =foo=, "foo", `foo' and so on.
>
> The challenge is to extend Org format with an open-ended syntax that 
> could handle many different inline markup opertions.

Yes, that's the idea behind using one-char constructs for basic inline
markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

-- 
 Bastien Guerry

Re: Org mode and Emacs

[Po Lu]

Bastien Guerry <bzg@gnu.org> writes:

> Yes, that's the idea behind using one-char constructs for basic inline
> markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
> specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

I forsee a problem with this approach: users will be compelled to study
and memorize dozens of two character combinations that closely resemble
one another.  Contrast this to Texinfo, where:

  The key sequence @kbd{C-x C-f}
  The string @samp{"sample text"}
  The function @code{defun}
  The formula @var{X}
  An @dfn{example}

leaves no latitude for visual ambiguity, whereas it will not take much
for one to mistake =~ for -~ or ~~.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It is not export backend fault per se - Org markup simply does not
  > define specialized markups more granular than ~code~.
  > So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
  > direct texinfo export snippet ...

That source construct might be acceptable, though it has 6 extra characters
and that would be rather inconvenient.  Could you manage to reduce the number
of them?

Could you explain what is implied/presumed by "direct texinfo export"?
Are you saying that Org would be a front-end for Texinfo, for all
output formats?  I'd hope it could generate Info files and HTML directly.

  > Though we do not provide similar macros for @var/@env/etc. We might.

They are very often used.  To make org adequate as a replacement for
Texinfo, they would have to be predefined.  (It is ok if the file
has to specify loading some extension to define them.)

  > Or we might allow custom inline special markup as I suggested in
  > https://orgmode.org/list/87bkqx4jyg.fsf@localhost

I skimmed that long message but I couldn't see any suggestion about
@var, @env, etc.  Could you please send me the actual proposal, by
itself, rather than an indirection to it?

  > On the other hand, Org provides a powerful citation syntax
  > [cite:see @citation-key], which may be also utilized if we support
  > info files as bibliography source. This will be a superset of what
  > @xref/@ref/@pxref does as cite/variant is an equivalent of \autocite in
  > LaTeX.

If that is smarter than the current Texinfo commands and can replace
them with no loss of flexibility and capability, that would be fine.
Provided it handles all output formats correctly.



-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Sorry, but I can only see a closed list of inline emphasis constructs in
  > https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup

  > I may be misunderstanding you.

I don't remember it any more precisely, so I can't tell you more.

I notice that you're talking about restructuredtext and I am talking
about Sphinx.  Are the two related?  I don't remember.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > It is not export backend fault per se - Org markup simply does not
>   > define specialized markups more granular than ~code~.
>   > So, users have to use macros like {{{kbd(C-c SPC)}}} that expands to
>   > direct texinfo export snippet ...
>
> That source construct might be acceptable, though it has 6 extra characters
> and that would be rather inconvenient.  Could you manage to reduce the number
> of them?

> Could you explain what is implied/presumed by "direct texinfo export"?
> Are you saying that Org would be a front-end for Texinfo, for all
> output formats?  I'd hope it could generate Info files and HTML directly.

To clarify, I was describing what already exists in Org. We leverage
Texinfo itself as intermediate format to export to Info. The above "kbd"
macro is specifically designed for Texinfo export.

We may eventually have an option to export to Info directly, but not
yet. Though nothing stops people from implementing such functionality.
Adding new export Backends is not very hard:
https://orgmode.org/worg/dev/org-export-reference.html#back-end

>   > Though we do not provide similar macros for @var/@env/etc. We might.
>
> They are very often used.  To make org adequate as a replacement for
> Texinfo, they would have to be predefined.  (It is ok if the file
> has to specify loading some extension to define them.)
>
>   > Or we might allow custom inline special markup as I suggested in
>   > https://orgmode.org/list/87bkqx4jyg.fsf@localhost
>
> I skimmed that long message but I couldn't see any suggestion about
> @var, @env, etc.  Could you please send me the actual proposal, by
> itself, rather than an indirection to it?

For more complete support of software manual-specific markup, we plan to
introduce more versatile, hackable markup that can be extended by users
to fit their needs. See
https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/ and
https://list.orgmode.org/orgmode/87mtaez8do.fsf@localhost/ 

One of the ideas was to follow TeXinfo markup closely, extending with
some Org-specific needs:

Simple case:
@name{<contents>}

If we need to escape { or } inside:
@name{{<contents}}

Extra configuration:
@name[:key value ...]{<contents>}

@color[:name red]{[[file:image.png][description]]}

Then, users or packages can define the details of how to fontify or
export a given @name{...} markup. That way, we can add anything,
including @var/@env/...

We already do something similar for links:
https://orgmode.org/manual/Adding-Hyperlink-Types.html (org-man-export),
except that links have limitations that prevent them from being used as
markup.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > Sorry, but I can only see a closed list of inline emphasis constructs in
>   > https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#inline-markup
>
>   > I may be misunderstanding you.
>
> I don't remember it any more precisely, so I can't tell you more.
>
> I notice that you're talking about restructuredtext and I am talking
> about Sphinx.  Are the two related?  I don't remember.

When I looked up Sphinx, the documentation directed me to
reStructuredtext for markup description:

https://www.sphinx-doc.org/en/master/
    Sphinx uses the reStructuredText markup language by default, and can
    read MyST markdown via third-party extensions. 

Also, Yuri pointed me to

>> The `interpreted text`:role: construct is extensible, by defining new
>> role codes.

which is similar to what we planned for Org: @role{interpreted text}

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > or, less consistently with the rest of Org

  > @latex
  > ...
  > @end latex

  > it is actually not very intuitive that such construct implies export,
  > but not something else.

I'd like to understand this issue.  Could you please explain what
"export" means here?  And what is the "something else", that it might
have been?  It appears to be a distinction that doesn't exist in
Texinfo.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > @latex
>   > ...
>   > @end latex
>
>   > it is actually not very intuitive that such construct implies export,
>   > but not something else.
>
> I'd like to understand this issue.  Could you please explain what
> "export" means here?  And what is the "something else", that it might
> have been?  It appears to be a distinction that doesn't exist in
> Texinfo.

You are right, Texinfo has a very narrow focus - authoring manuals in
multiple output formats. Org does a lot more.

-----

One feature of Org we are discussing here is "exporting":
https://orgmode.org/manual/Exporting.html - an ability to convert Org
documents into other textual formats.
Similar to Texinfo's raw formatter commands
(https://www.gnu.org/software/texinfo/manual/texinfo/html_node/Raw-Formatter-Commands.html),
Org provides a special markup that specifies text to be placed into
specific tex/html/odt/etc export output:

#+begin_export html
...
#+end_export
or
#+HTML: <single line>
or
@@html:<inline html>@@

See https://orgmode.org/manual/Quoting-HTML-tags.html, for example.

-----

However, Org has more features, served by other Org's markup.

-----

One of the popular features is implementing literal programming (babel).
https://orgmode.org/manual/Features-Overview.html:

    Org can manage the source code in the block delimited by
    ‘#+BEGIN_SRC’ … ‘#+END_SRC’ in several ways that can simplify
    housekeeping tasks essential to modern source code maintenance. Org
    can edit, format, extract, export, and publish source code blocks.

#+begin_src emacs-lisp :results value
  (+ 1 2)
#+end_src

#+RESULTS[b6b278abd782307b5eca9b1b761906bae5292bcc]:
: 3

Users can execute arbitrary pieces of source code in arbitrary
programming languages, get the results inline and even mix input/output
of code written in different languages:

#+name: bash-input
#+begin_src bash :results output table :var file="file.csv"
cat $file
#+end_src

#+begin_src elisp :var table=bash-input(file="table-input.csv")
(mapcar (lambda (line) (format "%S" line)) table)
#+end_src

The code evaluation results can even be used to generate exported text.
Here is a slightly edited snippet from Org manual that generates a table
of all export customizations:

#+begin_src emacs-lisp :exports results :results table :eval yes
(require 'ox)
(let ((alist org-export-options-alist))
 (let (result)
  (dolist (spec alist)
    (when (and (nth 3 spec) (symbolp (nth 3 spec))) ; has customization
      (push (list (format "~%s~" (car spec)) (format "~%s~" (nth 3 spec))) result)))
  (nreverse result)))
#+end_src

You can see the output in https://orgmode.org/manual/Publishing-options.html

Furthermore, users can "tangle" code from the Org document, extracting
only the appropriate source code pieces into actual code-only files.
https://orgmode.org/manual/Extracting-Source-Code.html
So, one can keep documentation and notes (with markup!) together with
the actual code, tangle the org file to produce the program, or export
the org file to produce readable notes - full literal programming
implementation.

-----

Apart from snippets of source code, Org has a markup for non-code
literal text that is not a subject of Org markup:

#+begin_example
  Some example from a text file.
#+end_example

Some people even prefer to keep snippets of source code as example
blocks as well (when a sole purpose of such source block is exporting):

#+begin_example bash
  ls $HOME
#+end_example

-----

Finally, there are dynamic blocks that can be used to generate parts of
Org file programatically from Elisp: https://orgmode.org/manual/Dynamic-Blocks.html

#+begin: blockname <parameters>
<automatically generated text>
#+end:

One example of built-in dynamic block is clocktable that is used by
people that record their work time using Org mode
(https://orgmode.org/manual/Clocking-Work-Time.html,
https://orgmode.org/manual/The-clock-table.html,
https://olddeuteronomy.github.io/post/org-export-clocktables/)

-----

To conclude, the common pattern for Org's block markup is

#+begin<something>
...
#+end<something>

What you proposed with @latex: ... @end is similar to dropping
<something> part. However, as you see in the above, there is a variety
of different block markups serving for entirely different purposes and
just saying @latex (or rather #+latex, if we follow the common Org
block convention) does make it intuitive whether the block is "export",
or "example" or "src" or dynamic. Furthermore, #+latex is already used
for export one-liners.

In theory, we might drop "begin" parts and get

#+src elisp
...
#+end

#+example
...
#+end

#+export html
#+end

but this is akin converting Pascal's begin/end into C's {/} - possible,
but an alternative convention is already established.

And, generally, we, in Org, try to refrain from changing syntax too much
- a number of external projects are relying upon the existing syntax.
So, making significant syntax changes should not be taken lightly.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The .org falls in the category of lightweight markup formats and I
  > believe it should stick close to this definition.

If people keep this unchanged, Org mode will remain useful for the
tasks it can do today.  But it won't be able to do the job for which
we now use Texinfo.

There are many different Texinfo markup constructs, so to do the job
of Texinfo, Org mode would need to be able to distinguish them all.

So the basic question is: to extend Org mode to do that job, or not?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > The .org falls in the category of lightweight markup formats and I
>   > believe it should stick close to this definition.
>
> If people keep this unchanged, Org mode will remain useful for the
> tasks it can do today.  But it won't be able to do the job for which
> we now use Texinfo.
> There are many different Texinfo markup constructs, so to do the job
> of Texinfo, Org mode would need to be able to distinguish them all.
>
> So the basic question is: to extend Org mode to do that job, or not?

I don't think that we need to compromise here. We can keep Org
lightweight yet allowing arbitrary markup.

We already have arbitrary environment markup (special blocks):

#+begin_<block type>
...
#+end_<block type>

like

#+begin_center
#+begin_abstract
#+begin_myweirdlatexenvironment
etc

These are all valid Org markups, but only a small subset of them has
special handling by Org. Other free-form special blocks are simply
ignored (or handled generically), for example, on export, unless the
specific export backend implements special handling for a given block
type.

Supporting TeXinfo-specific inline markup can be done using the same
approach, without complicating core Org markup specifications.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Alfred M. Szmidt]

     > The .org falls in the category of lightweight markup formats and I
     > believe it should stick close to this definition.

   If people keep this unchanged, Org mode will remain useful for the
   tasks it can do today.  But it won't be able to do the job for which
   we now use Texinfo.

   There are many different Texinfo markup constructs, so to do the job
   of Texinfo, Org mode would need to be able to distinguish them all.

   So the basic question is: to extend Org mode to do that job, or not?

It would make more sense to extend Texinfo to allow for org-mode like
markup (or lighter markup), than try to stuff Texinfo into org-mode.
Texinfo is despite its issues, very nice and easy to write.

Handling the version chasm between existing Texinfo and "new" Texinfo
can easily be solved by having a small @texinfo-version thing at the
top.

But replacing Texinfo outright with some org-mode variant seems like yak
shaving.  The big issue with Texinfo is that it is hard to make your own
complicated constructs (like @defn ..), and the lack of macro support of
any kind.

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > One feature of Org we are discussing here is "exporting":
  > https://orgmode.org/manual/Exporting.html - an ability to convert Org
  > documents into other textual formats.

I think I understand what you mean by exporting.  Maybe I understand
what you have in mind for "exporting" Org documents as Texinfo so as
to make GNU manuals.

But I think I see a disappointing difference between the two.  Texinfo
for manuals is semantic markup, but the proposed use of Org mode for
manuals is not.

With Texinfo, you specify what the manual contents should be, and in
each format you get that contents as output.  See what I mean?

I was hoping to see that same smoothness with Org mode -- that you
specify the contents and Org mode owuld give youa manual with that
contents.  But instead, it seems that Org mode for a manual would
specify Texinfo text to "export", and Org would generate Texinfo source
to export which Texinfo could then convert convert into manuals as output.

This exporting might _work_, but it not be as good, overall, as
Texinfo.  It would be disappointing.

Could it be it possible to make Org mode generate the manual output
formats using contructs that are semantic in meaning, that specify
semantic markup for the contents of the manual -- and have Org mode
generate the output formats to represent the semantics of the manual
contents?  That would be a good replacement for Texinfo.

I think this from Bastien Guerry

    Yes, that's the idea behind using one-char constructs for basic inline
    markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
    specific inline markup (eg ~~defn~~, =~defvar~=, etc.).

Is a sort of thing that might be smooth in this sense.  It would specify
semantics of the contents, rather than "what to export".

It might bot be the best way (as Po Lu says), but at least it is
semantic markup.

  > For more complete support of software manual-specific markup, we plan to
  > introduce more versatile, hackable markup that can be extended by users
  > to fit their needs. See
  > https://list.orgmode.org/orgmode/87a6b8pbhg.fsf@posteo.net/ and
  > https://list.orgmode.org/orgmode/87mtaez8do.fsf@localhost/

That coukd be the sort of thing that would be as smooth so Texinfo
format.

  > One of the ideas was to follow TeXinfo markup closely, extending with
  > some Org-specific needs:

  > Simple case:
  > @name{<contents>}

  > If we need to escape { or } inside:
  > @name{{<contents}}

  > Extra configuration:
  > @name[:key value ...]{<contents>}

This describes a syntax, but not the semantics.

For

  > Simple case:
  > @name{<contents>}

what's an example of what _name_ might be, and what would it mean?





-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I notice that you're talking about restructuredtext and I am talking
  > > about Sphinx.  Are the two related?  I don't remember.

  > When I looked up Sphinx, the documentation directed me to
  > reStructuredtext for markup description:

I guess you must be right.  Thanks.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Juergen Fenn]

Am 04.09.23 um 03:32 Uhr schrieb Richard Stallman:
> But I think I see a disappointing difference between the two. Texinfo
> for manuals is semantic markup, but the proposed use of Org mode for
> manuals is not. With Texinfo, you specify what the manual contents
> should be, and in each format you get that contents as output. See what
> I mean?


This is an interesting point indeed. I wonder, too, whether we could add
semantic markup to Org? It's not exactly what lightweight markup
languages such as Markdown or Org are meant to be. But we would not
necessarily need this in core, a semantic add-on would probably be
enough. We would need an interface that allows for more or less freely
defining semantic markup to Org text. A semantic org mode. Hm.

Best regards,
Jürgen.

Re: Org mode and Emacs

[Ihor Radchenko]

Juergen Fenn <jfenn@gmx.net> writes:

> Am 04.09.23 um 03:32 Uhr schrieb Richard Stallman:
>> But I think I see a disappointing difference between the two. Texinfo
>> for manuals is semantic markup, but the proposed use of Org mode for
>> manuals is not. With Texinfo, you specify what the manual contents
>> should be, and in each format you get that contents as output. See what
>> I mean?
>
>
> This is an interesting point indeed. I wonder, too, whether we could add
> semantic markup to Org? It's not exactly what lightweight markup
> languages such as Markdown or Org are meant to be. But we would not
> necessarily need this in core, a semantic add-on would probably be
> enough. We would need an interface that allows for more or less freely
> defining semantic markup to Org text. A semantic org mode. Hm.

This is exactly what I have in mind.
Similar idea is implemented in https://github.com/alhassy/org-special-block-extras

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is an interesting point indeed. I wonder, too, whether we could add
  > semantic markup to Org? It's not exactly what lightweight markup
  > languages such as Markdown or Org are meant to be. But we would not
  > necessarily need this in core, a semantic add-on would probably be
  > enough.

Implementing it in an extension would be ok -- it could still do the
job that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

> I was hoping to see that same smoothness with Org mode -- that you
> specify the contents and Org mode owuld give youa manual with that
> contents.  But instead, it seems that Org mode for a manual would
> specify Texinfo text to "export", and Org would generate Texinfo source
> to export which Texinfo could then convert convert into manuals as output.

I think I need to clarify. This is how export to info works _now_.
However, nothing stops us from implementing direct export org->info
without intermediate org->texi step. Someone just needs to do this job.

> Could it be it possible to make Org mode generate the manual output
> formats using contructs that are semantic in meaning, that specify
> semantic markup for the contents of the manual -- and have Org mode
> generate the output formats to represent the semantics of the manual
> contents?  That would be a good replacement for Texinfo.

> I think this from Bastien Guerry
>
>     Yes, that's the idea behind using one-char constructs for basic inline
>     markup (eg ~code~, /emphasis/, etc.) and two-chars constructs for more
>     specific inline markup (eg ~~defn~~, =~defvar~=, etc.).
>
> Is a sort of thing that might be smooth in this sense.  It would specify
> semantics of the contents, rather than "what to export".
>
> It might bot be the best way (as Po Lu says), but at least it is
> semantic markup.

Sorry, but I am lost here.

Currently, Org mode does not support all the markup supported by
texinfo. But we will change that.

Meanwhile, Org implements info/manual export by ad-hoc extension
allowing to add texinfo-specific markup. This is not ideal, and it does
not need to be done after we do add the missing markup (Org equivalents
of @kbd, @var, and other)

>   > One of the ideas was to follow TeXinfo markup closely, extending with
>   > some Org-specific needs:
>
>   > Simple case:
>   > @name{<contents>}
>
>   > If we need to escape { or } inside:
>   > @name{{<contents}}
>
>   > Extra configuration:
>   > @name[:key value ...]{<contents>}
>
> This describes a syntax, but not the semantics.
>
> For
>
>   > Simple case:
>   > @name{<contents>}
>
> what's an example of what _name_ might be, and what would it mean?

For example, @kbd{C-c C-x C-c} or @var{org-export-backends}.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 11:29:02 +0000
> 
> Richard Stallman <rms@gnu.org> writes:
> 
> > I was hoping to see that same smoothness with Org mode -- that you
> > specify the contents and Org mode owuld give youa manual with that
> > contents.  But instead, it seems that Org mode for a manual would
> > specify Texinfo text to "export", and Org would generate Texinfo source
> > to export which Texinfo could then convert convert into manuals as output.
> 
> I think I need to clarify. This is how export to info works _now_.
> However, nothing stops us from implementing direct export org->info
> without intermediate org->texi step. Someone just needs to do this job.

Unless the implementation will somehow invoke texi2any under the hood,
this would mean a separate implementation of texi2any in Emacs Lisp,
something that I don't recommend: it will need to play a constant
catchup game with the Texinfo development, and it will be probably
quite slow.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think I need to clarify. This is how export to info works _now_.
>> However, nothing stops us from implementing direct export org->info
>> without intermediate org->texi step. Someone just needs to do this job.
>
> Unless the implementation will somehow invoke texi2any under the hood,
> this would mean a separate implementation of texi2any in Emacs Lisp,

Yes. We already have org2pdf, org2ascii, org2html, but not org2info.

> something that I don't recommend: it will need to play a constant
> catchup game with the Texinfo development, and it will be probably
> quite slow.

I am not sure about catchup game. Unless Texinfo drastically changes the
output format, we can simply provide default settings to replicate the
Texinfo look of the produced html/pdf/plaintext/info files.

As for being slow, it is already not the case. Last time we discussed
Org export being slow, I have managed to speed things up on par with
Texinfo timings on Org manual.

In any case, the first step is extending Org markup to be more suitable
for software manuals. Whether we use texinfo under the hood or not might
be discussed afterwards.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> As for being slow, it is already not the case. Last time we discussed
> Org export being slow, I have managed to speed things up on par with
> Texinfo timings on Org manual.

Did you use Texinfo 4.13 as a baseline for the comparison, or a more
recent release written in Perl?

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> As for being slow, it is already not the case. Last time we discussed
>> Org export being slow, I have managed to speed things up on par with
>> Texinfo timings on Org manual.
>
> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
> recent release written in Perl?

More recent release.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Ihor Radchenko <yantar92@posteo.net> writes:
>>
>>> As for being slow, it is already not the case. Last time we discussed
>>> Org export being slow, I have managed to speed things up on par with
>>> Texinfo timings on Org manual.
>>
>> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
>> recent release written in Perl?
>
> More recent release.

makeinfo 7.0.2 requires approximately 18x more time to generate
emacs.info than 4.13, so an accurate performance comparison cannot be
made with the Perl versions as a basis.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 12:43:55 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I think I need to clarify. This is how export to info works _now_.
> >> However, nothing stops us from implementing direct export org->info
> >> without intermediate org->texi step. Someone just needs to do this job.
> >
> > Unless the implementation will somehow invoke texi2any under the hood,
> > this would mean a separate implementation of texi2any in Emacs Lisp,
> 
> Yes. We already have org2pdf, org2ascii, org2html, but not org2info.
> 
> > something that I don't recommend: it will need to play a constant
> > catchup game with the Texinfo development, and it will be probably
> > quite slow.
> 
> I am not sure about catchup game. Unless Texinfo drastically changes the
> output format, we can simply provide default settings to replicate the
> Texinfo look of the produced html/pdf/plaintext/info files.

Not the output format, but the language: new directives and sometimes
markup are added with every release of Texinfo.  Just read their NEWS
file.

> As for being slow, it is already not the case. Last time we discussed
> Org export being slow, I have managed to speed things up on par with
> Texinfo timings on Org manual.

That was for Org-to-Texinfo conversion.  Here we are talking about
Org-to-Info, which is much more work.

Re: Org mode and Emacs

[Ihor Radchenko]

Po Lu <luangruo@yahoo.com> writes:

>>>> As for being slow, it is already not the case. Last time we discussed
>>>> Org export being slow, I have managed to speed things up on par with
>>>> Texinfo timings on Org manual.
>>>
>>> Did you use Texinfo 4.13 as a baseline for the comparison, or a more
>>> recent release written in Perl?
>>
>> More recent release.
>
> makeinfo 7.0.2 requires approximately 18x more time to generate
> emacs.info than 4.13, so an accurate performance comparison cannot be
> made with the Perl versions as a basis.

If makeinfo 7.0.2 deems its performance acceptable, I do not see why
similar Org's performance should not be.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I am not sure about catchup game. Unless Texinfo drastically changes the
>> output format, we can simply provide default settings to replicate the
>> Texinfo look of the produced html/pdf/plaintext/info files.
>
> Not the output format, but the language: new directives and sometimes
> markup are added with every release of Texinfo.  Just read their NEWS
> file.

The way I see Org markup extension would make it easy to users add new
custom markup, as needed. Then, no frequent changes to the base markup
will be necessary to accommodate for less common use cases.

In other words, I do not see why Org should support every single Texinfo
markup. We just need to provide enough support to be on par in terms of
the needs of manual authors.

>> As for being slow, it is already not the case. Last time we discussed
>> Org export being slow, I have managed to speed things up on par with
>> Texinfo timings on Org manual.
>
> That was for Org-to-Texinfo conversion.  Here we are talking about
> Org-to-Info, which is much more work.

Why so?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Po Lu]

Ihor Radchenko <yantar92@posteo.net> writes:

> If makeinfo 7.0.2 deems its performance acceptable, I do not see why
> similar Org's performance should not be.

Many of us don't find makeinfo 7.0.2's performance tolerable, which is
why Emacs continues to support Texinfo 4.13.  It's not the only Texinfo
user to maintain such compatibility, BTW.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 13:20:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I am not sure about catchup game. Unless Texinfo drastically changes the
> >> output format, we can simply provide default settings to replicate the
> >> Texinfo look of the produced html/pdf/plaintext/info files.
> >
> > Not the output format, but the language: new directives and sometimes
> > markup are added with every release of Texinfo.  Just read their NEWS
> > file.
> 
> The way I see Org markup extension would make it easy to users add new
> custom markup, as needed. Then, no frequent changes to the base markup
> will be necessary to accommodate for less common use cases.

You missed the "new directives" part.  I do recommend reading their
NEWS to get the idea of the rate at which new features are added to
Texinfo.

> In other words, I do not see why Org should support every single Texinfo
> markup. We just need to provide enough support to be on par in terms of
> the needs of manual authors.

GNU Manuals use a large portion of what Texinfo provides, and limiting
what they could use when they write in Org would mean some Texinfo
features cannot be used that way, which is a disadvantage.  People
will have to think twice before they switch to Org, because at some
point they might want to use a feature you decided not to support.

> > That was for Org-to-Texinfo conversion.  Here we are talking about
> > Org-to-Info, which is much more work.
> 
> Why so?

Because the production rules are much more complex.  I suggest to take
a look at the Perl source code of texi2any to see what it entails.

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 21:33:50 +0800
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
> > If makeinfo 7.0.2 deems its performance acceptable, I do not see why
> > similar Org's performance should not be.
> 
> Many of us don't find makeinfo 7.0.2's performance tolerable, which is
> why Emacs continues to support Texinfo 4.13.

That's your personal view.  I find the time it takes to produce even
such large manuals as emacs.info and elisp.info with Texinfo 7
entirely reasonable, at least when Perl extensions are used.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Not the output format, but the language: new directives and sometimes
>> > markup are added with every release of Texinfo.  Just read their NEWS
>> > file.
>> 
>> The way I see Org markup extension would make it easy to users add new
>> custom markup, as needed. Then, no frequent changes to the base markup
>> will be necessary to accommodate for less common use cases.
>
> You missed the "new directives" part.  I do recommend reading their
> NEWS to get the idea of the rate at which new features are added to
> Texinfo.

There is a single word "directives" in
https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:

    . #line directives are recognized.

>> In other words, I do not see why Org should support every single Texinfo
>> markup. We just need to provide enough support to be on par in terms of
>> the needs of manual authors.
>
> GNU Manuals use a large portion of what Texinfo provides, and limiting
> what they could use when they write in Org would mean some Texinfo
> features cannot be used that way, which is a disadvantage.  People
> will have to think twice before they switch to Org, because at some
> point they might want to use a feature you decided not to support.

I understand. However, new features are added to Texinfo for a reason.
If the same reason is valid for Org, equivalent features may be added.
Similarly, we may add features to Org that have no equivalent in
Texinfo. So, this is a normal development process with new features
being proposed, discussed, and maybe added.

From my point of view, equivalence with Texinfo is not something we care
as much about yet. It is just a general direction we can use for Org
development. I deem it useful even if we cannot reach all the Texinfo
features at the end. But if we can, we can _later_ see further about how
we want to incorporate (or not) new Texinfo features.

>> > That was for Org-to-Texinfo conversion.  Here we are talking about
>> > Org-to-Info, which is much more work.
>> 
>> Why so?
>
> Because the production rules are much more complex.  I suggest to take
> a look at the Perl source code of texi2any to see what it entails.

AFAIU, the conversion is done in
/usr/share/texinfo/Texinfo/Convert/Info.pm
I do not see anything extraordinary. Org uses similar approach - working
with parse tree.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> Date: Wed, 06 Sep 2023 16:12:57 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Not the output format, but the language: new directives and sometimes
> >> > markup are added with every release of Texinfo.  Just read their NEWS
> >> > file.
> >> 
> >> The way I see Org markup extension would make it easy to users add new
> >> custom markup, as needed. Then, no frequent changes to the base markup
> >> will be necessary to accommodate for less common use cases.
> >
> > You missed the "new directives" part.  I do recommend reading their
> > NEWS to get the idea of the rate at which new features are added to
> > Texinfo.
> 
> There is a single word "directives" in
> https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:
> 
>     . #line directives are recognized.

Look for changes whose heading is "Language:".

> >> In other words, I do not see why Org should support every single Texinfo
> >> markup. We just need to provide enough support to be on par in terms of
> >> the needs of manual authors.
> >
> > GNU Manuals use a large portion of what Texinfo provides, and limiting
> > what they could use when they write in Org would mean some Texinfo
> > features cannot be used that way, which is a disadvantage.  People
> > will have to think twice before they switch to Org, because at some
> > point they might want to use a feature you decided not to support.
> 
> I understand. However, new features are added to Texinfo for a reason.
> If the same reason is valid for Org, equivalent features may be added.

The reason why features are added to Texinfo is that those features
are useful for writing software documentation.  After all, this is the
main purpose of Texinfo.  So I cannot see how the same reasons could
not be valid for Org -- assuming that Org wants to support writing
software documentation, not just the Emacs manuals in their current
form.

> Similarly, we may add features to Org that have no equivalent in
> Texinfo.

When you add features to Org, no one needs to catch up with them.  But
when Texinfo adds new features, users of Org who use Org for writing
software documentation will expect those features to be supported,
since Org will (AFAIU) claim that it can be used for that.

> >> Why so?
> >
> > Because the production rules are much more complex.  I suggest to take
> > a look at the Perl source code of texi2any to see what it entails.
> 
> AFAIU, the conversion is done in
> /usr/share/texinfo/Texinfo/Convert/Info.pm
> I do not see anything extraordinary. Org uses similar approach - working
> with parse tree.

Fine, then I guess you are all set and can simply forget what I said.

Re: Org mode and Emacs

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> There is a single word "directives" in
>> https://git.savannah.gnu.org/cgit/texinfo.git/plain/NEWS:
>> 
>>     . #line directives are recognized.
>
> Look for changes whose heading is "Language:".

I did, and in many cases Texinfo added things that are already available
in Org. Otherwise, these changes are not really that frequent. I do not
see much problem catching up.

>> I understand. However, new features are added to Texinfo for a reason.
>> If the same reason is valid for Org, equivalent features may be added.
>
> The reason why features are added to Texinfo is that those features
> are useful for writing software documentation.  After all, this is the
> main purpose of Texinfo.  So I cannot see how the same reasons could
> not be valid for Org -- assuming that Org wants to support writing
> software documentation, not just the Emacs manuals in their current
> form.

Org is different from Texinfo and certain things that are impossible
with Texinfo and require changing grammar are trivial in Org without
upstream changes. I really do not see the raised problem as real
problem.

To clarify, my stance on this idea with better support for authoring
software manuals is to be done in several stages:


   [ we are at this point now ]

1. Provide basic programmable constructs on Org syntax level. This is
   generally useful and should be done anyway for various reasons.
   
2. Use the programmable constructs as a basis for an optional library
   that will allow software manual-specific markup in Org, aiming for an
   equivalence of export output. This can be useful, at least, for
   people using Org to write README files.

   [ here, we may re-consider the whole idea and see if we should move
     further ]

3. Look into fine-tuning commands provided by Texinfo (like formatting
   and layout control) and see if they can (or should) be integrated
   into Org.

   [ the problem you describe is to be considered seriously here, and I
      do not see why it should be the show stopper ]

4. Look if people are interested to switch from Texinfo to Org.

5. Look about maintaining parity between Texinfo and Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I understand. However, new features are added to Texinfo for a reason.

That is correct.  We added each of these features to Texinfo because
some GNU manual needed it.  Indeed, the simple way to find out all the
features GNU manuals need is to look at the Texinfo manual and see
what they required us to support.

We're comsidering the idea of making Org format adequate for GNU
mamuals.  Currently it can't express the distinctions that are needed
in order to produce the proper output for each of the output formats.

If we want to achieve that, we come to this question:

* What specific extension syntax would we use for Org equivalents
of the Texinfo constructs that Org currently cannot express>

  > The way I see Org markup extension would make it easy to users add new
  > custom markup, as needed. Then, no frequent changes to the base markup
  > will be necessary to accommodate for less common use cases.

I see a possible ambiguity and point of confusion.  When you say,
"extension", do you mean "a package that gets loaded on top of
ordinary Org mode"?  That's what I thought it meant.

Implementing some of the Texinfo constructs in such a package, perhaps
called org-texinfo, is an implementation detail as far as I'm
concerned.

But now I think maybe you mean something else -- that you propose
to add some sort of limited macro definition facility and have the
missing Texinfo constructs be defined using that.  Is that it?

To be adequate for this job, the macro definition facility needs to be
more powerful than they usually are.  The expansion of one construct
needs to depend on the output format being generated, and sometimes
the expansion of construct A depends on whether it is inside construct
B.

If the facility can do that, I think it will suffice for nearly all of
the missing Texinfo constructs.  If you think of this as a method to
simplify part of the implementation of Texinfo in Org, it may work.

But be prepared for exeptions, constructs that need special handling!
If you think of this as a way to keep Org itself free of Texinfo
impurities, it won't work.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Currently, Org mode does not support all the markup supported by
  > texinfo. But we will change that.

  > Meanwhile, Org implements info/manual export by ad-hoc extension
  > allowing to add texinfo-specific markup. This is not ideal, and it does
  > not need to be done after we do add the missing markup (Org equivalents
  > of @kbd, @var, and other)

That is good news -- it means we have the same goal in mind.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Ihor Radchenko]

Richard Stallman <rms@gnu.org> writes:

>   > The way I see Org markup extension would make it easy to users add new
>   > custom markup, as needed. Then, no frequent changes to the base markup
>   > will be necessary to accommodate for less common use cases.
>
> I see a possible ambiguity and point of confusion.  When you say,
> "extension", do you mean "a package that gets loaded on top of
> ordinary Org mode"?  That's what I thought it meant.

A package or user Elisp snippet.

For example, Org currently allows extending hyperlinks like

     (defun org-man-export (link description format _)
       "Export a man page link from Org files."
       (let ((path (format "http://man.he.net/?topic=%s&section=all" link))
             (desc (or description link)))
         (pcase format
           (`html (format "<a target=\"_blank\" href=\"%s\">%s</a>" path desc))
           (`latex (format "\\href{%s}{%s}" path desc))
           (`texinfo (format "@uref{%s,%s}" path desc))
           (`ascii (format "%s (%s)" desc path))
           (t path))))

          (org-link-set-parameters "man" :export #'org-man-export)

Then, <man:emacs> links will be formatted arbitrarily during export.

The same idea will be for markup syntax:

@var{variable-name} will, in future, be defined as

    (org-markup-set-parameters "var" :export #'my-export-function-for-var)

@var is probably something we will have within Org, but if one needs
some weird markup for a specific manual, it will equally be possible to
define

@my-special-markup{contents}

via (org-markup-set-parameters "my-special-markup" :export #'custom-export-formatter)

> Implementing some of the Texinfo constructs in such a package, perhaps
> called org-texinfo, is an implementation detail as far as I'm
> concerned.
>
> But now I think maybe you mean something else -- that you propose
> to add some sort of limited macro definition facility and have the
> missing Texinfo constructs be defined using that.  Is that it?

We might allow in-document definitions like:

#+markup[html]: my-special-markup <foo>%s</foo>
#+markup[latex]: my-special-markup \foo{%s}
...

but the `org-markup-set-parameters' is what we preliminarily agreed upon
for now. Further features are to be discussed later.

> To be adequate for this job, the macro definition facility needs to be
> more powerful than they usually are.  The expansion of one construct
> needs to depend on the output format being generated, and sometimes
> the expansion of construct A depends on whether it is inside construct
> B.

> If the facility can do that, I think it will suffice for nearly all of
> the missing Texinfo constructs.  If you think of this as a method to
> simplify part of the implementation of Texinfo in Org, it may work.

Org is already capable to provide access to the full parse tree when
expanding links with custom :export function. The same can be done for
markup constructs. Much more difficult if one wants in-document
definitions though.

> But be prepared for exeptions, constructs that need special handling!
> If you think of this as a way to keep Org itself free of Texinfo
> impurities, it won't work.

May you elaborate about special handling?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Those proposed steps seem reasonable to me in principle.
It is just a matter of making them work.

I don't see any reason why any of them should turn out impossible,
except _maybe_ the task of finding good syntaxes to use for the
various constructs new in Org format.  The rest is just writing
software, and we know how to do that.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Org mode and Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, Org currently allows extending hyperlinks like

  >      (defun org-man-export (link description format _)
  >        "Export a man page link from Org files."

This method like it should be able to handle the various output formats
for Texinfo constructs -- as long as once in a while we can define
a function that's a little ideosyncratic, for a construct that is
a little more irregular in its behavior.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)