Re: Convert README.org to plain text README while installing package

[Ihor Radchenko <yantar92@gmail.com>]

Eli Zaretskii <eliz@gnu.org> writes:

> Thank you, I think this is a very good summary.  I hope some of these
> issues could be worked on and improved in the future.

It is in my todo-list. I may find some time in future to work on it.

> For a serious discussion of this example, I'd need more detail about
> the aspects you mentioned above (it is not trivial to deduce them from
> looking at the top levels of org-forward-paragraph).  Specifically:
>
>   . how does paragraph definition depend on context in Org, and what
>     are the possible definitions in the existing contexts?
>   . why don't paragraph-start and paragraph-separate fit Org, and can
>     we come up with a small number of additional variables that would
>     augment these two so that the built-in forward-paragraph could be
>     extended to cover Org?

I do not want to start a serious discussion on this just yet as I do not
plant to work on this specific area in near future, however I would like
to answer some of your questions in order to provide some insight for
Emacs developers.

To answer your question about paragraph definition, we need to clarify
what "paragraph" means.

(*)
Plain text documents, in their simplest forms, usually consist of a
series of sections containing a series of paragraphs each. With
paragraphs being a series of sentences. The paragraphs are usually
separated by blank lines or by indentation.

Org mode extends the above simple structure by defining additional
syntactic elements, which are equivalent to paragraphs. Apart from
paragraphs, there can be tables, lists, other specially treated blocks
of text (source code blocks, comments, verse blocks, drawers, etc).
These elements may or may not be separated with blank lines. For
example,
| This table    | is considered as a new "paragraph"-like element |
| Not separated | from the above paragraph by blank lines         |
#+begin_src emacs-lisp
(message "And here we have yet another paragraph-like element")

;; This element may contain empty lines inside, yet these empty lines
;; are not indicating "paragraph"-like element boundary.

(message "They may, in emacs-lisp specifically, from some point of view;

but consider other possible programming languages with arbitrary syntax;
or even elisp, with the above blank line inside string.")
#+end_src

When a user calls forward-paragraph on a simple paragraph as defined in
(*), built-in forward-paragraph will behave narutally. However, things
become tricky when we have various paragraph-like elements. How can one
configure forward-paragraph to forward classic paragraph, tables, source
blocks, and all other possible elements?

To make things more complex, Org mode syntax is not context-free.
#+end_src
in here does not belong to a source block because it lacks #+begin_src
opener.
Also,
:DRAWER-LIKE-THIS:
Cannot intersect with other source-block-looking constructs like
#+begin_src emacs-lisp
(message "I am not a source block")
:END:
The
#+end_src
here is also not a part of source block because it cannot intersect
drawer boundary.

>> If you know a better way to resolve the described limitation, please let
>> me know.
>
> The better way, IMO, is this:
>
>   . try to find a way of extending the built-in forward-paragraph to
>     cover the Org use cases, by adding variables in addition to the 2
>     existing ones (I don't yet understand why "regexps do not cut it
>     for Org mode -- it sounds like a very strong assertion)

For regexps, I hope that the above example illustrated the problem
clearly. Also, note that a number of people attempted to develop
BNF/EBNF parsers for Org mode, but failed because of issues tike the
above. Org is not context-free and cannot be parsed using LR parsers,
let alone simple regexps.

For more insight, you may also consult org-element-paragraph-parser
code, which has to deal with the problems illustrated above. (note that
even this function alone is not sufficient to find paragraphs; it also
relies on other parser code).

>   . if the above doesn't work, make forward-paragraph work through
>     forward-paragraph-function, so that Org could define its own
>     function

Introducing the required flexibility into Emacs core is certainly an
option. Though it is somewhat difficult one because Org also has to
support older versions of Emacs. We are somewhat limited in using newly
introduced built-in features.

>   . in any case, the add-on convenience features of
>     org-forward-paragraph should be provided as options that the user
>     can control

They are not necessarily add-ons. First of all, org-forward-paragraph is
dealing with the above complications. Using forward-paragraph directly
would behave unexpectedly on various Org elements.

> My main concern is that forward-paragraph is used when editing the
> purely textual parts of an Org document, and when used there, casual
> users of Org will expect it to behave as the command behaves in plain
> text.  Any deviation from the behavior of the built-in command will
> confuse such users when they edit the plain-text parts, and should
> therefore be avoided.

There is no difference between forward-paragraph and
org-forward-paragraph on purely texture parts. The difference is only on
the Org-specific constructs, where forward-paragraph would behave
unexpectedly.

>> The following functions have a natural fallback to default behavior and
>> might be moved to minor-modes enabled by default. However, their default
>> behavior is context dependent and motivated by the lack of flexibility
>> of the built-in equivalents. Similar to the above functions.
>> 
>> <remap> <open-line>		org-open-line
>> <remap> <move-beginning-of-line> org-beginning-of-line
>> <remap> <move-end-of-line>	org-end-of-line
>
> Each of these (and other examples you provided) should require a
> separate serious discussion.  Let's take open-line as an example.  It
> is basically the built-in open-line, unless org-special-ctrl-o is
> non-nil.  A trivial extension of the built-in command could have
> prevented the need to remap.  (And why does the support for tables in
> org-open-line need to be turned on outside of orgtbl-mode?)

You misunderstand orgtbl-mode. orgtbl-mode is optional minor-mode
provided for users who want to edit tables "like in Org mode", but in
_other_ major modes. Inside Org mode, tables are always supported.

Also, org-special-ctrl-o is non-nil by default. Using built-in open-line
on Org tables can produce incorrect formatting. For example, calling
open-line on the following table

| this | is        | table |
| with | <point> 2 | lines |

will produce

| this | is | table |
| with | 
 2 | lines |

while org-open-line will produce

| this | is | table |
|      |    |       |
| with |  2 | lines |

The default behavior is not satisfactory and somewhat unexpected.

>> <remap> <yank>			org-yank
>> may be instead implemented using yank-handler property (it takes care
>> about adjusting level of the inserted headlines), but not completely. We
>> cannot control properties of text from outside of Org mode and thus the
>> functionality cannot be fully implemented using built-in Emacs features
>
> IMO, org-yank is sufficiently different to justify not taking over the
> built-in command.  Again, consider a user who edits the plain-text
> portions of an Org document and expects the "usual" behavior of C-y.

I personally tend to agree here.

> When fill-paragraph-function and other means of customizing a command
> to the particular mode make sense, they will not need to be noticed,
> because they will do what the users expect.  Moreover, customizing the
> behavior of commands through those variables has a couple of important
> advantages:
>
>   . it doesn't use remapping, so it doesn't stick out and bother
>     conscientious Emacs users, who are used to study unfamiliar
>     commands before using them
>   . they don't require separate documentation, apart of saying that
>     paragraph-start and paragraph-separate can be modified by modes as
>     appropriate -- all the rest of the command's documentation is
>     still intact and easily grasped in the context of any mode (this
>     is also one more reason why "niceties" like special behavior
>     inside tables should be separately controlled)

I get your point. As long as default customisations are sufficient to
achive the expected and reasonable behavior, they should indeed be
preferred.

>> I get your point on VHDL. However, I am not sure why you reject the
>> non-built-in examples. Are you implying that text modes outside Emacs
>> core are worse than built-in?
>
> No, I'm saying that, sadly, we have no real control on what the
> developers of unbundled packages decide to do.  Thus, what you see
> there is the evidence of our lack of control more than anything else.

I do not get your point here. I was referring to the markdown-mode and
Auctex to illustrate the _need_ to have numerous key bindings in order
to edit complex Org documents. It is not just something introduced into
Org out of the blue. Other people solved similar problems and did not
come up with significantly less key bindings. If you say that usual
40-50 major mode bindings are sufficient, I am not convinced.

Best,
Ihor