bug#49558: fill paragraph in texinfo-mode fails with @

bug#49558: fill paragraph in texinfo-mode fails with @

[lisa-asket]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]



I am using `M-q` to fill my paragraphs.  I have noticed that in texinfo-mode,

doing `M-q` on a paragraph starting with `@` does not refill the paragraph.



This occurs in instances such as



@noindent Everyone is permitted to copy and distribute verbatim
copies of this license document, but changing it is not allowed.






[-- Attachment #2: Type: text/html, Size: 430 bytes --]

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

lisa-asket@perso.be writes:

> I am using `M-q` to fill my paragraphs.  I have noticed that in texinfo-mode,
> doing `M-q` on a paragraph starting with `@` does not refill the paragraph.
>
> This occurs in instances such as
>
> @noindent Everyone is permitted to copy and distribute verbatim
> copies of this license document, but changing it is not allowed.

This is because texinfo-mode regards almost any line that starts with a
@ as a paragraph separator, so forward-paragraph etc doesn't work
properly on those lines.

The following patch fixes the problem, but I'm not sure this is the
correct thing, but the manual says:

--
Lines that start a new paragraph and are
contained in it must match only ‘paragraph-start’, not
‘paragraph-separate’. 
--

So I think it is?  But this code has been basically like this for
decades...  anybody got any comments about why it is like it is today?

diff --git a/lisp/textmodes/texinfo.el b/lisp/textmodes/texinfo.el
index 11d60e1eb0..311b2055ed 100644
--- a/lisp/textmodes/texinfo.el
+++ b/lisp/textmodes/texinfo.el
@@ -411,9 +411,6 @@ texinfo-mode
 		      "\\)\\>"))
   (setq-local require-final-newline mode-require-final-newline)
   (setq-local indent-tabs-mode nil)
-  (setq-local paragraph-separate
-	      (concat "@[a-zA-Z]*[ \n]\\|"
-		      paragraph-separate))
   (setq-local paragraph-start (concat "@[a-zA-Z]*[ \n]\\|"
 				      paragraph-start))
   (setq-local sentence-end-base "\\(@\\(end\\)?dots{}\\|[.?!]\\)[]\"'”)}]*")


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 14 Jul 2021 09:47:11 +0200
> Cc: 49558@debbugs.gnu.org
> 
> lisa-asket@perso.be writes:
> 
> > I am using `M-q` to fill my paragraphs.  I have noticed that in texinfo-mode,
> > doing `M-q` on a paragraph starting with `@` does not refill the paragraph.
> >
> > This occurs in instances such as
> >
> > @noindent Everyone is permitted to copy and distribute verbatim
> > copies of this license document, but changing it is not allowed.
> 
> This is because texinfo-mode regards almost any line that starts with a
> @ as a paragraph separator, so forward-paragraph etc doesn't work
> properly on those lines.

I actually think this is a "feature" in Texinfo mode.  In Texinfo, we
have constructs that start with @ at BOL which we don't _want_ to
refill, and we _do_ want them to be treated as a separate paragraph.
A few examples:

@node Foo Bar Baz Quux Very Long Node Name

@subsection This is a very long subsection name

@cindex a very long index entry that could wrap

@defun my-func with many different arguments that could wrap

@end multitable

And I could probably come up with a few more.  So @noindent is more an
exception that the rule, and there actually is no reason to have any
text after it on the same line, you could leave it on a line by itself
without losing anything.

> The following patch fixes the problem, but I'm not sure this is the
> correct thing, but the manual says:
> 
> --
> Lines that start a new paragraph and are
> contained in it must match only ‘paragraph-start’, not
> ‘paragraph-separate’. 
> --

I never took this seriously.  E.g., even our default values of these
variables don't seem to heed this rule.

Stefan, any comments?

> So I think it is?  But this code has been basically like this for
> decades...  anybody got any comments about why it is like it is today?

See above.  I'd prefer not to change these variables in Texinfo, as it
could cause more trouble than it fixes.  If we want the likes of
@noindent to be exempt from this treatment of lines that begin with @,
we could perhaps introduce a fill-paragraph-function specific for
Texinfo, and in that function treat ^@noindent (and perhaps a few
more directives) specially.

bug#49558: fill paragraph in texinfo-mode fails with @

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> --
>> Lines that start a new paragraph and are
>> contained in it must match only ‘paragraph-start’, not
>> ‘paragraph-separate’. 
>> --
>
> I never took this seriously.  E.g., even our default values of these
> variables don't seem to heed this rule.
>
> Stefan, any comments?

I do remember some fuzzily-obeyed rule about relationship between these
two vars, but this one seems to be non-fuzzy: if a line matches
`paragraph-separate` then its content is not considered as being part of
a refillable paragraph, IOW its content will be left alone, never
cutting it shorter or adding to it text from the following line.

In which sense do our default values not heed this rule?


        Stefan

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  lisa-asket@perso.be,
>   49558@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 08:59:31 -0400
> 
> In which sense do our default values not heed this rule?

The same line could match both paragraph-start and paragraph-separate.

bug#49558: fill paragraph in texinfo-mode fails with @

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> In which sense do our default values not heed this rule?
> The same line could match both paragraph-start and paragraph-separate.

But that rule doesn't say they can't, it just implies that if a line
matches both then it will be treated as a paragraph separator:

    Lines that start a new paragraph and are contained in it must
    match only ‘paragraph-start’, not ‘paragraph-separate’. 


-- Stefan

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  lisa-asket@perso.be,  49558@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 09:49:27 -0400
> 
> >> In which sense do our default values not heed this rule?
> > The same line could match both paragraph-start and paragraph-separate.
> 
> But that rule doesn't say they can't, it just implies that if a line
> matches both then it will be treated as a paragraph separator:
> 
>     Lines that start a new paragraph and are contained in it must
>     match only ‘paragraph-start’, not ‘paragraph-separate’. 

That's not my reading of that text, so I guess it should be reworded
(to avoid the implied double negation).

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I actually think this is a "feature" in Texinfo mode.  In Texinfo, we
> have constructs that start with @ at BOL which we don't _want_ to
> refill, and we _do_ want them to be treated as a separate paragraph.
> A few examples:
>
> @node Foo Bar Baz Quux Very Long Node Name
>
> @subsection This is a very long subsection name
>
> @cindex a very long index entry that could wrap
>
> @defun my-func with many different arguments that could wrap
>
> @end multitable

The problem is that all of that is currently a single paragraph -- and I
don't think that's something we want?  (Try saying
`M-: (forward-paragraph)' at the start.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  lisa-asket@perso.be,
>   49558@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 15:00:48 +0200
> 
> > @node Foo Bar Baz Quux Very Long Node Name
> >
> > @subsection This is a very long subsection name
> >
> > @cindex a very long index entry that could wrap
> >
> > @defun my-func with many different arguments that could wrap
> >
> > @end multitable
> 
> The problem is that all of that is currently a single paragraph -- and I
> don't think that's something we want?  (Try saying
> `M-: (forward-paragraph)' at the start.)

What exactly do we not want in the result of forward-paragraph in
these cases?  I'm probably missing something.

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> > @node Foo Bar Baz Quux Very Long Node Name
>> >
>> > @subsection This is a very long subsection name
>> >
>> > @cindex a very long index entry that could wrap
>> >
>> > @defun my-func with many different arguments that could wrap
>> >
>> > @end multitable
>> 
>> The problem is that all of that is currently a single paragraph -- and I
>> don't think that's something we want?  (Try saying
>> `M-: (forward-paragraph)' at the start.)
>
> What exactly do we not want in the result of forward-paragraph in
> these cases?  I'm probably missing something.

I think the example text you pasted should be five paragraphs, not one.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  lisa-asket@perso.be,  49558@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 15:15:04 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > @node Foo Bar Baz Quux Very Long Node Name
> >> >
> >> > @subsection This is a very long subsection name
> >> >
> >> > @cindex a very long index entry that could wrap
> >> >
> >> > @defun my-func with many different arguments that could wrap
> >> >
> >> > @end multitable
> >> 
> >> The problem is that all of that is currently a single paragraph -- and I
> >> don't think that's something we want?  (Try saying
> >> `M-: (forward-paragraph)' at the start.)
> >
> > What exactly do we not want in the result of forward-paragraph in
> > these cases?  I'm probably missing something.
> 
> I think the example text you pasted should be five paragraphs, not one.

That's because you think about it as plain text.  It isn't.
Basically, lines that start with @foo are directives, not text.

It could be somewhat surprising, because it otherwise looks very much
like plain text, but without those definitions of paragraph-start and
paragraph-separate, things would be much worse.  E.g., copy this to a
text-mode buffer, then type M-q:

@itemize @bullet
@item
If the text consists of a special glyph, the glyph can specify a
particular face.  @xref{Glyphs}.

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That's because you think about it as plain text.  It isn't.
> Basically, lines that start with @foo are directives, not text.
>
> It could be somewhat surprising, because it otherwise looks very much
> like plain text, but without those definitions of paragraph-start and
> paragraph-separate, things would be much worse.  E.g., copy this to a
> text-mode buffer, then type M-q:
>
> @itemize @bullet
> @item
> If the text consists of a special glyph, the glyph can specify a
> particular face.  @xref{Glyphs}.

Filling is one thing, but paragraphs are another.  texinfo-mode relies
on the paragraph definition being...  eccentric... to avoid filling
lines that start with "@word ".

I think the paragraph definition should be more traditional, and
`fill-paragraph-function' should be adjusted to do the right thing on
the @directives that we don't want to have filled.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: monnier@iro.umontreal.ca,  lisa-asket@perso.be,  49558@debbugs.gnu.org
> Date: Wed, 14 Jul 2021 15:53:42 +0200
> 
> texinfo-mode relies on the paragraph definition being...
> eccentric... to avoid filling lines that start with "@word ".

Yes, that is true.

> I think the paragraph definition should be more traditional, and
> `fill-paragraph-function' should be adjusted to do the right thing on
> the @directives that we don't want to have filled.

Could be.

But even if we do so, I'm not sure that

  @cindex relative remapping, faces
  @cindex base remapping, faces
    The following functions implement a higher-level interface to
  @code{face-remapping-alist}.  Most Lisp code should use these
  functions instead of setting @code{face-remapping-alist} directly, to
  avoid trampling on remappings applied elsewhere.  These functions are
  intended for buffer-local remappings, so they all make
  @code{face-remapping-alist} buffer-local as a side-effect.  They manage
  @code{face-remapping-alist} entries of the form

should be a single paragraph, and likewise this:

  @defun face-remap-add-relative face &rest specs
  This function adds the face spec in @var{specs} as relative
  remappings for face @var{face} in the current buffer.  The remaining
  arguments, @var{specs}, should form either a list of face names, or a
  property list of attribute/value pairs.

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But even if we do so, I'm not sure that
>
>   @cindex relative remapping, faces
>   @cindex base remapping, faces
>     The following functions implement a higher-level interface to
>   @code{face-remapping-alist}.  Most Lisp code should use these
>   functions instead of setting @code{face-remapping-alist} directly, to
>   avoid trampling on remappings applied elsewhere.  These functions are
>   intended for buffer-local remappings, so they all make
>   @code{face-remapping-alist} buffer-local as a side-effect.  They manage
>   @code{face-remapping-alist} entries of the form
>
> should be a single paragraph,

But it is one paragraph today.  With my proposed patch, it becomes two
paragraphs (since @cindex is the start of a paragraph).

> and likewise this:
>
>   @defun face-remap-add-relative face &rest specs
>   This function adds the face spec in @var{specs} as relative
>   remappings for face @var{face} in the current buffer.  The remaining
>   arguments, @var{specs}, should form either a list of face names, or a
>   property list of attribute/value pairs.

There would be no difference here -- that's one paragraph both before
and after the patch.

My preference would be to have the first example be three paragraphs,
and the second example two paragraphs, but I have no idea how to make
that happen with any paragraph-start/separate settings.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> texinfo-mode relies on the paragraph definition being...
>> eccentric... to avoid filling lines that start with "@word ".
>
> Yes, that is true.
>
>> I think the paragraph definition should be more traditional, and
>> `fill-paragraph-function' should be adjusted to do the right thing on
>> the @directives that we don't want to have filled.
>
> Could be.

I've now done this, and tested with all of the examples in this file
(and also in a couple of real-life texinfo files), and this approach
seems to work (and solves the original reported problem).  (I also made
'C-x n d' narrow to the current node, because that's something I've
missed for years, but it's unrelated.)

Let me know if I broke anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49558: fill paragraph in texinfo-mode fails with @

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The problem is that all of that is currently a single paragraph -- and I
> don't think that's something we want?  (Try saying
> `M-: (forward-paragraph)' at the start.)

I think the problem here is that our filling code has no notion of
a line being both "a paragraph separator" and "a paragraph" at the
same time.

I agree that it can make sense to treat `@subsection BLABLA` as
a paragraph for navigation purposes, tho one that cannot be filled
because it has to stay as a single line.

There's also something to be said for treating "a sequence paragraph
separators" as a paragraph for navigation purposes.


        Stefan