bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Richard Brooksby]

Fill-paragraph (M-Q) in a reStructuredText document that is formatted
using "Line Blocks"
<https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#line-blocks>
such as:

   | In Xanadu did Kubla Khan
   | A stately pleasure-dome decree:
   | Where Alph, the sacred river, ran
   | Through caverns measureless to man
   | Down to a sunless sea.

results in this:

   | In Xanadu did Kubla Khan | A stately pleasure-dome decree: | Where
   Alph, the sacred river, ran | Through caverns measureless to man |
   Down to a sunless sea.

but in this case it should leave the line breaks intact.  The code
should probably treat the vertical bars something like list bullets,
since continuation lines are possible, e.g.

   | This is the all the first line
     of output even though it's two lines
     of input.
   | This is second line in the output.
   | And this is the third.

(I searched <https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs>
for a similar report.  Apologies if I was not thorough enough.)

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll 
bars)
  of 2024-09-17, modified by Debian built on lcy02-amd64-079
Windowing system distributor 'The X.Org Foundation', version 11.0.12201001
System Description: Ubuntu 22.04.5 LTS

Recent messages:
uncompressing rst.el.gz...done
Note: file is write protected
Mark set [3 times]
Making completion list...
delete-backward-char: Text is read-only
Quit [2 times]
Type C-x 1 to remove help window.  Type "q" in help window to restore 
its previous buffer.
You can run the command ‘info-emacs-bug’ with M-x inf-b RET
Making completion list...

Configured using:
  'configure --build x86_64-linux-gnu --prefix=/usr
  --sharedstatedir=/var/lib --libexecdir=/usr/lib
  --localstatedir=/var/lib --infodir=/usr/share/info
  --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
  --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
  --with-sound=alsa --without-gconf --with-mailutils --build
  x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
  --libexecdir=/usr/lib --localstatedir=/var/lib
  --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
  --with-pop=yes
  --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
  --with-sound=alsa --without-gconf --with-mailutils --with-x=yes
  --with-x-toolkit=lucid --with-toolkit-scroll-bars --without-gsettings
  'CFLAGS=-g -O2
  -ffile-prefix-map=/build/emacs-vPr175/emacs-27.1+1=. 
-fstack-protector-strong
  -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
  -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

Important settings:
   value of $LC_MONETARY: en_GB.UTF-8
   value of $LC_NUMERIC: en_GB.UTF-8
   value of $LC_TIME: en_GB.UTF-8
   value of $LANG: en_GB.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8-unix

Major mode: ReST

Minor modes in effect:
   shell-dirtrack-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   auto-fill-function: do-auto-fill
   transient-mark-mode: t

Load-path shadows:
/home/rb/.emacs.d/elpa/magit-20210117.2011/magit-section hides 
/home/rb/.emacs.d/elpa/magit-section-20241122.1431/magit-section
/usr/share/emacs/site-lisp/llvm-14/emacs hides 
/usr/share/emacs/site-lisp/llvm-15/emacs
/usr/share/emacs/site-lisp/llvm-14/tablegen-mode hides 
/usr/share/emacs/site-lisp/llvm-15/tablegen-mode
/usr/share/emacs/site-lisp/llvm-14/llvm-mode hides 
/usr/share/emacs/site-lisp/llvm-15/llvm-mode
/home/rb/.emacs.d/elpa/map-3.3.1/map hides 
/usr/share/emacs/27.1/lisp/emacs-lisp/map
/home/rb/.emacs.d/elpa/seq-2.24/seq hides 
/usr/share/emacs/27.1/lisp/emacs-lisp/seq

Features:
(shadow sort mail-extr emacsbug sendmail magit-utils crm eieio-opt
speedbar sb-image ezimage dframe apropos man jka-compr find-func
misearch multi-isearch cl-extra help-fns radix-tree help-mode mule-util
log-edit message rmc puny rfc822 mml mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log
flyspell ispell rst compile vc-git diff-mode easy-mmode vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs vc-dir ewoc vc vc-dispatcher dired-aux
iso-transl dired dired-loaddefs cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs cus-edit cus-start
cus-load wid-edit solarized-dark-theme solarized color dash lxd-tramp
tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat
shell pcomplete comint ansi-color ring parse-time iso8601 time-date
ls-lisp format-spec todotxt-mode edmacro kmacro server finder-inf info
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 445178 33251)
  (symbols 48 25380 2)
  (strings 32 133208 16553)
  (string-bytes 1 3430629)
  (vectors 16 36796)
  (vector-slots 8 838518 57172)
  (floats 8 299 280)
  (intervals 56 8211 530)
  (buffers 1000 36))

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Eli Zaretskii]

> Date: Wed, 11 Dec 2024 10:37:01 +0000
> From: Richard Brooksby <rb@ravenbrook.com>
> 
> 
> 
> Fill-paragraph (M-Q) in a reStructuredText document that is formatted
> using "Line Blocks"
> <https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#line-blocks>
> such as:
> 
>    | In Xanadu did Kubla Khan
>    | A stately pleasure-dome decree:
>    | Where Alph, the sacred river, ran
>    | Through caverns measureless to man
>    | Down to a sunless sea.
> 
> results in this:
> 
>    | In Xanadu did Kubla Khan | A stately pleasure-dome decree: | Where
>    Alph, the sacred river, ran | Through caverns measureless to man |
>    Down to a sunless sea.
> 
> but in this case it should leave the line breaks intact.  The code
> should probably treat the vertical bars something like list bullets,
> since continuation lines are possible, e.g.
> 
>    | This is the all the first line
>      of output even though it's two lines
>      of input.
>    | This is second line in the output.
>    | And this is the third.
> 
> (I searched <https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs>
> for a similar report.  Apologies if I was not thorough enough.)

Stefan and Stefan, any ideas or suggestions?

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Stefan Merten]

Hi Eli!

2 days ago Eli Zaretskii wrote:
>> Date: Wed, 11 Dec 2024 10:37:01 +0000
>> From: Richard Brooksby <rb@ravenbrook.com>
>> 
>> 
>> 
>> Fill-paragraph (M-Q) in a reStructuredText document that is formatted
>> using "Line Blocks"
>> <https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#line-blocks>
>> such as:
>> 
>>    | In Xanadu did Kubla Khan
>>    | A stately pleasure-dome decree:
>>    | Where Alph, the sacred river, ran
>>    | Through caverns measureless to man
>>    | Down to a sunless sea.
>> 
>> results in this:
>> 
>>    | In Xanadu did Kubla Khan | A stately pleasure-dome decree: | Where
>>    Alph, the sacred river, ran | Through caverns measureless to man |
>>    Down to a sunless sea.
>> 
>> but in this case it should leave the line breaks intact.  The code
>> should probably treat the vertical bars something like list bullets,
>> since continuation lines are possible, e.g.
>> 
>>    | This is the all the first line
>>      of output even though it's two lines
>>      of input.
>>    | This is second line in the output.
>>    | And this is the third.
>> 
>> (I searched <https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs>
>> for a similar report.  Apologies if I was not thorough enough.)
> 
> Stefan and Stefan, any ideas or suggestions?

Richard is right: Filling should consider line blocks. OTOH line
blocks are really a rarely used feature I guess, so it doesn't hurt
much.

I'll look deeper into the code. But don't hold your breath ;-) .


						Regards

						Stefan

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>    | This is the all the first line
>>>      of output even though it's two lines
>>>      of input.
>>>    | This is second line in the output.
>>>    | And this is the third.
[...]
>> Stefan and Stefan, any ideas or suggestions?

[ With all those Stefans around I feel like nothing can stop us.  ]

I think we "just" need to:

- set `paragraph-start` to match a line with a leading `|`.
- set `adaptive-fill-regexp` to match this leading `|`.

If the `paragraph-start` setting makes paragraph navigation inconvenient
(too fine-grained), we could try and set
`fill-forward-paragraph-function` to a function which let-binds
`paragraph-start` around a call to `forward-paragraph`.


        Stefan

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[James Cloos]

>>>>> Bug reports for GNU Emacs, the Swiss army knife of text editors <Stefan> writes:

this sort of wrapping used to work (at least in message buffers) for
many years.

you should be able to find my bug report about it breaking and the
resuting refusal even to accept that it is a bug.

-JimC
-- 
James Cloos <cloos@jhcloos.com>
            OpenPGP: https://jhcloos.com/0x997A9F17ED7DAEA6.asc

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Richard Brooksby <rb@ravenbrook.com>,
>   74785@debbugs.gnu.org
> Date: Mon, 16 Dec 2024 21:53:31 -0500
> 
> - set `adaptive-fill-regexp` to match this leading `|`.

Doesn't it already?

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Eli Zaretskii
>  <eliz@gnu.org>,  74785@debbugs.gnu.org,  Richard Brooksby
>  <rb@ravenbrook.com>
> Copyright: Copyright 2015 James Cloos
> Date: Tue, 17 Dec 2024 01:05:23 -0500
> 
> >>>>> Bug reports for GNU Emacs, the Swiss army knife of text editors <Stefan> writes:
> 
> this sort of wrapping used to work (at least in message buffers) for
> many years.
> 
> you should be able to find my bug report about it breaking and the
> resuting refusal even to accept that it is a bug.

If that's bug#24081, then I don't see any rejection there, only a
request for clarification (which you gave).  Then the bug was for some
reason closed several months later, with no rationale.

We could reopen the bug, of course (assuming it still relevant to
current versions of Emacs 30 and 31).  But that bug is about
message-mode, not about rst-mode, so I'm not sure it is the same
problem.

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> If that's bug#24081, then I don't see any rejection there, only a
EZ> request for clarification (which you gave).  Then the bug was for some
EZ> reason closed several months later, with no rationale.

that is the bug number.  i also received an email reply, which i see is
not in debbugs, claiming that it were not a bug and overtly refusing any
acceptance of ot work on a fix.  i don't know why that reply is elided.

-JimC
-- 
James Cloos <cloos@jhcloos.com>
            OpenPGP: https://jhcloos.com/0x997A9F17ED7DAEA6.asc

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: stefan@merten-home.de,  monnier@iro.umontreal.ca,
>   74785@debbugs.gnu.org,  rb@ravenbrook.com
> Copyright: Copyright 2015 James Cloos
> Date: Tue, 17 Dec 2024 09:49:03 -0500
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> If that's bug#24081, then I don't see any rejection there, only a
> EZ> request for clarification (which you gave).  Then the bug was for some
> EZ> reason closed several months later, with no rationale.
> 
> that is the bug number.  i also received an email reply, which i see is
> not in debbugs, claiming that it were not a bug and overtly refusing any
> acceptance of ot work on a fix.  i don't know why that reply is elided.

Neither do I.  Did that email CC the bug number, or was it sent only
to you?  And who sent it, if I may ask?

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Richard Brooksby]

On 2024-12-17 15:11, Eli Zaretskii wrote:
>> From: James Cloos <cloos@jhcloos.com>
>> Cc: stefan@merten-home.de,  monnier@iro.umontreal.ca,
>>    74785@debbugs.gnu.org,  rb@ravenbrook.com
>> Copyright: Copyright 2015 James Cloos
>> Date: Tue, 17 Dec 2024 09:49:03 -0500
>>
>>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
>>
>> EZ> If that's bug#24081, then I don't see any rejection there, only a
>> EZ> request for clarification (which you gave).  Then the bug was for some
>> EZ> reason closed several months later, with no rationale.
>>
>> that is the bug number.  i also received an email reply, which i see is
>> not in debbugs, claiming that it were not a bug and overtly refusing any
>> acceptance of ot work on a fix.  i don't know why that reply is elided.
> 
> Neither do I.  Did that email CC the bug number, or was it sent only
> to you?  And who sent it, if I may ask?

Excuse me for not being familiar with the bug system, but is it possible 
to discuss that bug in that bug's thread, rather than having a 
discussion about bug management in this thread about RST verse handling?

Thanks.

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Eli Zaretskii]

> Date: Tue, 17 Dec 2024 15:34:31 +0000
> Cc: stefan@merten-home.de, monnier@iro.umontreal.ca, 74785@debbugs.gnu.org
> From: Richard Brooksby <rb@ravenbrook.com>
> 
> On 2024-12-17 15:11, Eli Zaretskii wrote:
> >> From: James Cloos <cloos@jhcloos.com>
> >> Cc: stefan@merten-home.de,  monnier@iro.umontreal.ca,
> >>    74785@debbugs.gnu.org,  rb@ravenbrook.com
> >> Copyright: Copyright 2015 James Cloos
> >> Date: Tue, 17 Dec 2024 09:49:03 -0500
> >>
> >>>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> >>
> >> EZ> If that's bug#24081, then I don't see any rejection there, only a
> >> EZ> request for clarification (which you gave).  Then the bug was for some
> >> EZ> reason closed several months later, with no rationale.
> >>
> >> that is the bug number.  i also received an email reply, which i see is
> >> not in debbugs, claiming that it were not a bug and overtly refusing any
> >> acceptance of ot work on a fix.  i don't know why that reply is elided.
> > 
> > Neither do I.  Did that email CC the bug number, or was it sent only
> > to you?  And who sent it, if I may ask?
> 
> Excuse me for not being familiar with the bug system, but is it possible 
> to discuss that bug in that bug's thread, rather than having a 
> discussion about bug management in this thread about RST verse handling?

That bug is archived, so we cannot discuss anything unless we
unarchive it.

But we can stop discussing that here, since it's a separate issue.
Thanks for pointing that out.

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> But that bug is about message-mode, not about rst-mode, so I'm not
> sure it is the same problem.

It's not (that bug was about a prefix shared by lines of a paragraph,
whereas this bug is about a prefix which starts a new paragraph).


        Stefan

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> - set `adaptive-fill-regexp` to match this leading `|`.
> Doesn't it already?

Not in my test, no.  With a paragraph like:

    | bblblbl
    | asdfasdf
    | asdasdf

M-: (and (looking-at adaptive-fill-regexp) (match-string 0)) RET
returns a string that contains only the leading spaces.

`paragraph-start` doesn't match those lines either.


        Stefan

bug#74785: 27.1; Fill-paragraph in RST mode does not respect line blocks

[Richard Brooksby]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On 2024-12-17 17:12, Stefan Monnier wrote:
>>> - set `adaptive-fill-regexp` to match this leading `|`.
>> Doesn't it already?
> 
> Not in my test, no.  With a paragraph like:
> 
>      | bblblbl
>      | asdfasdf
>      | asdasdf

Please find attached a self-documenting test document that you may find 
useful.

Thank you!

[-- Attachment #2: verse-test.rst --]
[-- Type: text/x-rst, Size: 2694 bytes --]

================
 Bug 74785 test
================

This is a test file for `GNU Emacs bug 74785
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74785>`_, created to
help clarify the issue and help developers test solutions.  It's not
exhaustive.

You can check the output easily with rst-compile (C-c C-c C-c).

It would be nice if selecting the *entire* document and executing
fill-paragraph would result in a document that is:

1. filled and wrapped nicely as a text source

2. has unchanged output.

Example one.  In the simple case, as a top-level block.

| In Xanadu did Kubla Khan
| A stately pleasure-dome decree:
| Where Alph, the sacred river, ran
| Through caverns measureless to man
| Down to a sunless sea.

Example two.  Here's the same thing as a quote.

  | In Xanadu did Kubla Khan
  | A stately pleasure-dome decree:
  | Where Alph, the sacred river, ran
  | Through caverns measureless to man
  | Down to a sunless sea.

  -- Samuel Taylor Coleridge

Example three.  With continuation lines, this should produce identical
output to the previous example.

  | In Xanadu did Kubla Khan
  | A stately
    pleasure-dome
    decree:
  | Where Alph,
    the sacred river,
    ran
  | Through caverns
                measureless to man
  | Down to a sunless sea.

  -- Samuel Taylor Coleridge

It would be nice if fill-paragraph (M-q) would reformat example three
to look like example two.

Example four.  Conversely, here's a verse with long lines beyond the
default fill column of 70.

  | Begins the night’s shadows creeping, his eyes mocking, and beseeching.
  | I must stop his soul from joining, to fight with Poes’ shadows trying…
  | Trying once more, to take his soul, toward those demon littered shores.
  | Grabbing the ipod I forward tore, to give him tranquility evermore. 
  | Pouch and ear buds firmly seated, engaged in a fierce-some war.
  | Morning found all, in blissful snores… Quoted the raven nevermore.

  -- Carol Eastman

Example five.  It would be nice if fill-paragraph would wrap these
lines, but in a way that keeps the output identical.  For example
(with fill-column 70):

  | Begins the night’s shadows creeping, his eyes mocking, and
    beseeching.
  | I must stop his soul from joining, to fight with Poes’ shadows
    trying…
  | Trying once more, to take his soul, toward those demon littered
    shores.
  | Grabbing the ipod I forward tore, to give him tranquility
    evermore.
  | Pouch and ear buds firmly seated, engaged in a fierce-some war.
  | Morning found all, in blissful snores… Quoted the raven nevermore.

  -- Carol Eastman

Of course all of the above must work when nested within other forms.

Thanks all.

.. end