bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[James Cloos]

Since I switched from using gnus git to master's internal gnus, I've
noticed that fill-paragraph in message mode no longer treats the line
prefixes as a fill-prefix.

Due to the other bug I had to downgrade from master to 25.0.95.  It,
too, has this bug.



In GNU Emacs 25.0.95.1 (x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2016-07-21 built on carbon.jhcloos.org
Windowing system distributor 'The X.Org Foundation', version 11.0.11699901
System Description:	Gentoo Base System release 2.2

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --disable-dependency-tracking
 --disable-silent-rules --docdir=/usr/share/doc/emacs-vcs-25.0.95
 --htmldir=/usr/share/doc/emacs-vcs-25.0.95/html --libdir=/usr/lib64
 --program-suffix=-emacs-25-vcs --infodir=/usr/share/info/emacs-25-vcs
 --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --with-gameuser=:gamestat --without-compress-install
 --with-file-notification=inotify --enable-acl --with-dbus --with-gpm
 --without-hesiod --with-kerberos --with-kerberos5 --with-xml2
 --without-selinux --with-gnutls --without-wide-int --with-zlib
 --with-sound=alsa --with-x --without-ns --without-gconf
 --without-gsettings --with-toolkit-scroll-bars --with-gif --with-jpeg
 --with-png --with-rsvg --with-tiff --with-xpm --with-imagemagick
 --with-xft --without-cairo --with-libotf --with-m17n-flt
 --with-x-toolkit=lucid --with-xaw3d
 GENTOO_PACKAGE=app-editors/emacs-vcs-25.0.95 'CFLAGS=-pipe -O2 -g3
 -march=native' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS NOTIFY ACL
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
LUCID X11

Important settings:
  value of $LC_COLLATE: C
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_TIME: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  show-paren-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Recent messages:
Saving file /home/cloos/.newsrc.eld...
Wrote /home/cloos/.newsrc.eld
Saving /home/cloos/.newsrc.eld...done
o is undefined
Quit [2 times]
Note: file is write protected
View mode: type C-h for help, h for commands, q to quit.
M-q runs the command fill-paragraph
Quit
Making completion list...

Load-path shadows:
/usr/share/emacs/site-lisp/graphviz-dot-mode/graphviz-dot-mode hides /home/cloos/elisp/graphviz-dot-mode
/usr/share/emacs/site-lisp/git/git-blame hides /home/cloos/elisp/git-blame
/usr/share/emacs/site-lisp/auctex/context hides /home/cloos/elisp/context
/usr/share/emacs/site-lisp/fricas/fricas hides /home/cloos/elisp/fricas
/usr/share/emacs/site-lisp/git/git hides /home/cloos/elisp/git
/home/cloos/elisp/json hides /usr/share/emacs/25.0.95/lisp/json
/home/cloos/elisp/doc-view hides /usr/share/emacs/25.0.95/lisp/doc-view
/usr/share/emacs/site-lisp/css-mode/css-mode hides /usr/share/emacs/25.0.95/lisp/textmodes/css-mode

Features:
(shadow emacsbug view gnus-draft org-element org-rmail org-mhe org-irc
org-info org-gnus org-bibtex bibtex org-bbdb org-w3m org org-macro
org-footnote org-pcomplete org-list org-faces org-entities noutline
outline org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table
ob-exp org-src ob-keys ob-comint ob-core ob-eval org-compat org-macs
org-loaddefs cal-menu calendar cal-loaddefs binhex supercite easy-mmode
regi sendmail ispell nnir gnus-fun gnus-html xml url-cache mm-url
timezone nndoc url-http url url-proxy url-privacy url-history url-auth
url-cookie url-domsuf url-gw url-expand url-methods flow-fill shr-color
color url-util url-parse url-vars shr dom subr-x vc-filewise vc-rcs sort
gnus-cite qp mm-archive mail-extr gnus-async face-remap quilt gnus-kill
gnus-ml disp-table repeat misearch multi-isearch cus-edit nndraft nnmh
nnml utf-7 nnmaildir nnfolder network-stream nsm starttls gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig mailcap nntp gnus-cache gnus-sum
gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source tls
gnutls utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message
idna dired rfc822 mml mml-sec epg mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils
mailheader gnus-win gnus gnus-ems nnheader mail-utils wid-edit jka-compr
paren time delsel cus-start cus-load elscreen jhc-backend-xft-fonts
w3m-load preview-latex scala-mode-auto mmm-auto mmm-vars mmm-compat
edmacro kmacro imenu ido ess-toolbar ess-mouse mouseme thingatpt
browse-url ess-menu ess-swv ess-noweb ess-noweb-font-lock-mode
ess-bugs-l essd-els ess-sas-d ess-sas-l ess-sas-a ess-sta-d ess-sta-l
cc-vars cc-defs make-regexp ess-sp6-d ess-dde ess-sp3-d ess-julia
julia-mode ert pp find-func ewoc debug ess-r-d ess-r-syntax
ess-r-completion ess-roxy essddr hideshow ess-help ess-r-package ess-s-l
ess ess-inf ess-tracebug compile tramp tramp-compat auth-source cl-seq
eieio eieio-core gnus-util mm-util help-fns mail-prsvr password-cache
tramp-loaddefs cl-macs trampver ucs-normalize shell pcomplete comint
ansi-color ring format-spec advice ess-mode ess-noweb-mode ess-utils
ess-generics cl ess-custom executable ess-compat ess-site bbdb-loaddefs
tex-site auto-loads finder-inf package epg-config seq byte-opt gv
bytecomp byte-compile cl-extra help-mode easymenu cconv cl-loaddefs
pcase cl-lib time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 3898602 1571080)
 (symbols 48 270919 48)
 (miscs 40 5499 10162)
 (strings 32 424264 680452)
 (string-bytes 1 15559952)
 (vectors 16 71060)
 (vector-slots 8 1441723 157156)
 (floats 8 773 1222)
 (intervals 56 17966 1956)
 (buffers 976 1845)
 (heap 1024 267783 131018))

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Tue, 26 Jul 2016 17:00:32 -0400
> 
> 
> Since I switched from using gnus git to master's internal gnus, I've
> noticed that fill-paragraph in message mode no longer treats the line
> prefixes as a fill-prefix.
> 
> Due to the other bug I had to downgrade from master to 25.0.95.  It,
> too, has this bug.

Please show a recipe to reproduce this.

Thanks.

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

JC>> Since I switched from using gnus git to master's internal gnus, I've
JC>> noticed that fill-paragraph in message mode no longer treats the line
JC>> prefixes as a fill-prefix.

JC>> Due to the other bug I had to downgrade from master to 25.0.95.  It,
JC>> too, has this bug.

EZ> Please show a recipe to reproduce this.

Hmm.  Interesting.

It does honour prefixes which match the re /^>+ +$/

But does not match arbitrary prefixes, such as those one gets when
using sc-cite-original for message-cite-function and with:
(quoting from my custom file):

 '(message-cite-function (quote sc-cite-original))
 '(sc-auto-fill-region-p nil)
 '(sc-citation-leader "")
 '(sc-preferred-attribution-list
   (quote
    ("sc-lastchoice" "x-attribution" "initials" "firstname" "lastname")))

Eg, with citations like above.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[ Yes, this is a very old bug. 🙁  ]

> JC>> Since I switched from using gnus git to master's internal gnus, I've
> JC>> noticed that fill-paragraph in message mode no longer treats the line
> JC>> prefixes as a fill-prefix.
>
> JC>> Due to the other bug I had to downgrade from master to 25.0.95.  It,
> JC>> too, has this bug.
> EZ> Please show a recipe to reproduce this.
> Hmm.  Interesting.
> It does honour prefixes which match the re /^>+ +$/

Yet it uses `mail-citation-prefix-regexp` which is set by default to:

    "\\([ \t]*\\(\\w\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+"

and that "should" match prefixes like the `> JC>>` above (it's even
mentioned in the docstring).

And indeed refilling

    >FOO> sdkfjha lkhakd hasdjfh kaf akhf kajdf aksdjfhajk df asdjkfh akdfh
    >FOO> kadfh akdjfh adkfh akdf hakdfh adjkfh k adfh akdjfh kafh kajfh kajfh kadjfh kadjfh kadjfh akjfh akjfh kadfh kajfh kajfh
    >FOO> kajdfh kadjfh 
    >FOO> akdjfh kadjfh

works fine for me.  OTOH refilling

    FOO> sdkfjha lkhakd hasdjfh kaf akhf kajdf aksdjfhajk df asdjkfh akdfh
    FOO> kadfh akdjfh adkfh akdf hakdfh adjkfh k adfh akdjfh kafh kajfh kajfh kadjfh kadjfh kadjfh akjfh akjfh kadfh kajfh kajfh
    FOO> kajdfh kadjfh 
    FOO> akdjfh kadjfh

doesn't (the `FOO>`s are treated as part of the text to be filled rather
than as a prefix, so some of them end up in the middle of the text).
My impression is that the problem is that the first char on each line is
given a syntax-table property of "start comment" which prevents the
`\w` in the regexp from matching the `F`.

The patch below fixes it for me.
Can you check if it also fixes your use cases?
Note: There have been some recentish changes/fixes to the regexp handling of
`[:word:]` w.r.t syntax-table text properties, so it's possible that the
patch doesn't work in Emacs-29, but it should work in Emacs≥30 )I tested
it in Emacs-31).
 

        Stefan


diff --git a/lisp/mail/sendmail.el b/lisp/mail/sendmail.el
index a720df51d14..acb6a38436e 100644
--- a/lisp/mail/sendmail.el
+++ b/lisp/mail/sendmail.el
@@ -258,7 +258,9 @@ mail-citation-header
 
 ;;;###autoload
 (defcustom mail-citation-prefix-regexp
-  (purecopy "\\([ \t]*\\(\\w\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+")
+  ;; Use [[:word:]] rather than \w so we don't get tripped up if one
+  ;; of those chars has a weird `syntax-table' text property.
+  (purecopy "\\([ \t]*\\([[:word:]]\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+")
   "Regular expression to match a citation prefix plus whitespace.
 It should match whatever sort of citation prefixes you want to handle,
 with whitespace before and after; it should also match just whitespace.

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[James Cloos]

yes, the patch worked.

thank you!

i'd stopped trying to figure it out after the stroke.

-JimC
-- 
James Cloos <cloos@jhcloos.com>
            OpenPGP: https://jhcloos.com/0x997A9F17ED7DAEA6.asc

bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> yes, the patch worked.
> thank you!
> i'd stopped trying to figure it out after the stroke.

Great, thanks, pushed to `master`.


        Stefan