bug#24081: 25.0.95; fill-paragraph in message mode no longer honours line prefixes

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

[ Yes, this is a very old bug. 🙁  ]

> JC>> Since I switched from using gnus git to master's internal gnus, I've
> JC>> noticed that fill-paragraph in message mode no longer treats the line
> JC>> prefixes as a fill-prefix.
>
> JC>> Due to the other bug I had to downgrade from master to 25.0.95.  It,
> JC>> too, has this bug.
> EZ> Please show a recipe to reproduce this.
> Hmm.  Interesting.
> It does honour prefixes which match the re /^>+ +$/

Yet it uses `mail-citation-prefix-regexp` which is set by default to:

    "\\([ \t]*\\(\\w\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+"

and that "should" match prefixes like the `> JC>>` above (it's even
mentioned in the docstring).

And indeed refilling

    >FOO> sdkfjha lkhakd hasdjfh kaf akhf kajdf aksdjfhajk df asdjkfh akdfh
    >FOO> kadfh akdjfh adkfh akdf hakdfh adjkfh k adfh akdjfh kafh kajfh kajfh kadjfh kadjfh kadjfh akjfh akjfh kadfh kajfh kajfh
    >FOO> kajdfh kadjfh 
    >FOO> akdjfh kadjfh

works fine for me.  OTOH refilling

    FOO> sdkfjha lkhakd hasdjfh kaf akhf kajdf aksdjfhajk df asdjkfh akdfh
    FOO> kadfh akdjfh adkfh akdf hakdfh adjkfh k adfh akdjfh kafh kajfh kajfh kadjfh kadjfh kadjfh akjfh akjfh kadfh kajfh kajfh
    FOO> kajdfh kadjfh 
    FOO> akdjfh kadjfh

doesn't (the `FOO>`s are treated as part of the text to be filled rather
than as a prefix, so some of them end up in the middle of the text).
My impression is that the problem is that the first char on each line is
given a syntax-table property of "start comment" which prevents the
`\w` in the regexp from matching the `F`.

The patch below fixes it for me.
Can you check if it also fixes your use cases?
Note: There have been some recentish changes/fixes to the regexp handling of
`[:word:]` w.r.t syntax-table text properties, so it's possible that the
patch doesn't work in Emacs-29, but it should work in Emacs≥30 )I tested
it in Emacs-31).
 

        Stefan


diff --git a/lisp/mail/sendmail.el b/lisp/mail/sendmail.el
index a720df51d14..acb6a38436e 100644
--- a/lisp/mail/sendmail.el
+++ b/lisp/mail/sendmail.el
@@ -258,7 +258,9 @@ mail-citation-header
 
 ;;;###autoload
 (defcustom mail-citation-prefix-regexp
-  (purecopy "\\([ \t]*\\(\\w\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+")
+  ;; Use [[:word:]] rather than \w so we don't get tripped up if one
+  ;; of those chars has a weird `syntax-table' text property.
+  (purecopy "\\([ \t]*\\([[:word:]]\\|[_.]\\)+>+\\|[ \t]*[>|]\\)+")
   "Regular expression to match a citation prefix plus whitespace.
 It should match whatever sort of citation prefixes you want to handle,
 with whitespace before and after; it should also match just whitespace.