bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Eli Zaretskii]

> Date: Sat, 03 Aug 2024 16:14:39 +0300
> From:  Björn Bidar via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> Gnus doesn't handle international message reply prefixes as opposed
> to Re: or it's variants.
> 
> An example would be `gnus-simpliy-subject-re' which threats English
> subject prefixes.
> 
> Message.el doesn't do so either (by default) I don't think it makes
> sense to open a separate bug about this but please reply that should be
> done.
> 
> The message manual also judges users for using localized variants of
> "Re: " by calling them "nincompoops" for using these "abominations"
> which it says are "meaningless and evil".
> 
> The offending text can be found in the documentation for
> `message-subject-re-regexp' in the chapter "3.1 Message Headers".

FWIW, Rmail does support the various ways of saying "Re:", so the code
could be lifted from there.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 03 Aug 2024 16:14:39 +0300
>> From:  Björn Bidar via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> 
>> Gnus doesn't handle international message reply prefixes as opposed
>> to Re: or it's variants.
>> 
>> An example would be `gnus-simpliy-subject-re' which threats English
>> subject prefixes.
>> 
>> Message.el doesn't do so either (by default) I don't think it makes
>> sense to open a separate bug about this but please reply that should be
>> done.
>> 
>> The message manual also judges users for using localized variants of
>> "Re: " by calling them "nincompoops" for using these "abominations"
>> which it says are "meaningless and evil".
>> 
>> The offending text can be found in the documentation for
>> `message-subject-re-regexp' in the chapter "3.1 Message Headers".
>
> FWIW, Rmail does support the various ways of saying "Re:", so the code
> could be lifted from there.

Would it make sense to move regexp and intermittent variable to
message.el or mail-utils.el?

I would assume the latter.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: 72442@debbugs.gnu.org
> Date: Sat, 03 Aug 2024 19:58:46 +0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > FWIW, Rmail does support the various ways of saying "Re:", so the code
> > could be lifted from there.
> 
> Would it make sense to move regexp and intermittent variable to
> message.el or mail-utils.el?
> 
> I would assume the latter.

The latter, yes.  But note that Rmail does that for 2 different
purposes and in 2 different functions AFAIR.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

>>>>> On Sat, 03 Aug 2024 20:19:48 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Björn Bidar <bjorn.bidar@thaodan.de>
    >> Cc: 72442@debbugs.gnu.org
    >> Date: Sat, 03 Aug 2024 19:58:46 +0300
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > FWIW, Rmail does support the various ways of saying "Re:", so the code
    >> > could be lifted from there.

So does message, you just have to configure
`message-subject-re-regexp'. The message manual contains an
appropriate setting.

    >> 
    >> Would it make sense to move regexp and intermittent variable to
    >> message.el or mail-utils.el?
    >> 
    >> I would assume the latter.

    Eli> The latter, yes.  But note that Rmail does that for 2 different
    Eli> purposes and in 2 different functions AFAIR.

We could just add the example from the manual to the `defcustom' for
`message-subject-re-regexp' (and also align it with whatever Rmail
does).


Robert
-- 

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Björn Bidar <bjorn.bidar@thaodan.de>,
>   72442@debbugs.gnu.org
> Date: Tue, 20 Aug 2024 11:46:26 +0200
> 
> >>>>> On Sat, 03 Aug 2024 20:19:48 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> From: Björn Bidar <bjorn.bidar@thaodan.de>
>     >> Cc: 72442@debbugs.gnu.org
>     >> Date: Sat, 03 Aug 2024 19:58:46 +0300
>     >> 
>     >> Eli Zaretskii <eliz@gnu.org> writes:
>     >> 
>     >> > FWIW, Rmail does support the various ways of saying "Re:", so the code
>     >> > could be lifted from there.
> 
> So does message, you just have to configure
> `message-subject-re-regexp'. The message manual contains an
> appropriate setting.

I wouldn't expect users to know how to do that in advance of getting
the various "Re:" variants out there.  Should we perhaps improve the
default value to be less English-centric?

> We could just add the example from the manual to the `defcustom' for
> `message-subject-re-regexp' (and also align it with whatever Rmail
> does).

I think good defaults here win big time.  This corner is notoriously
dark and most users will not know how to make it better.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

>>>>> On Tue, 20 Aug 2024 15:20:54 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Björn Bidar <bjorn.bidar@thaodan.de>,
    >> 72442@debbugs.gnu.org
    >> Date: Tue, 20 Aug 2024 11:46:26 +0200
    >> 
    >> >>>>> On Sat, 03 Aug 2024 20:19:48 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    >> >> From: Björn Bidar <bjorn.bidar@thaodan.de>
    >> >> Cc: 72442@debbugs.gnu.org
    >> >> Date: Sat, 03 Aug 2024 19:58:46 +0300
    >> >> 
    >> >> Eli Zaretskii <eliz@gnu.org> writes:
    >> >> 
    >> >> > FWIW, Rmail does support the various ways of saying "Re:", so the code
    >> >> > could be lifted from there.
    >> 
    >> So does message, you just have to configure
    >> `message-subject-re-regexp'. The message manual contains an
    >> appropriate setting.

    Eli> I wouldn't expect users to know how to do that in advance of getting
    Eli> the various "Re:" variants out there.  Should we perhaps improve the
    Eli> default value to be less English-centric?

Yes, we should.

    >> We could just add the example from the manual to the `defcustom' for
    >> `message-subject-re-regexp' (and also align it with whatever Rmail
    >> does).

    Eli> I think good defaults here win big time.  This corner is notoriously
    Eli> dark and most users will not know how to make it better.

The first is easy. The second is a bit more difficult, since the two
donʼt cover exactly the same cases, and message matches
case-sensitively, unlike rmail, so weʼd need to either adapt eg "ΣΧΕΤ" to
"[Σσ][Χχ][Εε][Ττ]" or make the breaking change of binding
`case-fold-search' in message. Personally Iʼd prefer the latter.

(are there really mailers that prefix the subject with "R:"?)


Robert
-- 

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Francesco Potortì]

>(are there really mailers that prefix the subject with "R:"?)

Yes, I have seen them regularly in the past years.  I have just grepped my personal email backups for July and August and I found 3 different emails with "R:". Ok, it's 3 over some thousands, but anyway...

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

>>>>> On Tue, 20 Aug 2024 15:56:24 +0200, Robert Pluim <rpluim@gmail.com> said:

    Robert> The first is easy. The second is a bit more difficult, since the two
    Robert> donʼt cover exactly the same cases, and message matches
    Robert> case-sensitively, unlike rmail, so weʼd need to either adapt eg "ΣΧΕΤ" to
    Robert> "[Σσ][Χχ][Εε][Ττ]" or make the breaking change of binding
    Robert> `case-fold-search' in message. Personally Iʼd prefer the latter.

    Robert> (are there really mailers that prefix the subject with "R:"?)

The current setting for rmail is more complete than the suggestion in
the Gnus manual, except itʼs missing:

- "RÉ"
- an optional "." after "Antw"
- an optional space between the "RE" string and the ":"
- "FW"/"FWD"

The first 3 seem obvious. Adding the last might annoy people :-)

I can add those to `rmail-re-abbrevs', but perhaps itʼs better to have a
new defcustom that both rmail and message would use? Something like
'mail-re-regexps' in "mail-utils.el" perhaps? Plus some appropriate
editing of the manual.

Robert
-- 

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 20 Aug 2024 15:56:24 +0200, Robert Pluim <rpluim@gmail.com> said:
>
>     Robert> The first is easy. The second is a bit more difficult, since the two
>     Robert> donʼt cover exactly the same cases, and message matches
>     Robert> case-sensitively, unlike rmail, so weʼd need to either adapt eg "ΣΧΕΤ" to
>     Robert> "[Σσ][Χχ][Εε][Ττ]" or make the breaking change of binding
>     Robert> `case-fold-search' in message. Personally Iʼd prefer the latter.
>
>     Robert> (are there really mailers that prefix the subject with "R:"?)
>
> The current setting for rmail is more complete than the suggestion in
> the Gnus manual, except itʼs missing:
>
> - "RÉ"
> - an optional "." after "Antw"
> - an optional space between the "RE" string and the ":"
> - "FW"/"FWD"
>
> The first 3 seem obvious. Adding the last might annoy people :-)
>
> I can add those to `rmail-re-abbrevs', but perhaps itʼs better to have a
> new defcustom that both rmail and message would use? Something like
> 'mail-re-regexps' in "mail-utils.el" perhaps? Plus some appropriate
> editing of the manual.

If there would be an easy way to new prefix to the ignore list that
would work. Can't expect (avarage) users to build their regular
expression especially when multiple languages are involved.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: bjorn.bidar@thaodan.de,  72442@debbugs.gnu.org
> Date: Thu, 22 Aug 2024 13:37:15 +0200
> 
> The current setting for rmail is more complete than the suggestion in
> the Gnus manual, except itʼs missing:
> 
> - "RÉ"
> - an optional "." after "Antw"
> - an optional space between the "RE" string and the ":"
> - "FW"/"FWD"
> 
> The first 3 seem obvious. Adding the last might annoy people :-)
> 
> I can add those to `rmail-re-abbrevs', but perhaps itʼs better to have a
> new defcustom that both rmail and message would use? Something like
> 'mail-re-regexps' in "mail-utils.el" perhaps? Plus some appropriate
> editing of the manual.

Sounds like a good plan to me, thanks.

For backward compatibility, let's leave rmail-re-abbrevs in place, and
just initialize it from that new defcustom.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

>>>>> On Thu, 22 Aug 2024 15:15:03 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> For backward compatibility, let's leave rmail-re-abbrevs in place, and
    Eli> just initialize it from that new defcustom.

Patch attached. Works for gnus for me, and manual testing of the rmail
regexp works. Did you want me to obsolote the rmail and message
variables as well?

Robert
-- 


[-- Attachment #2: 0001-Teach-gnus-message-about-international-Re-variants.patch --]
[-- Type: text/x-diff, Size: 9823 bytes --]

From ae84c547234ab28cc50ae2bb288bd401978ec293 Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Thu, 22 Aug 2024 18:49:12 +0200
Subject: [PATCH] Teach gnus/message about international Re: variants
To: emacs-devel@gnu.org

Bug#72442

* lisp/mail/mail-utils.el (mail-re-regexps): New defcustom, contains the
components used to construct 'rmail-re-abbrevs' and
'message-subject-re-regexp'.
* lisp/gnus/message.el (message-subject-re-regexp): Derive from
'mail-re-regexps'.
(message-strip-subject-re): Make the match case-insensitive.
* lisp/mail/rmail.el (rmail-re-abbrevs): Derive from 'mail-re-regexps'.
Update 'rmail-reply-regexp' when it changes.
(rmail-reply-regexp): Set to nil, 'rmail-re-abbrevs' will set it.

* doc/emacs/rmail.texi (Rmail Reply): Describe 'mail-re-regexps'.
* doc/misc/message.texi (Message Headers): Describe 'mail-re-regexps'.
---
 doc/emacs/rmail.texi    |  8 ++++++++
 doc/misc/message.texi   | 13 ++++++++++++-
 etc/NEWS                | 26 ++++++++++++++++++++++++++
 lisp/gnus/message.el    | 29 ++++++++++++++++++++++-------
 lisp/mail/mail-utils.el | 19 +++++++++++++++++++
 lisp/mail/rmail.el      | 31 ++++++++++++++++++++-----------
 6 files changed, 107 insertions(+), 19 deletions(-)

diff --git a/doc/emacs/rmail.texi b/doc/emacs/rmail.texi
index 9c20d30c442..e090ec6fe76 100644
--- a/doc/emacs/rmail.texi
+++ b/doc/emacs/rmail.texi
@@ -776,6 +776,14 @@ Rmail Reply
 sent the message you received, and the @samp{CC} field starts out with
 all the other recipients of that message.
 
+@vindex rmail-re-abbrevs
+@vindex rmail-reply-prefix
+@vindex mail-re-regexps
+The @samp{Subject} header field may contain one or more instances of
+@samp{Re:} or localized variants thereof.  These are removed if they
+match @code{rmail-re-abbrevs} (which is initialized from
+@code{mail-re-regexps}), and @code{rmail-reply-prefix} is prepended.
+
 @vindex mail-dont-reply-to-names
   You can exclude certain recipients from being included automatically
 in replies, using the variable @code{mail-dont-reply-to-names}.  Its
diff --git a/doc/misc/message.texi b/doc/misc/message.texi
index 468bf81599d..f5399aecee1 100644
--- a/doc/misc/message.texi
+++ b/doc/misc/message.texi
@@ -1693,13 +1693,14 @@ Message Headers
 
 @item message-subject-re-regexp
 @vindex message-subject-re-regexp
+@vindex mail-re-regexps
 @cindex Aw
 @cindex Sv
 @cindex Re
 Responses to messages have subjects that start with @samp{Re: }.  This
 is @emph{not} an abbreviation of the English word ``response'', but it
 comes from the Latin ``res'', and means ``in the matter of''.  Some
-illiterate nincompoops have failed to grasp this fact, and have
+standards-challenged companies have failed to grasp this fact, and have
 ``internationalized'' their software to use abominations like
 @samp{Aw: } (``antwort'') or @samp{Sv: } (``svar'') instead, which is
 meaningless and evil.  However, you may have to deal with users that
@@ -1731,6 +1732,16 @@ Message Headers
        ))
 @end lisp
 
+You shouldn't need to do this, since the default value of
+@code{message-subject-re-regexp} is initialized based on
+@code{mail-re-regexps}, which covers most known cases of such
+internationalization, and is a lot easier to customize.  Customizing
+@code{mail-re-regexps} updates @code{message-subject-re-regexp} to
+match.
+
+Note that the regexp is matched case-insensitively against the
+@samp{Subject} header contents.
+
 @item message-subject-trailing-was-query
 @vindex message-subject-trailing-was-query
 @vindex message-subject-trailing-was-ask-regexp
diff --git a/etc/NEWS b/etc/NEWS
index 07d1cce0966..82c0eac4e5c 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -162,6 +162,32 @@ By passing '-t' or '--timeout', you can specify a maximum time to wait
 for the processes to exit.  Additionally, you can now wait for external
 processes by passing their PIDs.
 
++++
+** Mail-util
+
+*** New user option 'mail-re-regexps'.
+This contains the list of regular expressions used to match "Re:" and
+international variants of it when modifying the Subject field in
+replies.
+
++++
+** Rmail
+
+*** 'rmail-re-abbrevs' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'rmail-re-abbrevs'.  'rmail-re-abbrevs' is still honored if it was
+already set.
+
++++
+** Message
+
+*** 'message-subject-re-regexp' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'message-subject-re-regexp'.  'message-subject-re-regexp' is still
+honored if it was already set.
+
+*** 'message-strip-subject-re' now matches case-insensitively.
+
 ** SHR
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index 98f63571389..081c6986a66 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -305,11 +305,24 @@ message-ignored-supersedes-headers
 		 regexp))
 
 (defcustom message-subject-re-regexp
-  "^[ \t]*\\([Rr][Ee]\\(\\[[0-9]*\\]\\)* ?:[ \t]*\\)*[ \t]*"
-  "Regexp matching \"Re: \" in the subject line."
+  (concat "^[ \t]*"
+          "\\("
+          "\\("
+          (string-join mail-re-regexps "\\|")
+          "\\)"
+          "\\(\\[[0-9]*\\]\\)*"
+          " ?\u00a0*[:：][ \t]*"
+          "\\)"
+          "*[ \t]*")
+  "Regexp matching \"Re: \" in the subject line.
+Matching is done case-insensitively.
+Initialized from the value of `mail-re-regexps', which is easier to
+customize."
   :group 'message-various
   :link '(custom-manual "(message)Message Headers")
-  :type 'regexp)
+  :type 'regexp
+  :set-after '(mail-re-regexps)
+  :version "31.1")
 
 (defcustom message-screenshot-command '("import" "png:-")
   "Command to take a screenshot.
@@ -2257,10 +2270,12 @@ message-strip-list-identifiers
       subject)))
 
 (defun message-strip-subject-re (subject)
-  "Remove \"Re:\" from subject lines in string SUBJECT."
-  (if (string-match message-subject-re-regexp subject)
-      (substring subject (match-end 0))
-    subject))
+  "Remove \"Re:\" from subject lines in string SUBJECT.
+This uses `mail-re-regexps, matching is done case-insensitively."
+  (let ((case-fold-search t))
+    (if (string-match message-subject-re-regexp subject)
+        (substring subject (match-end 0))
+      subject)))
 
 (defcustom message-replacement-char "."
   "Replacement character used instead of unprintable or not decodable chars."
diff --git a/lisp/mail/mail-utils.el b/lisp/mail/mail-utils.el
index c9e4d0b3812..a04092b5727 100644
--- a/lisp/mail/mail-utils.el
+++ b/lisp/mail/mail-utils.el
@@ -46,6 +46,25 @@ mail-dont-reply-to-names
   :type '(choice regexp (const :tag "Your Name" nil))
   :group 'mail)
 
+;;;###autoload
+(defcustom mail-re-regexps
+  '("RE" "R\u00c9\\.?" "FWD?" "رد" "回复" "回覆" "SV" "Antw\\.?"
+  "VS" "REF" "AW" "ΑΠ" "ΣΧΕΤ" "השב" "Vá" "R" "RIF" "BLS" "RES"
+  "Odp" "YNT" "ATB")
+  "List of localized \"Re:\" abbreviations in various languages.
+Each component can be a regular expression or a simple string.  Matching
+is done case-insensitively.  Used to initialize the legacy
+`rmail-re-abbrevs' and `message-subject-re-regexp' user options."
+  :type '(repeat regexp)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (dolist (sym '(rmail-re-abbrevs
+                        message-subject-re-regexp))
+           (when (get sym 'standard-value)
+             (custom-reevaluate-setting sym))))
+  :group 'mail
+  :version "31.1")
+
 (defvar epa-inhibit)
 ;; Returns t if file FILE is an Rmail file.
 ;;;###autoload
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index e38ab12fae6..4965907916c 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -527,23 +527,32 @@ rmail-automatic-folder-directives
 (defvar rmail-reply-prefix "Re: "
   "String to prepend to Subject line when replying to a message.")
 
-;; Note: this is matched with case-fold-search bound to t.
-(defcustom rmail-re-abbrevs
-  "\\(RE\\|رد\\|回复\\|回覆\\|SV\\|Antw\\|VS\\|REF\\|AW\\|ΑΠ\\|ΣΧΕΤ\\|השב\\|Vá\\|R\\|RIF\\|BLS\\|RES\\|Odp\\|YNT\\|ATB\\)"
-  "Regexp with localized \"Re:\" abbreviations in various languages."
-  :version "28.1"
-  :type 'regexp)
-
 ;; Some mailers use "Re(2):" or "Re^2:" or "Re: Re:" or "Re[2]:".
 ;; This pattern should catch all the common variants.
 ;; rms: I deleted the change to delete tags in square brackets
 ;; because they mess up RT tags.
-(defvar rmail-reply-regexp
-  (concat "\\`\\("
-          rmail-re-abbrevs
-          "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?\u00a0*[:：] *\\)*")
+(defvar rmail-reply-regexp nil ;; set by `rmail-re-abbrevs
   "Regexp to delete from Subject line before inserting `rmail-reply-prefix'.")
 
+;; Note: this is matched with case-fold-search bound to t.
+(defcustom rmail-re-abbrevs
+  (concat "\\("
+          (string-join mail-re-regexps "\\|")
+          "\\)")
+  "Regexp with localized \"Re:\" abbreviations in various languages.
+Matching is done case-insensitively.
+Initialized from `mail-re-regexps', which is easier to customize."
+  :set-after '(mail-re-regexps)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (setq rmail-reply-regexp
+               (concat "\\`[ \t]*\\("
+                       val
+                       "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
+                       " ?\u00a0*[:：][ \t]*\\)*[ \t]*")))
+  :type 'regexp
+  :version "31.1")
+
 (defcustom rmail-display-summary nil
   "If non-nil, Rmail always displays the summary buffer."
   :group 'rmail-summary
-- 
2.39.2

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: bjorn.bidar@thaodan.de,  72442@debbugs.gnu.org
> Date: Tue, 27 Aug 2024 17:50:41 +0200
> 
> >>>>> On Thu, 22 Aug 2024 15:15:03 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> For backward compatibility, let's leave rmail-re-abbrevs in place, and
>     Eli> just initialize it from that new defcustom.
> 
> Patch attached. Works for gnus for me, and manual testing of the rmail
> regexp works.

Thanks.

> Did you want me to obsolote the rmail and message variables as well?

I see no reason to obsolete them.

> +          "\\(\\[[0-9]*\\]\\)*"
> +          " ?\u00a0*[:：][ \t]*"

These two parts could use a comment that explains what they handle.

> +                       "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
> +                       " ?\u00a0*[:：][ \t]*\\)*[ \t]*")))

Same here.

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

>>>>> Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 27 Aug 2024 19:39:28 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >>> From: Robert Pluim <rpluim@gmail.com>
    >>> Cc: bjorn.bidar@thaodan.de,  72442@debbugs.gnu.org
    >>> Date: Tue, 27 Aug 2024 17:50:41 +0200
    >>> 
    >>> >>>>> On Thu, 22 Aug 2024 15:15:03 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >>> 
    Eli> For backward compatibility, let's leave rmail-re-abbrevs in place, and
    Eli> just initialize it from that new defcustom.
    >>>

OK
    >>> +          "\\(\\[[0-9]*\\]\\)*"
    >>> +          " ?\u00a0*[:：][ \t]*"

    Eli> These two parts could use a comment that explains what they handle.

    >>> +                       "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
    >>> +                       " ?\u00a0*[:：][ \t]*\\)*[ \t]*")))

    Eli> Same here.

Done. The rmail and gnus code now use exactly the same final regexp
derived from the new user option. Patch attached, itʼs been working
fine for me in Gnus, but somebody who uses rmail should give it a go.

Robert
-- 


[-- Attachment #2: 0001-Teach-gnus-message-about-international-Re-variants.patch --]
[-- Type: text/x-diff, Size: 9951 bytes --]

From 2452929bc2a615a93a626c06ac8b70e726789fed Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Thu, 22 Aug 2024 18:49:12 +0200
Subject: [PATCH] Teach gnus/message about international Re: variants
To: emacs-devel@gnu.org

Bug#72442

* lisp/mail/mail-utils.el (mail-re-regexps): New defcustom, contains the
components used to construct 'rmail-re-abbrevs' and
'message-subject-re-regexp'.
* lisp/gnus/message.el (message-subject-re-regexp): Derive from
'mail-re-regexps'.
(message-strip-subject-re): Make the match case-insensitive.
* lisp/mail/rmail.el (rmail-re-abbrevs): Derive from 'mail-re-regexps'.
Update 'rmail-reply-regexp' when it changes.
(rmail-reply-regexp): Set to nil, 'rmail-re-abbrevs' will set it.

* doc/emacs/rmail.texi (Rmail Reply): Describe 'mail-re-regexps'.
* doc/misc/message.texi (Message Headers): Describe 'mail-re-regexps'.
---
 doc/emacs/rmail.texi    |  8 ++++++++
 doc/misc/message.texi   | 13 ++++++++++++-
 etc/NEWS                | 26 ++++++++++++++++++++++++++
 lisp/gnus/message.el    | 25 ++++++++++++++++++-------
 lisp/mail/mail-utils.el | 31 +++++++++++++++++++++++++++++++
 lisp/mail/rmail.el      | 27 ++++++++++++++++-----------
 6 files changed, 111 insertions(+), 19 deletions(-)

diff --git a/doc/emacs/rmail.texi b/doc/emacs/rmail.texi
index 9c20d30c442..e090ec6fe76 100644
--- a/doc/emacs/rmail.texi
+++ b/doc/emacs/rmail.texi
@@ -776,6 +776,14 @@ Rmail Reply
 sent the message you received, and the @samp{CC} field starts out with
 all the other recipients of that message.
 
+@vindex rmail-re-abbrevs
+@vindex rmail-reply-prefix
+@vindex mail-re-regexps
+The @samp{Subject} header field may contain one or more instances of
+@samp{Re:} or localized variants thereof.  These are removed if they
+match @code{rmail-re-abbrevs} (which is initialized from
+@code{mail-re-regexps}), and @code{rmail-reply-prefix} is prepended.
+
 @vindex mail-dont-reply-to-names
   You can exclude certain recipients from being included automatically
 in replies, using the variable @code{mail-dont-reply-to-names}.  Its
diff --git a/doc/misc/message.texi b/doc/misc/message.texi
index 468bf81599d..f5399aecee1 100644
--- a/doc/misc/message.texi
+++ b/doc/misc/message.texi
@@ -1693,13 +1693,14 @@ Message Headers
 
 @item message-subject-re-regexp
 @vindex message-subject-re-regexp
+@vindex mail-re-regexps
 @cindex Aw
 @cindex Sv
 @cindex Re
 Responses to messages have subjects that start with @samp{Re: }.  This
 is @emph{not} an abbreviation of the English word ``response'', but it
 comes from the Latin ``res'', and means ``in the matter of''.  Some
-illiterate nincompoops have failed to grasp this fact, and have
+standards-challenged companies have failed to grasp this fact, and have
 ``internationalized'' their software to use abominations like
 @samp{Aw: } (``antwort'') or @samp{Sv: } (``svar'') instead, which is
 meaningless and evil.  However, you may have to deal with users that
@@ -1731,6 +1732,16 @@ Message Headers
        ))
 @end lisp
 
+You shouldn't need to do this, since the default value of
+@code{message-subject-re-regexp} is initialized based on
+@code{mail-re-regexps}, which covers most known cases of such
+internationalization, and is a lot easier to customize.  Customizing
+@code{mail-re-regexps} updates @code{message-subject-re-regexp} to
+match.
+
+Note that the regexp is matched case-insensitively against the
+@samp{Subject} header contents.
+
 @item message-subject-trailing-was-query
 @vindex message-subject-trailing-was-query
 @vindex message-subject-trailing-was-ask-regexp
diff --git a/etc/NEWS b/etc/NEWS
index 8bd5c7c9515..35de85224d9 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -194,6 +194,32 @@ By passing '-t' or '--timeout', you can specify a maximum time to wait
 for the processes to exit.  Additionally, you can now wait for external
 processes by passing their PIDs.
 
++++
+** Mail-util
+
+*** New user option 'mail-re-regexps'.
+This contains the list of regular expressions used to match "Re:" and
+international variants of it when modifying the Subject field in
+replies.
+
++++
+** Rmail
+
+*** 'rmail-re-abbrevs' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'rmail-re-abbrevs'.  'rmail-re-abbrevs' is still honored if it was
+already set.
+
++++
+** Message
+
+*** 'message-subject-re-regexp' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'message-subject-re-regexp'.  'message-subject-re-regexp' is still
+honored if it was already set.
+
+*** 'message-strip-subject-re' now matches case-insensitively.
+
 ** SHR
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index d52ed9662a7..ad0c4d1f3f8 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -305,11 +305,20 @@ message-ignored-supersedes-headers
 		 regexp))
 
 (defcustom message-subject-re-regexp
-  "^[ \t]*\\([Rr][Ee]\\(\\[[0-9]*\\]\\)* ?:[ \t]*\\)*[ \t]*"
-  "Regexp matching \"Re: \" in the subject line."
+  (mail--wrap-re-regexp
+   (concat
+    "\\("
+    (string-join mail-re-regexps "\\|")
+    "\\)"))
+  "Regexp matching \"Re: \" in the subject line.
+Matching is done case-insensitively.
+Initialized from the value of `mail-re-regexps', which is easier to
+customize."
   :group 'message-various
   :link '(custom-manual "(message)Message Headers")
-  :type 'regexp)
+  :type 'regexp
+  :set-after '(mail-re-regexps)
+  :version "31.1")
 
 (defcustom message-screenshot-command '("import" "png:-")
   "Command to take a screenshot.
@@ -2257,10 +2266,12 @@ message-strip-list-identifiers
       subject)))
 
 (defun message-strip-subject-re (subject)
-  "Remove \"Re:\" from subject lines in string SUBJECT."
-  (if (string-match message-subject-re-regexp subject)
-      (substring subject (match-end 0))
-    subject))
+  "Remove \"Re:\" from subject lines in string SUBJECT.
+This uses `mail-re-regexps, matching is done case-insensitively."
+  (let ((case-fold-search t))
+    (if (string-match message-subject-re-regexp subject)
+        (substring subject (match-end 0))
+      subject)))
 
 (defcustom message-replacement-char "."
   "Replacement character used instead of unprintable or not decodable chars."
diff --git a/lisp/mail/mail-utils.el b/lisp/mail/mail-utils.el
index c9e4d0b3812..cfac56c0125 100644
--- a/lisp/mail/mail-utils.el
+++ b/lisp/mail/mail-utils.el
@@ -46,6 +46,37 @@ mail-dont-reply-to-names
   :type '(choice regexp (const :tag "Your Name" nil))
   :group 'mail)
 
+(defun mail--wrap-re-regexp (re)
+  (concat "\\`[ \t]*"
+          "\\("
+          re
+          ; Re(1) or Re[1] or Re^1
+          "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
+          ; SPC/NBSP followed by colon and TAB/SPC
+          " ?\u00a0*[:：][ \t]*"
+          ; Handle repetition, eg "Re[1]: Re[2]:"
+          "\\)*"
+          "[ \t]*"))
+
+;;;###autoload
+(defcustom mail-re-regexps
+  '("RE" "R\u00c9\\.?" "FWD?" "رد" "回复" "回覆" "SV" "Antw\\.?"
+  "VS" "REF" "AW" "ΑΠ" "ΣΧΕΤ" "השב" "Vá" "R" "RIF" "BLS" "RES"
+  "Odp" "YNT" "ATB")
+  "List of localized \"Re:\" abbreviations in various languages.
+Each component can be a regular expression or a simple string.  Matching
+is done case-insensitively.  Used to initialize the legacy
+`rmail-re-abbrevs' and `message-subject-re-regexp' user options."
+  :type '(repeat regexp)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (dolist (sym '(rmail-re-abbrevs
+                        message-subject-re-regexp))
+           (when (get sym 'standard-value)
+             (custom-reevaluate-setting sym))))
+  :group 'mail
+  :version "31.1")
+
 (defvar epa-inhibit)
 ;; Returns t if file FILE is an Rmail file.
 ;;;###autoload
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index e38ab12fae6..7fc5363f30c 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -527,22 +527,27 @@ rmail-automatic-folder-directives
 (defvar rmail-reply-prefix "Re: "
   "String to prepend to Subject line when replying to a message.")
 
-;; Note: this is matched with case-fold-search bound to t.
-(defcustom rmail-re-abbrevs
-  "\\(RE\\|رد\\|回复\\|回覆\\|SV\\|Antw\\|VS\\|REF\\|AW\\|ΑΠ\\|ΣΧΕΤ\\|השב\\|Vá\\|R\\|RIF\\|BLS\\|RES\\|Odp\\|YNT\\|ATB\\)"
-  "Regexp with localized \"Re:\" abbreviations in various languages."
-  :version "28.1"
-  :type 'regexp)
+(defvar rmail-reply-regexp nil ;; set by `rmail-re-abbrevs
+  "Regexp to delete from Subject line before inserting `rmail-reply-prefix'.")
 
 ;; Some mailers use "Re(2):" or "Re^2:" or "Re: Re:" or "Re[2]:".
 ;; This pattern should catch all the common variants.
 ;; rms: I deleted the change to delete tags in square brackets
 ;; because they mess up RT tags.
-(defvar rmail-reply-regexp
-  (concat "\\`\\("
-          rmail-re-abbrevs
-          "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?\u00a0*[:：] *\\)*")
-  "Regexp to delete from Subject line before inserting `rmail-reply-prefix'.")
+;; Note: this is matched with case-fold-search bound to t.
+(defcustom rmail-re-abbrevs
+  (concat "\\("
+          (string-join mail-re-regexps "\\|")
+          "\\)")
+  "Regexp with localized \"Re:\" abbreviations in various languages.
+Matching is done case-insensitively.
+Initialized from `mail-re-regexps', which is easier to customize."
+  :set-after '(mail-re-regexps)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (setq rmail-reply-regexp (mail--wrap-re-regexp val)))
+  :type 'regexp
+  :version "31.1")
 
 (defcustom rmail-display-summary nil
   "If non-nil, Rmail always displays the summary buffer."
-- 
2.39.2

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

>>>>> On Thu, 22 Aug 2024 15:04:56 +0300, Björn Bidar <bjorn.bidar@thaodan.de> said:

    Björn> Robert Pluim <rpluim@gmail.com> writes:
    >>>>>>> On Tue, 20 Aug 2024 15:56:24 +0200, Robert Pluim <rpluim@gmail.com> said:
    >> 
    Robert> The first is easy. The second is a bit more difficult, since the two
    Robert> donʼt cover exactly the same cases, and message matches
    Robert> case-sensitively, unlike rmail, so weʼd need to either adapt eg "ΣΧΕΤ" to
    Robert> "[Σσ][Χχ][Εε][Ττ]" or make the breaking change of binding
    Robert> `case-fold-search' in message. Personally Iʼd prefer the latter.
    >> 
    Robert> (are there really mailers that prefix the subject with "R:"?)
    >> 
    >> The current setting for rmail is more complete than the suggestion in
    >> the Gnus manual, except itʼs missing:
    >> 
    >> - "RÉ"
    >> - an optional "." after "Antw"
    >> - an optional space between the "RE" string and the ":"
    >> - "FW"/"FWD"
    >> 
    >> The first 3 seem obvious. Adding the last might annoy people :-)
    >> 
    >> I can add those to `rmail-re-abbrevs', but perhaps itʼs better to have a
    >> new defcustom that both rmail and message would use? Something like
    >> 'mail-re-regexps' in "mail-utils.el" perhaps? Plus some appropriate
    >> editing of the manual.

    Björn> If there would be an easy way to new prefix to the ignore list that
    Björn> would work. Can't expect (avarage) users to build their regular
    Björn> expression especially when multiple languages are involved.

The new defcustom could be a list of regexps, ie

  ("RE" "R\u00c9\\.?" "FWD?" "رد" "回复" "回覆" "SV" "Antw\\.?"
  "VS" "REF" "AW" ΑΠ" "ΣΧΕΤ" "השב" "Vá" "R" "RIF" "BLS" "RES"
  "Odp" "YNT" "ATB")

which weʼd concatenate with "\\|" to get the real regexp. That would
be quite easy to edit in custom.

Robert
-- 

bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Björn Bidar via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Thu, 22 Aug 2024 15:04:56 +0300, Björn Bidar <bjorn.bidar@thaodan.de> said:
>
>     Björn> Robert Pluim <rpluim@gmail.com> writes:
>     >>>>>>> On Tue, 20 Aug 2024 15:56:24 +0200, Robert Pluim <rpluim@gmail.com> said:
>     >> 
>     Robert> The first is easy. The second is a bit more difficult, since the two
>     Robert> donʼt cover exactly the same cases, and message matches
>     Robert> case-sensitively, unlike rmail, so weʼd need to either adapt eg "ΣΧΕΤ" to
>     Robert> "[Σσ][Χχ][Εε][Ττ]" or make the breaking change of binding
>     Robert> `case-fold-search' in message. Personally Iʼd prefer the latter.
>     >> 
>     Robert> (are there really mailers that prefix the subject with "R:"?)
>     >> 
>     >> The current setting for rmail is more complete than the suggestion in
>     >> the Gnus manual, except itʼs missing:
>     >> 
>     >> - "RÉ"
>     >> - an optional "." after "Antw"
>     >> - an optional space between the "RE" string and the ":"
>     >> - "FW"/"FWD"
>     >> 
>     >> The first 3 seem obvious. Adding the last might annoy people :-)
>     >> 
>     >> I can add those to `rmail-re-abbrevs', but perhaps itʼs better to have a
>     >> new defcustom that both rmail and message would use? Something like
>     >> 'mail-re-regexps' in "mail-utils.el" perhaps? Plus some appropriate
>     >> editing of the manual.
>
>     Björn> If there would be an easy way to new prefix to the ignore list that
>     Björn> would work. Can't expect (avarage) users to build their regular
>     Björn> expression especially when multiple languages are involved.
>
> The new defcustom could be a list of regexps, ie
>
>   ("RE" "R\u00c9\\.?" "FWD?" "رد" "回复" "回覆" "SV" "Antw\\.?"
>   "VS" "REF" "AW" ΑΠ" "ΣΧΕΤ" "השב" "Vá" "R" "RIF" "BLS" "RES"
>   "Odp" "YNT" "ATB")
>
> which weʼd concatenate with "\\|" to get the real regexp. That would
> be quite easy to edit in custom.

Does it make sense to build the regexp using regexp-search-word or
regexp-opt after the user has entered the words to filter in defcustom?

bug#72442: : bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim]

>>>>> On Fri, 23 Aug 2024 08:39:25 +0300, Björn Bidar <bjorn.bidar@thaodan.de> said:

    Björn> Does it make sense to build the regexp using regexp-search-word or
    Björn> regexp-opt after the user has entered the words to filter in defcustom?

Not really. `regexp-opt' only works with fixed strings, and the
components here are regexps. I donʼt know what `regexp-search-word' is.

Robert
--