bug#72442: 31.0.50; Gnus or message.el doesn't handle international message reply subject prefixes and insults users

[Robert Pluim <rpluim@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

>>>>> Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 27 Aug 2024 19:39:28 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >>> From: Robert Pluim <rpluim@gmail.com>
    >>> Cc: bjorn.bidar@thaodan.de,  72442@debbugs.gnu.org
    >>> Date: Tue, 27 Aug 2024 17:50:41 +0200
    >>> 
    >>> >>>>> On Thu, 22 Aug 2024 15:15:03 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >>> 
    Eli> For backward compatibility, let's leave rmail-re-abbrevs in place, and
    Eli> just initialize it from that new defcustom.
    >>>

OK
    >>> +          "\\(\\[[0-9]*\\]\\)*"
    >>> +          " ?\u00a0*[:：][ \t]*"

    Eli> These two parts could use a comment that explains what they handle.

    >>> +                       "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
    >>> +                       " ?\u00a0*[:：][ \t]*\\)*[ \t]*")))

    Eli> Same here.

Done. The rmail and gnus code now use exactly the same final regexp
derived from the new user option. Patch attached, itʼs been working
fine for me in Gnus, but somebody who uses rmail should give it a go.

Robert
-- 


[-- Attachment #2: 0001-Teach-gnus-message-about-international-Re-variants.patch --]
[-- Type: text/x-diff, Size: 9951 bytes --]

From 2452929bc2a615a93a626c06ac8b70e726789fed Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Thu, 22 Aug 2024 18:49:12 +0200
Subject: [PATCH] Teach gnus/message about international Re: variants
To: emacs-devel@gnu.org

Bug#72442

* lisp/mail/mail-utils.el (mail-re-regexps): New defcustom, contains the
components used to construct 'rmail-re-abbrevs' and
'message-subject-re-regexp'.
* lisp/gnus/message.el (message-subject-re-regexp): Derive from
'mail-re-regexps'.
(message-strip-subject-re): Make the match case-insensitive.
* lisp/mail/rmail.el (rmail-re-abbrevs): Derive from 'mail-re-regexps'.
Update 'rmail-reply-regexp' when it changes.
(rmail-reply-regexp): Set to nil, 'rmail-re-abbrevs' will set it.

* doc/emacs/rmail.texi (Rmail Reply): Describe 'mail-re-regexps'.
* doc/misc/message.texi (Message Headers): Describe 'mail-re-regexps'.
---
 doc/emacs/rmail.texi    |  8 ++++++++
 doc/misc/message.texi   | 13 ++++++++++++-
 etc/NEWS                | 26 ++++++++++++++++++++++++++
 lisp/gnus/message.el    | 25 ++++++++++++++++++-------
 lisp/mail/mail-utils.el | 31 +++++++++++++++++++++++++++++++
 lisp/mail/rmail.el      | 27 ++++++++++++++++-----------
 6 files changed, 111 insertions(+), 19 deletions(-)

diff --git a/doc/emacs/rmail.texi b/doc/emacs/rmail.texi
index 9c20d30c442..e090ec6fe76 100644
--- a/doc/emacs/rmail.texi
+++ b/doc/emacs/rmail.texi
@@ -776,6 +776,14 @@ Rmail Reply
 sent the message you received, and the @samp{CC} field starts out with
 all the other recipients of that message.
 
+@vindex rmail-re-abbrevs
+@vindex rmail-reply-prefix
+@vindex mail-re-regexps
+The @samp{Subject} header field may contain one or more instances of
+@samp{Re:} or localized variants thereof.  These are removed if they
+match @code{rmail-re-abbrevs} (which is initialized from
+@code{mail-re-regexps}), and @code{rmail-reply-prefix} is prepended.
+
 @vindex mail-dont-reply-to-names
   You can exclude certain recipients from being included automatically
 in replies, using the variable @code{mail-dont-reply-to-names}.  Its
diff --git a/doc/misc/message.texi b/doc/misc/message.texi
index 468bf81599d..f5399aecee1 100644
--- a/doc/misc/message.texi
+++ b/doc/misc/message.texi
@@ -1693,13 +1693,14 @@ Message Headers
 
 @item message-subject-re-regexp
 @vindex message-subject-re-regexp
+@vindex mail-re-regexps
 @cindex Aw
 @cindex Sv
 @cindex Re
 Responses to messages have subjects that start with @samp{Re: }.  This
 is @emph{not} an abbreviation of the English word ``response'', but it
 comes from the Latin ``res'', and means ``in the matter of''.  Some
-illiterate nincompoops have failed to grasp this fact, and have
+standards-challenged companies have failed to grasp this fact, and have
 ``internationalized'' their software to use abominations like
 @samp{Aw: } (``antwort'') or @samp{Sv: } (``svar'') instead, which is
 meaningless and evil.  However, you may have to deal with users that
@@ -1731,6 +1732,16 @@ Message Headers
        ))
 @end lisp
 
+You shouldn't need to do this, since the default value of
+@code{message-subject-re-regexp} is initialized based on
+@code{mail-re-regexps}, which covers most known cases of such
+internationalization, and is a lot easier to customize.  Customizing
+@code{mail-re-regexps} updates @code{message-subject-re-regexp} to
+match.
+
+Note that the regexp is matched case-insensitively against the
+@samp{Subject} header contents.
+
 @item message-subject-trailing-was-query
 @vindex message-subject-trailing-was-query
 @vindex message-subject-trailing-was-ask-regexp
diff --git a/etc/NEWS b/etc/NEWS
index 8bd5c7c9515..35de85224d9 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -194,6 +194,32 @@ By passing '-t' or '--timeout', you can specify a maximum time to wait
 for the processes to exit.  Additionally, you can now wait for external
 processes by passing their PIDs.
 
++++
+** Mail-util
+
+*** New user option 'mail-re-regexps'.
+This contains the list of regular expressions used to match "Re:" and
+international variants of it when modifying the Subject field in
+replies.
+
++++
+** Rmail
+
+*** 'rmail-re-abbrevs' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'rmail-re-abbrevs'.  'rmail-re-abbrevs' is still honored if it was
+already set.
+
++++
+** Message
+
+*** 'message-subject-re-regexp' default value is now derived from 'mail-re-regexps'.
+'mail-re-regexps' is a new user option that is easier to customize than
+'message-subject-re-regexp'.  'message-subject-re-regexp' is still
+honored if it was already set.
+
+*** 'message-strip-subject-re' now matches case-insensitively.
+
 ** SHR
 
 +++
diff --git a/lisp/gnus/message.el b/lisp/gnus/message.el
index d52ed9662a7..ad0c4d1f3f8 100644
--- a/lisp/gnus/message.el
+++ b/lisp/gnus/message.el
@@ -305,11 +305,20 @@ message-ignored-supersedes-headers
 		 regexp))
 
 (defcustom message-subject-re-regexp
-  "^[ \t]*\\([Rr][Ee]\\(\\[[0-9]*\\]\\)* ?:[ \t]*\\)*[ \t]*"
-  "Regexp matching \"Re: \" in the subject line."
+  (mail--wrap-re-regexp
+   (concat
+    "\\("
+    (string-join mail-re-regexps "\\|")
+    "\\)"))
+  "Regexp matching \"Re: \" in the subject line.
+Matching is done case-insensitively.
+Initialized from the value of `mail-re-regexps', which is easier to
+customize."
   :group 'message-various
   :link '(custom-manual "(message)Message Headers")
-  :type 'regexp)
+  :type 'regexp
+  :set-after '(mail-re-regexps)
+  :version "31.1")
 
 (defcustom message-screenshot-command '("import" "png:-")
   "Command to take a screenshot.
@@ -2257,10 +2266,12 @@ message-strip-list-identifiers
       subject)))
 
 (defun message-strip-subject-re (subject)
-  "Remove \"Re:\" from subject lines in string SUBJECT."
-  (if (string-match message-subject-re-regexp subject)
-      (substring subject (match-end 0))
-    subject))
+  "Remove \"Re:\" from subject lines in string SUBJECT.
+This uses `mail-re-regexps, matching is done case-insensitively."
+  (let ((case-fold-search t))
+    (if (string-match message-subject-re-regexp subject)
+        (substring subject (match-end 0))
+      subject)))
 
 (defcustom message-replacement-char "."
   "Replacement character used instead of unprintable or not decodable chars."
diff --git a/lisp/mail/mail-utils.el b/lisp/mail/mail-utils.el
index c9e4d0b3812..cfac56c0125 100644
--- a/lisp/mail/mail-utils.el
+++ b/lisp/mail/mail-utils.el
@@ -46,6 +46,37 @@ mail-dont-reply-to-names
   :type '(choice regexp (const :tag "Your Name" nil))
   :group 'mail)
 
+(defun mail--wrap-re-regexp (re)
+  (concat "\\`[ \t]*"
+          "\\("
+          re
+          ; Re(1) or Re[1] or Re^1
+          "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?"
+          ; SPC/NBSP followed by colon and TAB/SPC
+          " ?\u00a0*[:：][ \t]*"
+          ; Handle repetition, eg "Re[1]: Re[2]:"
+          "\\)*"
+          "[ \t]*"))
+
+;;;###autoload
+(defcustom mail-re-regexps
+  '("RE" "R\u00c9\\.?" "FWD?" "رد" "回复" "回覆" "SV" "Antw\\.?"
+  "VS" "REF" "AW" "ΑΠ" "ΣΧΕΤ" "השב" "Vá" "R" "RIF" "BLS" "RES"
+  "Odp" "YNT" "ATB")
+  "List of localized \"Re:\" abbreviations in various languages.
+Each component can be a regular expression or a simple string.  Matching
+is done case-insensitively.  Used to initialize the legacy
+`rmail-re-abbrevs' and `message-subject-re-regexp' user options."
+  :type '(repeat regexp)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (dolist (sym '(rmail-re-abbrevs
+                        message-subject-re-regexp))
+           (when (get sym 'standard-value)
+             (custom-reevaluate-setting sym))))
+  :group 'mail
+  :version "31.1")
+
 (defvar epa-inhibit)
 ;; Returns t if file FILE is an Rmail file.
 ;;;###autoload
diff --git a/lisp/mail/rmail.el b/lisp/mail/rmail.el
index e38ab12fae6..7fc5363f30c 100644
--- a/lisp/mail/rmail.el
+++ b/lisp/mail/rmail.el
@@ -527,22 +527,27 @@ rmail-automatic-folder-directives
 (defvar rmail-reply-prefix "Re: "
   "String to prepend to Subject line when replying to a message.")
 
-;; Note: this is matched with case-fold-search bound to t.
-(defcustom rmail-re-abbrevs
-  "\\(RE\\|رد\\|回复\\|回覆\\|SV\\|Antw\\|VS\\|REF\\|AW\\|ΑΠ\\|ΣΧΕΤ\\|השב\\|Vá\\|R\\|RIF\\|BLS\\|RES\\|Odp\\|YNT\\|ATB\\)"
-  "Regexp with localized \"Re:\" abbreviations in various languages."
-  :version "28.1"
-  :type 'regexp)
+(defvar rmail-reply-regexp nil ;; set by `rmail-re-abbrevs
+  "Regexp to delete from Subject line before inserting `rmail-reply-prefix'.")
 
 ;; Some mailers use "Re(2):" or "Re^2:" or "Re: Re:" or "Re[2]:".
 ;; This pattern should catch all the common variants.
 ;; rms: I deleted the change to delete tags in square brackets
 ;; because they mess up RT tags.
-(defvar rmail-reply-regexp
-  (concat "\\`\\("
-          rmail-re-abbrevs
-          "\\(([0-9]+)\\|\\[[0-9]+\\]\\|\\^[0-9]+\\)?\u00a0*[:：] *\\)*")
-  "Regexp to delete from Subject line before inserting `rmail-reply-prefix'.")
+;; Note: this is matched with case-fold-search bound to t.
+(defcustom rmail-re-abbrevs
+  (concat "\\("
+          (string-join mail-re-regexps "\\|")
+          "\\)")
+  "Regexp with localized \"Re:\" abbreviations in various languages.
+Matching is done case-insensitively.
+Initialized from `mail-re-regexps', which is easier to customize."
+  :set-after '(mail-re-regexps)
+  :set (lambda (sym val)
+         (custom-set-default sym val)
+         (setq rmail-reply-regexp (mail--wrap-re-regexp val)))
+  :type 'regexp
+  :version "31.1")
 
 (defcustom rmail-display-summary nil
   "If non-nil, Rmail always displays the summary buffer."
-- 
2.39.2