bug#63332: 28.2.50; pop3.el does not support UTF-8 (RFC 6856)

bug#63332: 28.2.50; pop3.el does not support UTF-8 (RFC 6856)

[Benjamin Riefenstahl]

[-- Attachment #1: Type: text/plain, Size: 1314 bytes --]

I run Gnus.  I get my mail from a POP3 server (courier-pop
1.0.16-3build3 on Ubuntu 22.4).  The server returns the following kind
of mail when there is a non-ASCII character somewhere in the headers of
the mail:

    From: Mail Delivery Subsystem <postmaster>
    Subject: Cannot display Unicode content
    Attachment: [2. message/global]...

    This E-mail message was determined to be Unicode-formatted
    but your E-mail reader does not support Unicode E-mail.

    Please use an E-mail reader that supports POP3 with UTF-8
    (see https://tools.ietf.org/html/rfc6856.html).

    This can also happen when the sender's E-mail program does not
    correctly format the sent message.

    The original message is included as a separate attachment
    so that it can be downloaded manually.

    [2. message/global]...

I can provide a full message, if needed.

Skimming RFC 6856, it looks like when the mail server provides the
capability "UTF8", the client should issue the command "UTF8" to
indicate that it is prepared to receive that encoding.

The attached patch tries to do just that.  I assume that there is not
any serious problem in the rest of the code with UTF-8.  The change does
not raise errors and it seems to work on test mails.  Please advise if
anything needs to be changed with the patch.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Support-RFC-6856-bug-XXXXX.patch --]
[-- Type: text/x-diff, Size: 1502 bytes --]

From eb2e8760e3b5004f13c769fa077b941844cc74da Mon Sep 17 00:00:00 2001
From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
Date: Sat, 6 May 2023 19:06:06 +0200
Subject: [PATCH] Support RFC 6856 (bug#XXXXX)

* lisp/net/pop3.el (pop3-open-server): Send "UTF8" when needed.
---
 lisp/net/pop3.el | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/lisp/net/pop3.el b/lisp/net/pop3.el
index d2fdbc3ffff..20f509bbd9c 100644
--- a/lisp/net/pop3.el
+++ b/lisp/net/pop3.el
@@ -551,13 +551,20 @@ pop3-open-server
 	       (and (string-match "\\bSTLS\\b" capabilities)
 		    "STLS\r\n"))))
       (when result
-	(let ((response (plist-get (cdr result) :greeting)))
+	(let* ((process (car result))
+               (properties (cdr result))
+               (response (plist-get properties :greeting))
+               (capabilities (plist-get properties :capabilities)))
 	  (setq pop3-timestamp
 		(substring response (or (string-search "<" response) 0)
-			   (+ 1 (or (string-search ">" response) -1)))))
-	(set-process-query-on-exit-flag (car result) nil)
-	(erase-buffer)
-	(car result)))))
+			   (+ 1 (or (string-search ">" response) -1))))
+	  (set-process-query-on-exit-flag process nil)
+	  (erase-buffer)
+          ;; Support RFC 6856.
+          (when (string-match "\\bUTF8\\b" capabilities)
+            (pop3-send-command process "UTF8")
+            (ignore-error t (pop3-read-response process)))
+	  process)))))
 
 ;; Support functions
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 6968 bytes --]


Note 1: I assume similar code is needed in imap.el (RFC 6855), but I do
not have such a server, so I am not able to test a corresponding change.

Note 2: Using UTF-8 in the "Subject:" header does not work.  While the
subject is shown correctly in the summary, I get the usual bad display
in the mail itself.  But that should be a separate bug report, I think.

----

In GNU Emacs 28.2.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2022-11-22 built on zenobia
Repository revision: 0636e1066bbafcd8219d9a8e09301c77f875d78b
Repository branch: emacs-28
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  global-git-commit-mode: t
  shell-dirtrack-mode: t
  gnus-topic-mode: t
  gnus-undo-mode: t
  desktop-save-mode: t
  display-time-mode: t
  xclip-mode: t
  xterm-mouse-mode: t
  delete-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/Projects/ttf-mode/arc-mode-compat hides ~/emacs/arc-mode-compat
/home/benny/.emacs.d/elpa/transient-20230315.1520/transient hides /usr/local/share/emacs/28.2.50/lisp/transient
/home/benny/.emacs.d/elpa/soap-client-3.2.1/soap-client hides /usr/local/share/emacs/28.2.50/lisp/net/soap-client
/home/benny/.emacs.d/elpa/soap-client-3.2.1/soap-inspect hides /usr/local/share/emacs/28.2.50/lisp/net/soap-inspect

Features:
(shadow bbdb-message emacsbug edebug find-dired grep vc tramp-archive
tramp-gvfs tramp-cache zeroconf tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat ls-lisp magit-find-file goto-addr
bug-reference magit-extras face-remap cus-edit cus-start gnus-html nndoc
sh-script smie executable org-indent org-element avl-tree generator
ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search
ol-docview doc-view image-mode exif ol-bibtex ol-bbdb ol-w3m ol-doi
org-link-doi org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob-core ob-eval
org-table oc-basic bibtex ol org-keys oc org-compat org-macs
org-loaddefs compile gnus-kill cl-print debug backtrace eieio-opt
speedbar ezimage dframe shortdoc ffap find-func help-fns magit-submodule
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func magit-diff smerge-mode diff
git-commit rx log-edit pcvs-util add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor shell pcomplete magit-mode transient magit-git magit-base
magit-section dash compat compat-29 dired-aux ps-mule flow-fill
format-spec benny-nitter eww xdg url-queue thingatpt shr-color color
misearch multi-isearch benny-url mule-util sort gnus-cite mail-extr
gnus-async gnus-bcklg gnus-dup gnus-ml pp gnus-topic jka-compr url-http
url-gw mm-archive url-file url-dired url-cache url-auth qp gnutls nndir
nnml nndraft nnmh nnfolder bbdb-gnus bbdb-mua network-stream nsm
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp
gnus-cache pop3 nnrss gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum shr kinsoku svg dom gnus-group gnus-undo mm-url bbdb-com crm
message-x bbdb bbdb-site timezone gnus-start gnus-dbus dbus xml
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601
gnus-spec gnus-int gnus-range message rmc puny rfc822 mml mml-sec epa
epg rfc6068 epg-config mm-decode mm-bodies mm-encode mailabbrev
gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs text-property-search wid-edit imenu time-date desktop
frameset highline benny-calendar-cfg ange-ftp comint ansi-color ring
generic-x autoinsert cc-mode cc-fonts cc-guess cc-menus cc-styles
cc-align cc-cmds cc-engine cc-vars cc-defs ps-print ps-print-loaddefs
ps-def lpr advice cl-extra help-mode dired dired-loaddefs
benny-x-clipboard disp-table time server protbuf xclip term/xterm xterm
xt-mouse cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew
holidays hol-loaddefs vc-git diff-mode easy-mmode vc-dispatcher
diary-lib diary-loaddefs cal-menu calendar cal-loaddefs delsel cua-base
cus-load benny-file-cache filecache derived sendmail mail-utils
.loaddefs benny-tools autoload radix-tree lisp-mnt mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr finder-inf pcase edmacro
kmacro info package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 578780 116125)
 (symbols 48 39754 19)
 (strings 32 214519 18167)
 (string-bytes 1 8662819)
 (vectors 16 90518)
 (vector-slots 8 1464975 88125)
 (floats 8 1213 837)
 (intervals 56 14138 2200)
 (buffers 992 55))

bug#63332: Acknowledgement (28.2.50; pop3.el does not support UTF-8 (RFC 6856))

[Benjamin Riefenstahl]

[-- Attachment #1: Type: text/plain, Size: 62 bytes --]

With bug number and ignore-error*s* instead of ignore-error.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Support-RFC-6856-bug-63332.patch --]
[-- Type: text/x-diff, Size: 1501 bytes --]

From 8af24a0d39f5104615cafd45af30fd5765108197 Mon Sep 17 00:00:00 2001
From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
Date: Sat, 6 May 2023 19:06:06 +0200
Subject: [PATCH] Support RFC 6856 (bug#63332)

* lisp/net/pop3.el (pop3-open-server): Send "UTF8" when needed.
---
 lisp/net/pop3.el | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/lisp/net/pop3.el b/lisp/net/pop3.el
index d2fdbc3ffff..aacb96dc5ec 100644
--- a/lisp/net/pop3.el
+++ b/lisp/net/pop3.el
@@ -551,13 +551,20 @@ pop3-open-server
 	       (and (string-match "\\bSTLS\\b" capabilities)
 		    "STLS\r\n"))))
       (when result
-	(let ((response (plist-get (cdr result) :greeting)))
+	(let* ((process (car result))
+               (properties (cdr result))
+               (response (plist-get properties :greeting))
+               (capabilities (plist-get properties :capabilities)))
 	  (setq pop3-timestamp
 		(substring response (or (string-search "<" response) 0)
-			   (+ 1 (or (string-search ">" response) -1)))))
-	(set-process-query-on-exit-flag (car result) nil)
-	(erase-buffer)
-	(car result)))))
+			   (+ 1 (or (string-search ">" response) -1))))
+	  (set-process-query-on-exit-flag process nil)
+	  (erase-buffer)
+          ;; Support RFC 6856.
+          (when (string-match "\\bUTF8\\b" capabilities)
+            (pop3-send-command process "UTF8")
+            (ignore-errors (pop3-read-response process)))
+	  process)))))
 
 ;; Support functions
 
-- 
2.30.2