From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Benjamin Riefenstahl Newsgroups: gmane.emacs.bugs Subject: bug#63332: 28.2.50; pop3.el does not support UTF-8 (RFC 6856) Date: Sat, 06 May 2023 19:21:57 +0200 Message-ID: <87bkix74ui.fsf@turtle-trading.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="39411"; mail-complaints-to="usenet@ciao.gmane.io" To: 63332@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat May 06 19:23:26 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pvLcz-000A5d-P7 for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 06 May 2023 19:23:25 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvLcg-0008Ud-6s; Sat, 06 May 2023 13:23:06 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvLcc-0008UR-Qx for bug-gnu-emacs@gnu.org; Sat, 06 May 2023 13:23:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pvLcc-0003pj-J0 for bug-gnu-emacs@gnu.org; Sat, 06 May 2023 13:23:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pvLcc-0007kl-Dd for bug-gnu-emacs@gnu.org; Sat, 06 May 2023 13:23:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Benjamin Riefenstahl Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 06 May 2023 17:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 63332 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.168339373129726 (code B ref -1); Sat, 06 May 2023 17:23:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 6 May 2023 17:22:11 +0000 Original-Received: from localhost ([127.0.0.1]:35708 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pvLbm-0007jM-BT for submit@debbugs.gnu.org; Sat, 06 May 2023 13:22:11 -0400 Original-Received: from lists.gnu.org ([209.51.188.17]:51306) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pvLbi-0007jA-4r for submit@debbugs.gnu.org; Sat, 06 May 2023 13:22:09 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvLbh-0008Og-VZ for bug-gnu-emacs@gnu.org; Sat, 06 May 2023 13:22:05 -0400 Original-Received: from odoacer.turtle-trading.net ([93.241.193.16]) by eggs.gnu.org with esmtps (TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.90_1) (envelope-from ) id 1pvLbe-0003l1-Qx for bug-gnu-emacs@gnu.org; Sat, 06 May 2023 13:22:05 -0400 Original-Received: from zenobia.turtle-trading.net ([192.168.2.111]) by odoacer.turtle-trading.net with esmtp (Exim 4.80) (envelope-from ) id 1pvLbZ-0005Wx-FV; Sat, 06 May 2023 19:21:57 +0200 Original-Received: from benny by zenobia.turtle-trading.net with local (Exim 4.94.2) (envelope-from ) id 1pvLbZ-000EAU-7W; Sat, 06 May 2023 19:21:57 +0200 Received-SPF: none client-ip=93.241.193.16; envelope-from=benny@turtle-trading.net; helo=odoacer.turtle-trading.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:261207 Archived-At: --=-=-= Content-Type: text/plain I run Gnus. I get my mail from a POP3 server (courier-pop 1.0.16-3build3 on Ubuntu 22.4). The server returns the following kind of mail when there is a non-ASCII character somewhere in the headers of the mail: From: Mail Delivery Subsystem Subject: Cannot display Unicode content Attachment: [2. message/global]... This E-mail message was determined to be Unicode-formatted but your E-mail reader does not support Unicode E-mail. Please use an E-mail reader that supports POP3 with UTF-8 (see https://tools.ietf.org/html/rfc6856.html). This can also happen when the sender's E-mail program does not correctly format the sent message. The original message is included as a separate attachment so that it can be downloaded manually. [2. message/global]... I can provide a full message, if needed. Skimming RFC 6856, it looks like when the mail server provides the capability "UTF8", the client should issue the command "UTF8" to indicate that it is prepared to receive that encoding. The attached patch tries to do just that. I assume that there is not any serious problem in the rest of the code with UTF-8. The change does not raise errors and it seems to work on test mails. Please advise if anything needs to be changed with the patch. --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Support-RFC-6856-bug-XXXXX.patch >From eb2e8760e3b5004f13c769fa077b941844cc74da Mon Sep 17 00:00:00 2001 From: Benjamin Riefenstahl Date: Sat, 6 May 2023 19:06:06 +0200 Subject: [PATCH] Support RFC 6856 (bug#XXXXX) * lisp/net/pop3.el (pop3-open-server): Send "UTF8" when needed. --- lisp/net/pop3.el | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lisp/net/pop3.el b/lisp/net/pop3.el index d2fdbc3ffff..20f509bbd9c 100644 --- a/lisp/net/pop3.el +++ b/lisp/net/pop3.el @@ -551,13 +551,20 @@ pop3-open-server (and (string-match "\\bSTLS\\b" capabilities) "STLS\r\n")))) (when result - (let ((response (plist-get (cdr result) :greeting))) + (let* ((process (car result)) + (properties (cdr result)) + (response (plist-get properties :greeting)) + (capabilities (plist-get properties :capabilities))) (setq pop3-timestamp (substring response (or (string-search "<" response) 0) - (+ 1 (or (string-search ">" response) -1))))) - (set-process-query-on-exit-flag (car result) nil) - (erase-buffer) - (car result))))) + (+ 1 (or (string-search ">" response) -1)))) + (set-process-query-on-exit-flag process nil) + (erase-buffer) + ;; Support RFC 6856. + (when (string-match "\\bUTF8\\b" capabilities) + (pop3-send-command process "UTF8") + (ignore-error t (pop3-read-response process))) + process))))) ;; Support functions -- 2.30.2 --=-=-= Content-Type: text/plain Note 1: I assume similar code is needed in imap.el (RFC 6855), but I do not have such a server, so I am not able to test a corresponding change. Note 2: Using UTF-8 in the "Subject:" header does not work. While the subject is shown correctly in the summary, I get the usual bad display in the mail itself. But that should be a separate bug report, I think. ---- In GNU Emacs 28.2.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0) of 2022-11-22 built on zenobia Repository revision: 0636e1066bbafcd8219d9a8e09301c77f875d78b Repository branch: emacs-28 Windowing system distributor 'The X.Org Foundation', version 11.0.12011000 System Description: Debian GNU/Linux 11 (bullseye) Configured features: ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: Group Minor modes in effect: global-git-commit-mode: t shell-dirtrack-mode: t gnus-topic-mode: t gnus-undo-mode: t desktop-save-mode: t display-time-mode: t xclip-mode: t xterm-mouse-mode: t delete-selection-mode: t cua-mode: t tooltip-mode: t global-eldoc-mode: t show-paren-mode: t electric-indent-mode: t mouse-wheel-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t column-number-mode: t line-number-mode: t transient-mark-mode: t Load-path shadows: ~/Projects/ttf-mode/arc-mode-compat hides ~/emacs/arc-mode-compat /home/benny/.emacs.d/elpa/transient-20230315.1520/transient hides /usr/local/share/emacs/28.2.50/lisp/transient /home/benny/.emacs.d/elpa/soap-client-3.2.1/soap-client hides /usr/local/share/emacs/28.2.50/lisp/net/soap-client /home/benny/.emacs.d/elpa/soap-client-3.2.1/soap-inspect hides /usr/local/share/emacs/28.2.50/lisp/net/soap-inspect Features: (shadow bbdb-message emacsbug edebug find-dired grep vc tramp-archive tramp-gvfs tramp-cache zeroconf tramp tramp-loaddefs trampver tramp-integration files-x tramp-compat ls-lisp magit-find-file goto-addr bug-reference magit-extras face-remap cus-edit cus-start gnus-html nndoc sh-script smie executable org-indent org-element avl-tree generator ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search ol-docview doc-view image-mode exif ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete org-list org-faces org-entities noutline outline org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic bibtex ol org-keys oc org-compat org-macs org-loaddefs compile gnus-kill cl-print debug backtrace eieio-opt speedbar ezimage dframe shortdoc ffap find-func help-fns magit-submodule magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit magit-sequence magit-notes magit-worktree magit-tag magit-merge magit-branch magit-reset magit-files magit-refs magit-status magit magit-repos magit-apply magit-wip magit-log which-func magit-diff smerge-mode diff git-commit rx log-edit pcvs-util add-log magit-core magit-autorevert autorevert filenotify magit-margin magit-transient magit-process with-editor shell pcomplete magit-mode transient magit-git magit-base magit-section dash compat compat-29 dired-aux ps-mule flow-fill format-spec benny-nitter eww xdg url-queue thingatpt shr-color color misearch multi-isearch benny-url mule-util sort gnus-cite mail-extr gnus-async gnus-bcklg gnus-dup gnus-ml pp gnus-topic jka-compr url-http url-gw mm-archive url-file url-dired url-cache url-auth qp gnutls nndir nnml nndraft nnmh nnfolder bbdb-gnus bbdb-mua network-stream nsm gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp gnus-cache pop3 nnrss gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-sum shr kinsoku svg dom gnus-group gnus-undo mm-url bbdb-com crm message-x bbdb bbdb-site timezone gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601 gnus-spec gnus-int gnus-range message rmc puny rfc822 mml mml-sec epa epg rfc6068 epg-config mm-decode mm-bodies mm-encode mailabbrev gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail rmail-loaddefs text-property-search wid-edit imenu time-date desktop frameset highline benny-calendar-cfg ange-ftp comint ansi-color ring generic-x autoinsert cc-mode cc-fonts cc-guess cc-menus cc-styles cc-align cc-cmds cc-engine cc-vars cc-defs ps-print ps-print-loaddefs ps-def lpr advice cl-extra help-mode dired dired-loaddefs benny-x-clipboard disp-table time server protbuf xclip term/xterm xterm xt-mouse cal-china lunar solar cal-dst cal-bahai cal-islam cal-hebrew holidays hol-loaddefs vc-git diff-mode easy-mmode vc-dispatcher diary-lib diary-loaddefs cal-menu calendar cal-loaddefs delsel cua-base cus-load benny-file-cache filecache derived sendmail mail-utils .loaddefs benny-tools autoload radix-tree lisp-mnt mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr finder-inf pcase edmacro kmacro info package browse-url url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button loaddefs faces cus-face macroexp files window text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads dbusbind inotify lcms2 dynamic-setting system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs) Memory information: ((conses 16 578780 116125) (symbols 48 39754 19) (strings 32 214519 18167) (string-bytes 1 8662819) (vectors 16 90518) (vector-slots 8 1464975 88125) (floats 8 1213 837) (intervals 56 14138 2200) (buffers 992 55)) --=-=-=--