bug#44307: 27.1; UTF-8 parts transferred as 8bit in multipart messages fail to decode

[Thomas Schneider <qsx@chaotikum.eu>]

Hi,

Gnus fails to display certain messages correctly.  So far, I have
identified the following conditions, though there might be other
constellations:

- The message is multipart/alternative
- The text/* part is UTF-8
- Content-Transfer-Encoding is 8bit

Any non-ASCII printable character (e. g., German umlauts) are
interpreted as something else (probably ISO 8859) in the HTML part, or
not interpreted at all and displayed as octal escapes in the plain part.

A working example can be found below – uuencoded to prevent any further
charset transfer issues.  It should display 'ääää', that is, four 'a'
with a trema above, in either part.  I’m unsure how to reproduce it
without a working Gnus setup.  I inserted the message in a nnmaildir
group to display it, so this way should definitely work.

begin 644 example.eml
M1G)O;3H@97AA;7!L92`\97AA;7!L94!E>&%M<&QE+F]R9SX-"E1O.B!E>&%M
M<&QE(#QE>&%M<&QE0&5X86UP;&4N;W)G/@T*0V]N=&5N="U4>7!E.B!M=6QT
M:7!A<G0O86QT97)N871I=F4[(&)O=6YD87)Y/2(]/3T]/3T]/3T]/3T]/3TR
M.#<W,3DU,#<U.30V.3<T,C0V/3TB#0I$871E.B!4:'4L(#(Y($]C="`R,#(P
M(#$T.C0W.C4U("LP,3`P#0I-24U%+59E<G-I;VXZ(#$N,`T*4W5B:F5C=#H@
M=&5S=`T*#0HM+3T]/3T]/3T]/3T]/3T]/3(X-S<Q.34P-S4Y-#8Y-S0R-#8]
M/0T*0V]N=&5N="U4>7!E.B!T97AT+W!L86EN.R!C:&%R<V5T/2)U=&8M."(-
M"D-O;G1E;G0M5')A;G-F97(M16YC;V1I;F<Z(#AB:70-"@T*PZ3#I,.DPZ0-
M"@T*+2T]/3T]/3T]/3T]/3T]/3TR.#<W,3DU,#<U.30V.3<T,C0V/3T-"D-O
M;G1E;G0M5'EP93H@=&5X="]H=&UL.R!C:&%R<V5T/2)U=&8M."(-"D-O;G1E
M;G0M5')A;G-F97(M16YC;V1I;F<Z(#AB:70-"@T*/"%D;V-T>7!E(&AT;6P^
M/&AT;6P^/&AE860^/&UE=&$@:'1T<"UE<75I=CTB8V]N=&5N="UT>7!E(B!C
M;VYT96YT/2)T97AT+VAT;6P[(&-H87)S970]551&+3@B/CPO:&5A9#X\8F]D
M>3[#I,.DPZ3#I#PO8F]D>3X\+VAT;6P^#0H-"BTM/3T]/3T]/3T]/3T]/3T]
9,C@W-S$Y-3`W-3DT-CDW-#(T-CT]+2T-"@``
`
end

Thanks for your time,
Thomas


In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.16.0)
 of 2020-09-01 built on localhost
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Gentoo/Linux

Recent messages:
Checking new news...done
Mark set
A bookmark has been added to the current article. [2 times]
Moved to bookmark [2 times]
Auto-saving...done

Configured using:
 'configure --prefix=/usr --build=x86_64-pc-linux-gnu
 --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
 --localstatedir=/var/lib --disable-silent-rules
 --docdir=/usr/share/doc/emacs-27.1-r1
 --htmldir=/usr/share/doc/emacs-27.1-r1/html --libdir=/usr/lib64
 --program-suffix=-emacs-27 --includedir=/usr/include/emacs-27
 --infodir=/usr/share/info/emacs-27 --localstatedir=/var
 --enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp
 --without-compress-install --without-hesiod --without-pop
 --with-dumping=pdumper --with-file-notification=inotify --enable-acl
 --with-dbus --without-modules --without-gameuser --with-libgmp
 --with-gpm --without-json --with-kerberos --with-kerberos5 --with-lcms2
 --with-xml2 --without-mailutils --without-selinux --with-gnutls
 --with-libsystemd --with-threads --without-wide-int --with-zlib
 --with-sound=alsa --with-x --without-ns --without-gconf
 --with-gsettings --with-toolkit-scroll-bars --with-gif --with-jpeg
 --with-png --with-rsvg --with-tiff --with-xpm --with-imagemagick
 --with-xft --with-cairo --without-harfbuzz --with-libotf
 --with-m17n-flt --with-x-toolkit=gtk3 --without-xwidgets 'CFLAGS=-O2
 -pipe -march=native -g' CPPFLAGS= 'LDFLAGS=-Wl,-O1 -Wl,--as-needed''

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO IMAGEMAGICK SOUND GPM DBUS GSETTINGS
GLIB NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD PDUMPER LCMS2
GMP

Important settings:
  value of $LC_MESSAGES: en_GB.UTF-8
  value of $LC_TIME: en_DK.UTF-8
  value of $LANG: de_DE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  hl-line-mode: t
  cursor-sensor-mode: t
  gnus-undo-mode: t
  TeX-PDF-mode: t
  TeX-source-correlate-mode: t
  pdf-occur-global-minor-mode: t
  stripe-buffer-mode: t
  helm-mode: t
  helm-ff-cache-mode: t
  helm-autoresize-mode: t
  helm--remap-mouse-mode: t
  global-magit-file-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  recentf-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-idle-scheduler-mode: t
  semantic-mode: t
  display-battery-mode: t
  display-time-mode: t
  show-paren-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/qsx/.emacs.d/elpa/dpkg-dev-el-20190824.2314/debian-autoloads hides /home/qsx/.emacs.d/elpa/debian-el-20201011.1543/debian-autoloads
/home/qsx/.emacs.d/elpa/auctex-12.2.5/context hides /usr/share/emacs/site-lisp/auctex/context
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex hides /usr/share/emacs/site-lisp/auctex/tex
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/home/qsx/.emacs.d/elpa/auctex-12.2.5/preview hides /usr/share/emacs/site-lisp/auctex/preview
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-ispell hides /usr/share/emacs/site-lisp/auctex/tex-ispell
/home/qsx/.emacs.d/elpa/auctex-12.2.5/latex-flymake hides /usr/share/emacs/site-lisp/auctex/latex-flymake
/home/qsx/.emacs.d/elpa/auctex-12.2.5/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/home/qsx/.emacs.d/elpa/auctex-12.2.5/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/home/qsx/.emacs.d/elpa/auctex-12.2.5/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/home/qsx/.emacs.d/elpa/auctex-12.2.5/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/home/qsx/.emacs.d/elpa/auctex-12.2.5/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/home/qsx/.emacs.d/elpa/auctex-12.2.5/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/home/qsx/.emacs.d/elpa/auctex-12.2.5/auctex hides /usr/share/emacs/site-lisp/auctex/auctex
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/home/qsx/.emacs.d/elpa/auctex-12.2.5/latex hides /usr/share/emacs/site-lisp/auctex/latex
/home/qsx/.emacs.d/elpa/auctex-12.2.5/plain-tex hides /usr/share/emacs/site-lisp/auctex/plain-tex
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/home/qsx/.emacs.d/elpa/auctex-12.2.5/tex-site hides /usr/share/emacs/site-lisp/auctex/tex-site
/home/qsx/.emacs.d/elpa/auctex-12.2.5/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs/site-lisp/cmake-mode hides /usr/share/emacs/site-lisp/cmake/cmake-mode
/home/qsx/.emacs.d/elpa/dash-functional-20200617.702/dash-functional hides /usr/share/emacs/site-lisp/dash/dash-functional
/home/qsx/.emacs.d/elpa/dash-20200803.1520/dash hides /usr/share/emacs/site-lisp/dash/dash
/usr/share/emacs/site-lisp/desktop-entry-mode hides /usr/share/emacs/site-lisp/desktop-file-utils/desktop-entry-mode
/home/qsx/.emacs.d/elpa/f-20191110.1357/f hides /usr/share/emacs/site-lisp/f/f
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-post hides /usr/share/emacs/site-lisp/ledger-mode/ledger-post
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-navigate hides /usr/share/emacs/site-lisp/ledger-mode/ledger-navigate
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-reconcile hides /usr/share/emacs/site-lisp/ledger-mode/ledger-reconcile
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-commodities hides /usr/share/emacs/site-lisp/ledger-mode/ledger-commodities
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-texi hides /usr/share/emacs/site-lisp/ledger-mode/ledger-texi
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-init hides /usr/share/emacs/site-lisp/ledger-mode/ledger-init
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-fontify hides /usr/share/emacs/site-lisp/ledger-mode/ledger-fontify
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-report hides /usr/share/emacs/site-lisp/ledger-mode/ledger-report
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-sort hides /usr/share/emacs/site-lisp/ledger-mode/ledger-sort
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-exec hides /usr/share/emacs/site-lisp/ledger-mode/ledger-exec
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-mode hides /usr/share/emacs/site-lisp/ledger-mode/ledger-mode
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-check hides /usr/share/emacs/site-lisp/ledger-mode/ledger-check
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-test hides /usr/share/emacs/site-lisp/ledger-mode/ledger-test
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-occur hides /usr/share/emacs/site-lisp/ledger-mode/ledger-occur
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-xact hides /usr/share/emacs/site-lisp/ledger-mode/ledger-xact
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-regex hides /usr/share/emacs/site-lisp/ledger-mode/ledger-regex
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-context hides /usr/share/emacs/site-lisp/ledger-mode/ledger-context
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-state hides /usr/share/emacs/site-lisp/ledger-mode/ledger-state
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-complete hides /usr/share/emacs/site-lisp/ledger-mode/ledger-complete
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-schedule hides /usr/share/emacs/site-lisp/ledger-mode/ledger-schedule
/home/qsx/.emacs.d/elpa/ledger-mode-20200530.1710/ledger-fonts hides /usr/share/emacs/site-lisp/ledger-mode/ledger-fonts
/home/qsx/.emacs.d/elpa/s-20180406.808/s hides /usr/share/emacs/site-lisp/s/s
/home/qsx/.emacs.d/elpa/with-editor-20200930.1912/with-editor hides /usr/share/emacs/site-lisp/with-editor/with-editor
/usr/share/emacs/site-lisp/mercury/gud hides /usr/share/emacs/27.1/lisp/progmodes/gud
/usr/share/emacs/site-lisp/mercurial/mercurial hides /home/qsx/.emacs.d/elisp/mercurial

Features:
(shadow nnir emacsbug sendmail helm-x-files helm-for-files helm-bookmark
helm-adaptive helm-external helm-net mule-util flow-fill ace-window avy
eieio-opt rfc1843 help-fns radix-tree gnus-cite smiley mm-archive
mail-extr gnus-bcklg qp helm-config gnus-async sort gnus-ml disp-table
hl-line cursor-sensor nnagent nnml nndraft nnmh nnfolder nnmaildir nnnil
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum url
url-proxy url-privacy url-expand url-methods url-history mailcap shr
url-cookie url-domsuf svg dom gnus-group gnus-undo gnus-start gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
gnus-win winner helm-command helm-elisp helm-eval edebug backtrace
helm-info gnus-alias gnus nnheader calc calc-loaddefs calc-macs
ledger-mode ledger-check ledger-texi ledger-test ledger-sort
ledger-report ledger-reconcile ledger-occur ledger-fonts ledger-fontify
ledger-state ledger-complete ledger-schedule ledger-init ledger-xact
ledger-post ledger-exec ledger-navigate eshell esh-cmd esh-ext esh-opt
esh-proc esh-io esh-arg esh-module esh-groups esh-util ledger-context
ledger-commodities org org-macro org-footnote org-pcomplete org-list
org-faces org-entities org-version ob-sqlite ob ob-tangle org-src ob-ref
ob-lob ob-table ob-exp ob-comint ob-emacs-lisp ob-core ob-eval org-table
ol org-keys org-compat org-macs org-loaddefs cal-menu calendar
cal-loaddefs ledger-regex haskell-mode haskell-cabal haskell-utils
haskell-font-lock haskell-indentation haskell-string
haskell-sort-imports haskell-lexeme haskell-align-imports
haskell-complete-module haskell-ghc-support etags fileloop generator
dabbrev haskell-customize adoc-mode tempo markup-faces json-mode
json-reformat json-snatcher js toml-mode conf-mode align auctex-latexmk
tex-buf latex latex-flymake flymake-proc flymake tex-ispell tex-style
tex dbus xml texmathp pdf-occur ibuf-ext ibuffer ibuffer-loaddefs
tablist tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent pdf-isearch let-alist pdf-misc pdf-tools cus-edit
pdf-view magit-bookmark bookmark jka-compr pdf-cache pdf-info tq
pdf-util image-mode exif dockerfile-mode sh-script executable
poly-ansible poly-ansible-jinja2-filters polymode poly-lock
polymode-base polymode-weave polymode-export polymode-compat
polymode-methods polymode-core polymode-classes ansible salt-mode rst
mmm-jinja2 mmm-auto mmm-vars mmm-utils mmm-compat yaml-mode
rainbow-delimiters stripe-buffer meson-mode smie apache-mode form-feed
helm-rg helm-mode helm-files tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat parse-time iso8601 ls-lisp
helm-buffers helm-occur helm-tags helm-locate helm-grep helm-regexp
helm-utils helm-help helm-types helm helm-global-bindings helm-easymenu
helm-source eieio-compat helm-multi-match helm-lib magit-submodule
magit-obsolete magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func magit-diff
smerge-mode diff diff-mode magit-core magit-autorevert autorevert
magit-margin magit-transient magit-process magit-mode git-commit
transient magit-git magit-section magit-utils crm log-edit message dired
dired-loaddefs format-spec rfc822 mml mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log with-editor async-bytecomp advice async shell
pcomplete lsp-ui lsp-ui-doc goto-addr lsp-ui-imenu lsp-ui-peek
lsp-ui-sideline face-remap lsp-mode yasnippet xref project url-util
spinner network-stream puny nsm rmc markdown-mode color thingatpt
noutline outline lv inline imenu filenotify f ewoc dash-functional
compile comint ansi-color bindat lsp-protocol ht srefactor srefactor-ui
recentf tree-widget cl srecode/semantic semantic/senator
semantic/decorate pulse srecode/insert srecode/filters srecode/args
ede/speedbar ede/files ede ede/detect ede/base ede/auto ede/source
eieio-speedbar speedbar sb-image dframe eieio-custom wid-edit
srecode/find srecode/map srecode/ctxt srecode/compile srecode/dictionary
srecode/fields srecode/table srecode semantic/doc semantic/tag-file
quilt semantic/db-file data-debug ring cedet-files semantic/bovine/c
hideif semantic/bovine/c-by semantic/lex-spp semantic/bovine/gcc
semantic/dep semantic/bovine semantic/analyze/refs semantic/db-find
semantic/db-ref cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs semantic/db-mode semantic/idle
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
semantic/db eieio-base semantic/format ezimage semantic/tag-ls
semantic/find semantic/ctxt semantic/util-modes semantic/util semantic
pp semantic/tag semantic/lex semantic/fw mode-local find-func cedet
company-shell dash company-ansible company-ansible-keywords
company-reftex s reftex-cite reftex reftex-loaddefs reftex-vars
company-bibtex parsebib bibtex warnings company-math math-symbol-lists
company edmacro kmacro pcase cl-extra help-mode battery time
deeper-blue-theme paren cus-start cus-load server use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
site-gentoo w3m-load preview-latex erlang-start tex-site dpkg-dev-el
debian-el rx info package easymenu browse-url url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 3763758 413383)
 (symbols 48 66383 3)
 (strings 32 2065449 238006)
 (string-bytes 1 108958761)
 (vectors 16 707848)
 (vector-slots 8 10736219 416300)
 (floats 8 667 485)
 (intervals 56 8152 1322)
 (buffers 1000 47))