bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

[Justus-dev]

Since Emacs 27.1 and still in the current development version, non-ASCII
characters display incorrectly if the encoding is 7bit or 8bit, at least
in my Gnus setup with nnmaildir backend.  This holds for both text/plain
and text/html content types.

To test this, send yourself a MIME message such as the following:

<#multipart type=mixed>
<#part type=text/plain charset="utf-8" encoding="7bit" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="8bit" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="quoted-printable" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#part type=text/plain charset="utf-8" encoding="base64" disposition=inline>
PUT SOME NON-ASCII CHARACTERS HERE
<#/multipart>

The first two parts will show non-ASCII characters in \ddd form (in HTML
parts, this would be printable but wrong characters instead), while the
third and fourth parts will display correctly.

I have not bothered reporting this till now because there is an easy
workaround (hit e (to edit) followed immediately by C-c C-c (to commit
the void edit), which will reencode the message in quoted-printable),
and because I figured I cannot be the only one who observes this
problem...???


This bug persists in my current Emacs build:

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.17.4)
 of 2021-01-03 built on piater
Repository revision: dc5dab469242c40803891518efe9c9f6bd650cea
Repository branch: makepkg
Windowing system distributor 'System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-sound=alsa --with-modules --without-gconf --without-gsettings
 --with-nativecomp --with-pgtk --with-x-toolkit=gtk3 --without-xaw3d
 --without-m17n-flt --with-cairo --with-xwidgets
 --without-compress-install 'CFLAGS=-march=x86-64 -mtune=generic -O2
 -pipe -fno-plt -g -fuse-ld=gold -g -fuse-ld=gold'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3
PGTK XIM MODULES NATIVE_COMP THREADS XWIDGETS LIBSYSTEMD JSON PDUMPER
LCMS2

Important settings:
  value of $LC_COLLATE: de_AT.UTF-8
  value of $LC_MONETARY: de_AT.UTF-8
  value of $LC_TIME: de_AT.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  semantic-minor-modes-format: ((:eval (if (or semantic-highlight-edits-mode semantic-show-unmatched-syntax-mode)  S)))
  gnus-agent-group-mode: t
  gnus-topic-mode: t
  shell-dirtrack-mode: t
  gnus-undo-mode: t
  show-paren-mode: t
  override-global-mode: t
  pdfgrep-mode: t
  pdf-occur-global-minor-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/usr/share/emacs/site-lisp/haskell-mode/w3m-haddock hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/w3m-haddock
/usr/share/emacs/site-lisp/haskell-mode/inf-haskell hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/inf-haskell
/usr/share/emacs/site-lisp/haskell-mode/highlight-uses-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/highlight-uses-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell
/usr/share/emacs/site-lisp/haskell-mode/haskell-utils hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-utils
/usr/share/emacs/site-lisp/haskell-mode/haskell-unicode-input-method hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-unicode-input-method
/usr/share/emacs/site-lisp/haskell-mode/haskell-string hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-string
/usr/share/emacs/site-lisp/haskell-mode/haskell-sort-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-sort-imports
/usr/share/emacs/site-lisp/haskell-mode/haskell-session hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-session
/usr/share/emacs/site-lisp/haskell-mode/haskell-sandbox hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-sandbox
/usr/share/emacs/site-lisp/haskell-mode/haskell-repl hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-repl
/usr/share/emacs/site-lisp/haskell-mode/haskell-process hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-process
/usr/share/emacs/site-lisp/haskell-mode/haskell-presentation-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-presentation-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell-navigate-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-navigate-imports
/usr/share/emacs/site-lisp/haskell-mode/haskell-move-nested hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-move-nested
/usr/share/emacs/site-lisp/haskell-mode/haskell-modules hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-modules
/usr/share/emacs/site-lisp/haskell-mode/haskell-menu hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-menu
/usr/share/emacs/site-lisp/haskell-mode/haskell-load hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-load
/usr/share/emacs/site-lisp/haskell-mode/haskell-lexeme hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-lexeme
/usr/share/emacs/site-lisp/haskell-mode/haskell-interactive-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-interactive-mode
/usr/share/emacs/site-lisp/haskell-mode/haskell-indentation hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-indentation
/usr/share/emacs/site-lisp/haskell-mode/haskell-indent hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-indent
/usr/share/emacs/site-lisp/haskell-mode/haskell-hoogle hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-hoogle
/usr/share/emacs/site-lisp/haskell-mode/haskell-font-lock hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-font-lock
/usr/share/emacs/site-lisp/haskell-mode/haskell-doc hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-doc
/usr/share/emacs/site-lisp/haskell-mode/haskell-decl-scan hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-decl-scan
/usr/share/emacs/site-lisp/haskell-mode/haskell-debug hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-debug
/usr/share/emacs/site-lisp/haskell-mode/haskell-customize hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-customize
/usr/share/emacs/site-lisp/haskell-mode/haskell-completions hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-completions
/usr/share/emacs/site-lisp/haskell-mode/haskell-complete-module hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-complete-module
/usr/share/emacs/site-lisp/haskell-mode/haskell-compile hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-compile
/usr/share/emacs/site-lisp/haskell-mode/haskell-compat hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-compat
/usr/share/emacs/site-lisp/haskell-mode/haskell-commands hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-commands
/usr/share/emacs/site-lisp/haskell-mode/haskell-collapse hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-collapse
/usr/share/emacs/site-lisp/haskell-mode/haskell-checkers hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-checkers
/usr/share/emacs/site-lisp/haskell-mode/haskell-cabal hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-cabal
/usr/share/emacs/site-lisp/haskell-mode/haskell-c2hs hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-c2hs
/usr/share/emacs/site-lisp/haskell-mode/haskell-align-imports hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/haskell-align-imports
/usr/share/emacs/site-lisp/haskell-mode/ghci-script-mode hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/ghci-script-mode
/usr/share/emacs/site-lisp/haskell-mode/ghc-core hides /usr/share/emacs/site-lisp/haskell-mode/build-26.1/ghc-core

Features:
(shadow emacsbug timezone vc-hg vc-git vc-bzr pdf-sync pdf-annot
pdf-outline pdf-links pdf-history smerge-mode diff diff-mode dabbrev rx
term ehelp shr-color flow-fill edebug debug backtrace pop3 gnus-gravatar
gravatar dns gnus-cus gnus-html gnus-kill gnus-logic gnus-mh mh-comp
mh-scan mh-gnus mh-e mh-compat mh-buffers mh-loaddefs gnus-registry
registry gnus-salt gnus-vm cl-print shortdoc locate mule-util ffap
spam-stat nnmairix nnml gnus-uu yenc gnus-sieve gnus-notifications
gnus-fun notifications gnus-eform gnus-dup gnus-diary nndiary gnus-demon
gnus-delay gnus-bookmark deuglify tramp-cmds misearch multi-isearch view
smiley utf-7 imap rfc2104 epa-file mailalias smtpmail sendmail
gnus-bcklg gnus-draft gnus-cite mail-extr gnus-async qp sort gnus-ml
disp-table gnus-topic mm-archive gnutls network-stream url-http url-gw
nsm url-cache url-auth nnrss nndraft nnmh nnfolder nnmaildir gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp gnus-cache
tramp-cache tramp-sh tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat native-complete shell ls-lisp cus-theme xwidget
wid-browse tree-widget dired-aux org-clock diary-lib diary-loaddefs
cal-iso org-indent ol-eww eww xdg url-queue thingatpt mm-url ol-rmail
ol-mhe ol-irc ol-info ol-gnus nnselect gnus-search eieio-opt speedbar
ezimage dframe gnus-art mm-uu mml2015 mm-view mml-smime smime dig
gnus-sum shr kinsoku svg dom gnus-group gnus-undo gnus-start gnus-dbus
dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec
gnus-int gnus-range message rmc puny rfc822 mml mml-sec epa epg
epg-config mailabbrev gmm-utils mailheader gnus-win ol-docview ol-bibtex
bibtex ol-bbdb ol-w3m poly-org polymode poly-lock polymode-base
polymode-weave polymode-export polymode-compat polymode-methods
polymode-core polymode-classes eieio-custom eieio-base color matlab
derived matlab-compat pulse face-remap org-agenda paren gnus nnheader
gnus-util rmail rmail-loaddefs mail-utils gnus-dired parse-time iso8601
use-package-bind-key bind-key quelpa-use-package cl-extra
use-package-core quelpa mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr lisp-mnt help-fns
radix-tree help-mode ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar ox-html table ox-ascii ox-publish ox org-element avl-tree
generator org-id org-refile haskell-mode-autoloads yaml-mode shell-here
org-recoll edmacro kmacro doc-view org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
pcomplete org-list org-faces org-entities time-date noutline outline
easy-mmode org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs cal-menu calendar cal-loaddefs
pdfgrep grep pdf-occur ibuf-ext ibuffer ibuffer-loaddefs tablist
tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local find-func cedet
pdf-isearch let-alist pdf-misc imenu pdf-tools compile comint ansi-color
ring cus-edit cus-start cus-load wid-edit pdf-view bookmark
text-property-search pp jka-compr pdf-cache pdf-info tq pdf-util advice
format-spec image-mode dired dired-loaddefs exif server info package
easymenu browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/pgtk-win
pgtk-win term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face pcase macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify dynamic-setting font-render-setting cairo move-toolbar gtk
x-toolkit pgtk lcms2 multi-tty make-network-process nativecomp emacs)

Memory information:
((conses 16 2133617 264148)
 (symbols 48 55146 3)
 (strings 32 1087649 82508)
 (string-bytes 1 63791349)
 (vectors 16 393221)
 (vector-slots 8 6065362 537361)
 (floats 8 598 661)
 (intervals 56 87491 968)
 (buffers 984 90))

bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

[Eli Zaretskii]

> From: Justus-dev@Piater.name
> Date: Mon, 04 Jan 2021 18:17:51 +0100
> 
> Since Emacs 27.1 and still in the current development version, non-ASCII
> characters display incorrectly if the encoding is 7bit or 8bit, at least
> in my Gnus setup with nnmaildir backend.  This holds for both text/plain
> and text/html content types.
> 
> To test this, send yourself a MIME message such as the following:
> 
> <#multipart type=mixed>
> <#part type=text/plain charset="utf-8" encoding="7bit" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="8bit" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="quoted-printable" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#part type=text/plain charset="utf-8" encoding="base64" disposition=inline>
> PUT SOME NON-ASCII CHARACTERS HERE
> <#/multipart>
> 
> The first two parts will show non-ASCII characters in \ddd form (in HTML
> parts, this would be printable but wrong characters instead), while the
> third and fourth parts will display correctly.
> 
> I have not bothered reporting this till now because there is an easy
> workaround (hit e (to edit) followed immediately by C-c C-c (to commit
> the void edit), which will reencode the message in quoted-printable),
> and because I figured I cannot be the only one who observes this
> problem...???

Isn't this a duplicate of bug#44307?

bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

[Justus-dev]

Eli Zaretskii <eliz@gnu.org> wrote on Mon, 04 Jan 2021 19:50:59 +0200:

> Isn't this a duplicate of bug#44307?

Yes, it is.  Sorry, my bad.  I checked only "emacs" bugs, not "gnus"
bugs.

Justus

bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

[Eli Zaretskii]

merge 45657 44307
thanks

> From: Justus-dev@Piater.name
> Cc: Justus-dev@Piater.name, 45657@debbugs.gnu.org
> Date: Mon, 04 Jan 2021 19:57:10 +0100
> 
> Eli Zaretskii <eliz@gnu.org> wrote on Mon, 04 Jan 2021 19:50:59 +0200:
> 
> > Isn't this a duplicate of bug#44307?
> 
> Yes, it is.  Sorry, my bad.  I checked only "emacs" bugs, not "gnus"
> bugs.

No sweat, I merged them.

Regardless, I hope someone of the Gnus bunch looks into this real
soon, as I think we should fix this on the release branch for the
upcoming Emacs 27.2.

bug#45657: 27.1; Gnus: Non-ASCII characters garbled with 7bit and 8bit encoding

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Regardless, I hope someone of the Gnus bunch looks into this real
> soon, as I think we should fix this on the release branch for the
> upcoming Emacs 27.2.

I'd love to, as soon as somebody can give me a case to reproduce the
error, but so far all the attempts have been garbled files...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no