bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Pascal Quesseveur]

I have put vcard.el by Noah Friedmann in the load path.

I launch gnus and open a mail with an attached vcard:

,----
| ...
| MIME-Version: 1.0
| In-Reply-To: <PH0PR13MB48100B05DC5B2141F321EAC3F3AD9@PH0PR13MB4810.namprd13.prod.outlook.com>
| Content-Type: multipart/mixed;
| ...
| Content-Type: text/x-vcard; charset=utf-8;
|  name="dupont.vcf"
| Content-Transfer-Encoding: 7bit
| Content-Disposition: attachment;
|  filename="dupont.vcf"
| 
| begin:vcard
| fn:Jean
| n:Dupont;Jean
| org;quoted-printable:Syst=C3=A8mes DUPONT
| adr;quoted-printable:BP X;;La Cit=C3=A9;Province;00000;France
| email;internet:dupont@systemes.fr
| title:Directeur
| tel;work:+33 (0)000000000
| tel;fax:+33 (0)111111111
| tel;cell:+33 (0)222222222
| url:http://www.dupont.fr
| version:2.1
| end:vcard
`----

Accented characters are not displayed correctly, due to charset utf-8:

+-----------------------------------------------------------------------------+
| Jean-Dupont <dupont@systemes.fr>                     Work: +33 (0)000000000 |
| Directeur                                            Cell: +33 (0)222222222 |
| Systèmes DUPONT                                     Fax:  +33 (0)111111111 |
| La Cité, Province 00000 France                                             |
+-----------------------------------------------------------------------------+

I have defined the following function, similar to mm-inline-text-vcard
but considering the charset:

(defun qsr-inline-text-vcard (handle)
  (let ((inhibit-read-only t)
        (charset (or (mail-content-type-get (mm-handle-type handle) 'charset)
             mail-parse-charset))
        text)
    (setq text (if (fboundp 'vcard-pretty-print)
                   (vcard-pretty-print (mm-get-part handle))
                 (vcard-format-string
                  (vcard-parse-string (mm-get-part handle)
                                      'vcard-standard-filter))))
    (mm-insert-inline
     handle
     (concat "\n-- \n"
             (if (and charset text)
                 (mm-decode-string (string-make-unibyte text) charset)
               text)))))

When I use this function in mm-inline-media-tests display is OK.



In GNU Emacs 28.1 (build 2, x86_64-w64-mingw32)
 of 2022-04-21 built on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19044
System Description: Microsoft Windows 10 Pro (v10.0.2009.19044.1766)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES
NATIVE_COMP NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: FRA
  locale-coding-system: cp1252

Major mode: Group

Minor modes in effect:
  gnus-topic-mode: t
  gnus-undo-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  icomplete-mode: t
  auto-image-file-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
c:/Users/Public/emacs-site/lisp/utils/wdired hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/wdired
c:/Users/Public/emacs-site/lisp/utils/ls-lisp hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/ls-lisp
c:/Users/Public/emacs-site/lisp/utils/iimage hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/iimage
c:/Users/Public/emacs-site/lisp/utils/calculator hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/calculator
c:/Users/Public/emacs-site/lisp/utils/table hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/textmodes/table
c:/Users/Public/emacs-site/lisp/remember/remember hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/textmodes/remember
c:/Users/Public/emacs-site/lisp/utils/rlogin hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/net/rlogin
c:/Users/Public/emacs-site/lisp/dictionary-1.8.7/dictionary hides c:/Program Files/Emacs/emacs-28.1/share/emacs/28.1/lisp/net/dictionary

Features:
(shadow mailalias warnings emacsbug emms-info-libtag emms-player-vlc
emms-player-mpv emms-player-mplayer emms-playlist-limit emms-volume
emms-volume-mixerctl emms-volume-pulse emms-volume-amixer emms-i18n
emms-stream-info emms-mode-line-icon emms-playlist-sort
emms-last-played emms-playing-time emms-player-simple emms-streams
emms-show-all emms-tag-editor emms-mark emms-mode-line
emms-info-ogginfo emms-info-mp3info emms-info later-do
emms-playlist-mode emms-source-playlist emms-source-file locate
music-list music-album emms-setup emms emms-compat mule-util sort
vcard gnus-cite smiley shr-color color mm-archive mail-extr gnus-async
gnus-bcklg qp gnus-ml gnus-topic nndraft nnmh gnus-agent gnus-srvr
gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015
mm-view mml-smime smime dig nnml utf-7 gnutls nnfolder cl-extra
help-mode gnus-cache gnus-sum network-stream nsm nntp gnus-group
gnus-undo gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range gnus-win
time highlight-current-line color-theme smtpmail tumblesocks
tumblesocks-view tumblesocks-compose markdown-mode noutline outline
htmlize tumblesocks-user tumblesocks-api oauth sasl sasl-anonymous
sasl-login sasl-plain hex-util hmac-sha1 shr kinsoku svg dom
browse-url plantuml-mode xml dash thingatpt html2help footnote rx
muse-odf muse-xml muse-help muse-bbcode muse-blosxom muse-wiki
muse-texinfo texnfo-upd texinfo texinfo-loaddefs muse-latex muse-html
muse-docbook muse-xml-common cus-edit pp cus-load muse-publish
muse-project muse-protocols info muse-regexps muse muse-nested-tags
muse-mode u-vm-color org-import-icalendar icalendar diary-lib
diary-loaddefs bbdb-sc supercite regi bbdb-migrate bbdb-snarf
bbdb-message sendmail message rmc puny dired-explore dired-sort-menu
acid dired-arc file-op dired-x dired dired-loaddefs rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 gmm-utils mailheader
bbdb-rmail rmailsum bbdb-gnus bbdb-mua gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mail-utils mm-util
mail-prsvr wid-edit bbdb-com pcase crm mailabbrev find-func bbdb
bbdb-site timezone tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete parse-time iso8601 time-date
format-spec which idb gud easy-mmode compile text-property-search
comint ansi-color ring qproj-opascal jsee javadoc-lookup ido jserial
jswat qproj-java jdok tempo url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
auth-source eieio eieio-core eieio-loaddefs password-cache json subr-x
map seq byte-opt bytecomp byte-compile cconv url-vars mailcap
xml-parse doxymacs qproj-cpp qproj server dos-indent generic generic-x
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align
cc-engine cc-vars cc-defs lunar solar cal-dst cal-tex cal-iso cal-menu
calendar cal-loaddefs epa-file epa derived epg rfc6068 epg-config
ps-mule ipp cl-seq cl-macs cl gv printing ps-print ps-print-loaddefs
ps-def lpr icomplete advice image-file image-converter edmacro kmacro
cl-loaddefs cl-lib iso-transl tooltip eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet
lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese composite emoji-zwj
charscript charprop case-table epa-hook jka-cmpr-hook help simple
abbrev obarray cl-preloaded nadvice button loaddefs faces cus-face
macroexp files window text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
threads w32notify w32 lcms2 multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 641780 93369)
 (symbols 48 33480 2)
 (strings 32 167813 5043)
 (string-bytes 1 5455599)
 (vectors 16 48996)
 (vector-slots 8 854372 65672)
 (floats 8 1198 274)
 (intervals 56 1542 73)
 (buffers 992 30))

-- 
Pascal Quesseveur

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Lars Ingebrigtsen]

Pascal Quesseveur <quesseveur@abaksystemes.fr> writes:

>     (mm-insert-inline
>      handle
>      (concat "\n-- \n"
>              (if (and charset text)
>                  (mm-decode-string (string-make-unibyte text) charset)
>                text)))))
>
> When I use this function in mm-inline-media-tests display is OK.

Thanks.  I think it would be better to decode the data before handling
it to the vcard functions, so I've reworked the function that way
instead.

Can you check whether this fixes the problem, too?

diff --git a/lisp/gnus/mm-view.el b/lisp/gnus/mm-view.el
index 57ce36a944..1bfef637bf 100644
--- a/lisp/gnus/mm-view.el
+++ b/lisp/gnus/mm-view.el
@@ -366,16 +366,15 @@ mm-inline-text-html
 	(apply (car func) handle (cdr func)))))))
 
 (defun mm-inline-text-vcard (handle)
-  (let ((inhibit-read-only t))
-    (mm-insert-inline
-     handle
-     (concat "\n-- \n"
-	     (ignore-errors
-	       (if (fboundp 'vcard-pretty-print)
-		   (vcard-pretty-print (mm-get-part handle))
-		 (vcard-format-string
-		  (vcard-parse-string (mm-get-part handle)
-				      'vcard-standard-filter))))))))
+  (let* ((inhibit-read-only t)
+         (charset (mail-content-type-get (mm-handle-type handle) 'charset))
+         (data (mm-decode-string (mm-get-part handle) charset))
+         (text (and data
+                    (if (fboundp 'vcard-pretty-print)
+                        (vcard-pretty-print data)
+                      (vcard-format-string
+                       (vcard-parse-string data 'vcard-standard-filter))))))
+    (mm-insert-inline handle (concat "\n-- \n" text))))
 
 (defun mm-inline-text (handle)
   (let ((b (point))


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Lars Ingebrigtsen]

(Please keep the debbugs address in the CCs -- otherwise it won't reach
the bug tracker.)

Pascal Quesseveur <pquessev@gmail.com> writes:

>>"LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>   LI> Can you check whether this fixes the problem, too?
>
> Unfortunately not. To get text correctly encoded I have to call:
>
> (mm-decode-string (string-make-unibyte
>    (vcard-pretty-print (mm-get-part handle))) charset)
>
> FWIW buffer-file-coding-system in article buffer is iso-latin-1-dos.

That's odd -- is vcard-pretty-print doing charset encoding?  What we're
passing in should already be decoded (with that patch):

+         (data (mm-decode-string (mm-get-part handle) charset))

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Pascal Quesseveur]

>"LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

  LI> (Please keep the debbugs address in the CCs -- otherwise it won't reach
  LI> the bug tracker.)

Sorry, I missed it.

  LI> That's odd -- is vcard-pretty-print doing charset encoding?  What we're
  LI> passing in should already be decoded (with that patch):

The problem comes from quoted-printable fields in vcard. It works if
you decode QP before:


   (data (mm-decode-string (quoted-printable-decode-string
       (mm-get-part handle)) charset))

But in that case vcard-pretty-print receives a vcard containing fields
marked as quoted-printable, but they never are anymore.


-- 
Pascal Quesseveur
pquessev@gmail.com

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Roland Winkler]

On Tue, Jul 12 2022, Lars Ingebrigtsen wrote:
>> FWIW buffer-file-coding-system in article buffer is iso-latin-1-dos.
>
> That's odd -- is vcard-pretty-print doing charset encoding?  What we're
> passing in should already be decoded (with that patch):
>
> +         (data (mm-decode-string (mm-get-part handle) charset))

Below I am re-posting an example of a vcard MIME attachment that Pascal
had originally posted on the BBDB mailing list.  The odd thing about
these MIME attachments is that vcard properties such as "org" and "adr"
in the example below may include "quoted-printable" paramaters as part
of the vcard itself.  So my understanding is that the decoding of the
quoted-printable encoded strings and any subsequent coding-system
decoding should be done by a vcard parser and not by a MUA when
processing MIME attachments.

The vCard format 4.0 (rfc6350) requires that vCards use utf-8, it does
not permit other coding-systems.  I do not know about earlier versions
of the vCard format.

,----
| Content-Type: text/x-vcard; charset=utf-8;
|  name="dupont.vcf"
| Content-Transfer-Encoding: 7bit
| Content-Disposition: attachment;
|  filename="dupont.vcf"
| 
| begin:vcard
| fn:Jean
| n:Dupont;Jean
| org;quoted-printable:Syst=C3=A8mes DUPONT
| adr;quoted-printable:BP X;;La Cit=C3=A9;Province;00000;France
| email;internet:dupont@systemes.fr
| title:Directeur
| tel;work:+33 (0)000000000
| tel;fax:+33 (0)111111111
| tel;cell:+33 (0)222222222
| url:http://www.dupont.fr
| version:2.1
| end:vcard
`----

bug#56513: 28.1; Vcard in a mime part with a utf-8 charset is displayed incorrectly

[Lars Ingebrigtsen]

Roland Winkler <winkler@gnu.org> writes:

> Below I am re-posting an example of a vcard MIME attachment that Pascal
> had originally posted on the BBDB mailing list.  The odd thing about
> these MIME attachments is that vcard properties such as "org" and "adr"
> in the example below may include "quoted-printable" paramaters as part
> of the vcard itself.  So my understanding is that the decoding of the
> quoted-printable encoded strings and any subsequent coding-system
> decoding should be done by a vcard parser and not by a MUA when
> processing MIME attachments.

Ah, I see.

> The vCard format 4.0 (rfc6350) requires that vCards use utf-8, it does
> not permit other coding-systems.  I do not know about earlier versions
> of the vCard format.

I've now adjusted the vcard.el parser in GNU ELPA to decode the data
using utf-8, and that seems to fix the problem.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no