bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[micah anderson]

Hello,

I was trying to send an encrypted email while in notmuch, to a user
whose openpgp key had expired. When I tried to send the mail, I would
get the following in the mini-buffer:

mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

I did M-x toggle-debug-on-error and tried again, and I see this:

Debugger entered--Lisp error: (scan-error "Unbalanced parentheses" 7 32)
  scan-sexps(7 1)
  forward-sexp(1)
  mail-header-parse-address("dr ebs (\\_O<) <dre@foo.net>")
  mml-secure-check-user-id([cl-struct-epg-key none
  ([cl-struct-epg-sub-key expired (sign certify) nil 1 4096
  "A5E6BCA629BA4127" (20587 . 32329) (23459 . 54374)
  "B2B397904D39F3B3D4BA511EA5E6BCA629BA4127"] [cl-struct-epg-sub-key
  expired (authentication) nil 1 2048 "D35B7CEAEC88B93E" (21257 . 63618)
  nil "12B793B0865848CE63DEEA37D35B7CEAEC88B93E"] [cl-struct-epg-sub-key
  expired (encrypt) nil 1 4096 "55D07B549A11DF03" (20587 . 32329) nil
  "525B1A94392DE6015619ED4F55D07B549A11DF03"]) ([cl-struct-epg-user-id
  expired "dr ebs <dre@foo.net>" nil] [cl-struct-epg-user-id revoked "dr
  ebs <dre@bar.org>" nil] [cl-struct-epg-user-id revoked "dr ebs (>O_/)
  <dre@bar.org>" nil] [cl-struct-epg-user-id revoked "dr ebs (\\_O<) <dre@foo.net>" nil])] "<dre@foo.net>")
  mml-secure-find-usable-keys([cl-struct-epg-context OpenPGP "/usr/bin/gpg2" nil nil nil nil nil nil nil (epg-passphrase-callback-function) nil nil nil nil nil nil nil nil nil "" nil] "<dre@foo.net>" encrypt)
  #[(name) "\306\b	\n#\211G\211\307U\203&

Once I refreshed this user's key, I was able to send the message without
error. The user has some UIDs on their key that have a comment in them
with ascii ducks:

(3)  dre ebs (>O_/) <dre@bar.org>
(4)  dre ebs (\_O<) <dre@foo.net>

They have been revoked now that I refreshed, so I guess it did not
consider these keys because of that, but before it was considering them
and was choking on the ducks, a sexp injection?

Thanks!
micah


In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2018-08-26, modified by Debian built on x86-csail-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description:	Debian GNU/Linux testing (buster)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --build x86_64-linux-gnu
 --prefix=/usr --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/25.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-x=yes --with-x-toolkit=gtk3
 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-7bZ65w/emacs-25.2+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8

Major mode: notmuch-show

Minor modes in effect:
  dumb-jump-mode: t
  nyan-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  ido-vertical-mode: t
  ido-everywhere: t
  icomplete-mode: t
  display-time-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  minibuffer-depth-indicate-mode: t
  helm-descbinds-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Recent messages:
Doing Fcc...
Sending...done
debug: postpone cleanup hook
debug: done exit actions
View mode: type C-h for help, h for commands, q to quit.
Mark saved where search started
Mark set [4 times]
Mark deactivated
Mark set [4 times]
<s-mouse-1> is undefined [2 times]

Load-path shadows:
/home/user/.emacs.d/plugins/post hides /home/user/.emacs.d/post
/home/user/.emacs.d/elpa/emms-player-mpv-20180415.1958/emms-player-mpv hides /home/user/.emacs.d/elpa/emms-20180417.2218/emms-player-mpv
~/.emacs.d/plugins/puppet-flymake/flymake-puppet hides /home/user/.emacs.d/elpa/flymake-puppet-20170731.2254/flymake-puppet
~/.emacs.d/plugins/puppet-syntax-emacs/puppet-mode hides /home/user/.emacs.d/elpa/puppet-mode-20171221.649/puppet-mode
/home/user/.emacs.d/elpa/color-theme-solarized-20171024.825/solarized-theme hides /home/user/.emacs.d/elpa/solarized-theme-20180621.2107/solarized-theme
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-maildir-fcc hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-maildir-fcc
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-show hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-show
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-pkg hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-pkg
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-tag hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-tag
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-parser hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-parser
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-message hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-message
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-draft hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-draft
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-mua hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-mua
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-tree hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-tree
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-print hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-print
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-address hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-address
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-company hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-company
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-compat hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-compat
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-jump hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-jump
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-lib hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-lib
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-query hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-query
/home/user/.emacs.d/elpa/notmuch-20180503.1659/coolj hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/coolj
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-hello hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-hello
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-wash hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-wash
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-autoloads hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-autoloads
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-crypto hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-crypto
/home/user/.emacs.d/elpa/notmuch-20180503.1659/make-deps hides /usr/share/emacs/site-lisp/elpa/notmuch-0.28/make-deps
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-maildir-fcc hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-maildir-fcc
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-pkg hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-pkg
/usr/share/emacs/site-lisp/elpa/notmuch-0.28/notmuch-version hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-version
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-draft hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-draft
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-message hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-message
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-address hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-address
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-mua hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-mua
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-tag hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-tag
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-jump hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-jump
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-show hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-show
/home/user/.emacs.d/elpa/notmuch-20180503.1659/coolj hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/coolj
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-company hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-company
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-hello hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-hello
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-print hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-print
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-tree hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-tree
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-lib hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-lib
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-query hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-query
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-wash hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-wash
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-autoloads hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-autoloads
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-crypto hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-crypto
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-parser hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-parser
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch
/home/user/.emacs.d/elpa/notmuch-20180503.1659/make-deps hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/make-deps
/home/user/.emacs.d/elpa/notmuch-20180503.1659/notmuch-compat hides /usr/share/emacs/site-lisp/elpa-src/notmuch-0.28/notmuch-compat
/home/user/.emacs.d/custom hides /usr/share/emacs/25.2/lisp/custom
/home/user/.emacs.d/elpa/let-alist-1.0.5/let-alist hides /usr/share/emacs/25.2/lisp/emacs-lisp/let-alist
/home/user/.emacs.d/elpa/seq-20151121.1017/seq hides /usr/share/emacs/25.2/lisp/emacs-lisp/seq
/home/user/.emacs.d/plugins/avl-tree hides /usr/share/emacs/25.2/lisp/emacs-lisp/avl-tree

Features:
(shadow emacsbug misearch multi-isearch view sendmail debug gnus
gnus-ems nnheader mml2015 qp sort gnus-alias flyspell ispell mail-extr
mm-archive warnings dumb-jump popup f etags xref project my-ace my-avy
avy my-elpa my-puppet flymake-puppet flymake compile my-nyan nyan-mode
my-ediff ediff dired+ dired-x ediff-merg ediff-wind ediff-diff
ediff-mult ediff-help ediff-init ediff-util dired-aux my-magit
magit-gh-pulls s gh gh-users gh-issues gh-pulls gh-repos gh-comments
gh-gist gh-oauth gh-api logito gh-cache pcache eieio-base gh-auth gh-url
gh-profile magit-obsolete magit-blame magit-stash magit-bisect
magit-remote magit-commit epa magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-collab ghub
url-http tls gnutls url-gw url-auth url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
url-vars let-alist json map auth-source magit-files magit-refs
magit-status subr-x magit magit-repos magit-apply magit-wip magit-log
magit-diff smerge-mode magit-core magit-autorevert autorevert filenotify
magit-process magit-margin magit-mode git-commit recentf tree-widget
magit-git magit-section magit-utils vc-git magit-popup log-edit
pcvs-util add-log with-editor async-bytecomp shell server dash
my-notmuch eudc cus-edit eudc-vars notmuch hl-line notmuch-message
notmuch-hello wid-edit notmuch-tree notmuch-show notmuch-print
notmuch-crypto notmuch-mua notmuch-draft notmuch-maildir-fcc
notmuch-address notmuch-company notmuch-parser notmuch-wash diff-mode
coolj notmuch-query goto-addr thingatpt icalendar diary-lib
diary-loaddefs notmuch-tag crm notmuch-lib notmuch-version
notmuch-compat message rfc822 mml mailabbrev mail-utils gmm-utils
mailheader mm-view mml-smime mml-sec epg gnus-util smime password-cache
dig mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231 rfc2047
rfc2045 ietf-drums mm-util help-fns mail-prsvr my-smex smex my-org
org-clock org org-macro org-footnote org-pcomplete pcomplete org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint comint
ansi-color ring ob-core ob-eval org-compat org-macs org-loaddefs
format-spec find-func cal-menu calendar cal-loaddefs org-install my-ido
ido-vertical-mode flx-ido flx ido my-tls my-generickeys my-generic
speedbar sb-image ezimage dframe icomplete saveplace windmove time
delsel avoid paren mb-depth helm-descbinds helm derived helm-source
helm-multi-match easy-mmode helm-lib wdired dired advice async cus-start
cus-load my-package finder-inf edmacro kmacro rx gh-common marshal
eieio-compat cl-seq cl eieio eieio-core cl-macs logito-autoloads info
package epg-config seq byte-opt gv bytecomp byte-compile cl-extra
help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1690192 85853)
 (symbols 48 48520 0)
 (miscs 40 4379 1198)
 (strings 32 221439 20423)
 (string-bytes 1 6583248)
 (vectors 16 39009)
 (vector-slots 8 997769 40884)
 (floats 8 963 638)
 (intervals 56 143870 4992)
 (buffers 976 28))
-- 
        micah

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[Glenn Morris]

micah anderson wrote:

>   mail-header-parse-address("dr ebs (\\_O<) <dre@foo.net>")

It could probably be handled more elegantly, but IIUC

"dr ebs (\\_O<) <dre@foo.net>"

isn't valid in an email header, because the () are special characters
that should be quoted in the display name, per RFC5322.

Ie this works:

(mail-header-parse-address "\"dr ebs (\\_O<)\" <dre@foo.net>")


A nice summary at
https://stackoverflow.com/questions/24940588/validation-format-of-display-name-in-from-header

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[micah anderson]

Glenn Morris <rgm@gnu.org> writes:

> micah anderson wrote:
>
>>   mail-header-parse-address("dr ebs (\\_O<) <dre@foo.net>")
>
> It could probably be handled more elegantly, but IIUC
>
> "dr ebs (\\_O<) <dre@foo.net>"
>
> isn't valid in an email header, because the () are special characters
> that should be quoted in the display name, per RFC5322.

Yeah, the text that comes before the <dre@foo.net> in an OpenPGP userID
is defined as arbitrary UTF-8 text, so it should not be considered to be
following any email address standard.

-- 
        micah

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[Glenn Morris]

micah anderson wrote:

> Yeah, the text that comes before the <dre@foo.net> in an OpenPGP userID
> is defined as arbitrary UTF-8 text, so it should not be considered to be
> following any email address standard.

OK, but there's a de facto standard [1] that it be a normal email
address. GPG for example won't create a uid like the one in this report
without --allow-freeform-uid, which "should only be used in very special
environments".

[1] https://tools.ietf.org/html/rfc4880#section-5.11
"By convention, it includes an RFC 2822 mail name-addr".

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[micah anderson]

Glenn Morris <rgm@gnu.org> writes:

> micah anderson wrote:
>
>> Yeah, the text that comes before the <dre@foo.net> in an OpenPGP userID
>> is defined as arbitrary UTF-8 text, so it should not be considered to be
>> following any email address standard.
>
> OK, but there's a de facto standard [1] that it be a normal email
> address. GPG for example won't create a uid like the one in this report
> without --allow-freeform-uid, which "should only be used in very special
> environments".

You are right.

> [1] https://tools.ietf.org/html/rfc4880#section-5.11
> "By convention, it includes an RFC 2822 mail name-addr"

yes, except when it doesn't.

That quote continues, here is the full thing:

"By convention, it includes an RFC 2822 [RFC2822] mail name-addr, but
there are no restrictions on its content."

I think we agree on this - these are not normal, they aren't
conventional, but mail-header-parse-address should be able to deal with
these cases.

-- 
        micah

bug#33613: 25.2; mail-header-parse-address: scan-error "Unbalanced parenthesis" 7 32

[Lars Ingebrigtsen]

micah anderson <micah@riseup.net> writes:

>   mail-header-parse-address("dr ebs (\\_O<) <dre@foo.net>")
>   mml-secure-check-user-id([cl-struct-epg-key none

I've now made mml-secure-check-user-id ignore keys that aren't valid
email addresses instead of bugging out in Emacs 27.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no