bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Ed Sabol via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Steps:
1. Evaluate the following:
(setq mail-user-agent 'sendmail-user-agent)
(setq send-mail-function 'sendmail-send-it)
2. Hit `C-x m' or `M-x compose-mail'.
3. In the To: header type the following:

"LastName, FirstName I. (CODE-999.9)[CONTRACTOR COMPANY INC]"
	<your_email@address.com>

except replace "your_email@address.com" with your actual email address.

4. Put "test" in the Subject: header.
5. Add a "Bcc:" header with your actual email address.
6. In the body of the email, put "test".
7. Hit `C-c C-c' or `M-x mail-send' to send the email.

Result:
When you receive the sent email, the To: header of the email will look
like so:

"LastName\, FirstName I. \(CODE-999.9\)\[CONTRACTING COMPANY INC\]"
        <your_email@address.com>

It should like like this:

"LastName, FirstName I. (CODE-999.9)[CONTRACTING COMPANY INC]"
        <your_email@address.com>

At some point, something in sendmail.el changed between Emacs version
21.2 and 26.2 that resulted in these backslashes being
added. According to RFC822 and other email standards, these
backslashes are not needed. Emacs should not be modifying the headers
of emails needlessly.


In GNU Emacs 26.2 (build 2, x86_64-pc-linux-gnu, X toolkit)
 of 2019-04-12 built on heasarcdev
Windowing system distributor 'Scientific Linux', version 11.0.11704000
System Description:	Scientific Linux release 6.10 (Carbon)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set [2 times]
Sending...done

Configured using:
 'configure --prefix=/software/usr/contrib
 --exec-prefix=/software/usr/contrib/linux-6.x --with-x-toolkit=athena
 --with-toolkit-scroll-bars=no --with-jpeg=no --with-gif=no
 --with-tiff=no --with-gnutls=no'

Configured features:
XAW3D XPM PNG SOUND GSETTINGS GLIB NOTIFY LIBSELINUX LIBXML2 FREETYPE
XFT ZLIB LUCID X11 XDBE XIM THREADS

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug message rmc puny seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs format-spec
rfc822 mml easymenu mml-sec password-cache epa derived epg epg-config
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader mailalias sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils elec-pair
time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet
lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese composite charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
threads inotify dynamic-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 95678 7833)
 (symbols 48 19846 1)
 (miscs 40 54 168)
 (strings 32 29174 958)
 (string-bytes 1 769837)
 (vectors 16 14705)
 (vector-slots 8 503996 12136)
 (floats 8 53 115)
 (intervals 56 272 0)
 (buffers 992 9)
 (heap 1024 28666 935))

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Lars Ingebrigtsen]

Ed Sabol <esabol@milkyway.gsfc.nasa.gov> writes:

> 1. Evaluate the following:
> (setq mail-user-agent 'sendmail-user-agent)
> (setq send-mail-function 'sendmail-send-it)

[...]

> "LastName\, FirstName I. \(CODE-999.9\)\[CONTRACTING COMPANY INC\]"
>         <your_email@address.com>
>
> It should like like this:
>
> "LastName, FirstName I. (CODE-999.9)[CONTRACTING COMPANY INC]"
>         <your_email@address.com>

This isn't due to sendmail or anything, but because

(defcustom rfc2047-header-encoding-alist

[...]

    ("\\(Resent-\\)?\\(From\\|Cc\\|To\\|Bcc\\|\\(In-\\)?Reply-To\\|Sender\
\\|Mail-Followup-To\\|Mail-Copies-To\\|Approved\\)" . address-mime)

and

		    (when (eq method 'address-mime)
		      (rfc2047-quote-special-characters-in-quoted-strings))

> At some point, something in sendmail.el changed between Emacs version
> 21.2 and 26.2 that resulted in these backslashes being
> added. According to RFC822 and other email standards, these
> backslashes are not needed. Emacs should not be modifying the headers
> of emails needlessly.

I've been perusing the relevant RFCs, but I can't find anything about
having to quote commas and parentheses in quoted-strings.  It's not
disallowed, either -- the RFC says

3.2.1.  Quoted characters

   Some characters are reserved for special interpretation, such as
   delimiting lexical tokens.  To permit use of these characters as
   uninterpreted data, a quoting mechanism is provided.

   quoted-pair     =   ("\" (VCHAR / WSP)) / obs-qp

   Where any quoted-pair appears, it is to be interpreted as the
   character alone.  That is to say, the "\" character that appears as
   part of a quoted-pair is semantically "invisible".

so you can put a backslash in anywhere you want, basically, and it
should make no semantic difference.

But I'm not sure why this was added (in 2006).  Perhaps some MTAs/MUAs
react negatively to having those characters unquoted in strings?  It
looks like this was added by Katsumi, but the changelog doesn't say what
problem it's trying to fix.

Katsumi, do you remember?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Katsumi Yamaoka]

On Thu, 14 Nov 2019 05:59:37 +0100, Lars Ingebrigtsen wrote:
> But I'm not sure why this was added (in 2006).  Perhaps some MTAs/MUAs
> react negatively to having those characters unquoted in strings?  It
> looks like this was added by Katsumi, but the changelog doesn't say what
> problem it's trying to fix.

> Katsumi, do you remember?

I don't remember why I added the function
`rfc2047-quote-special-characters-in-quoted-strings' at all.  So
do I about `rfc2047-strip-backslashes-in-quoted-strings' either,
sorry.  But as for at least rfc2047-quote-*, it looks surely
needless.  With some brief tests, I verified the address-mime
encoder works without it.  Even if it is necessary for some cases,
it would probably be very rare, so I'd like to remove it now.
May I do that?

Thanks.

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I don't remember why I added the function
> `rfc2047-quote-special-characters-in-quoted-strings' at all.  So
> do I about `rfc2047-strip-backslashes-in-quoted-strings' either,
> sorry.

I seem to vaguely recall there perhaps being a problem with things
like

From: "Foo, Bar" <foo@zot.com>, zot@foo.com

being tokenized into three email addresses

Foo
Bar <foo@zot.com>
zot@foo.com

But on the other hand, if there's anything that broken out there, you'd
think that adding a backslash before the comma wouldn't help, either.

> But as for at least rfc2047-quote-*, it looks surely
> needless.  With some brief tests, I verified the address-mime
> encoder works without it.  Even if it is necessary for some cases,
> it would probably be very rare, so I'd like to remove it now.
> May I do that?

Sure, go ahead.  If there's any interoperability problems here, we'll
find out.  But I think it's pretty unlikely.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

On Thu, 14 Nov 2019 08:34:24 +0100, Lars Ingebrigtsen wrote:
> I seem to vaguely recall there perhaps being a problem with things
> like

> From: "Foo, Bar" <foo@zot.com>, zot@foo.com

> being tokenized into three email addresses

> Foo
> Bar <foo@zot.com>
> zot@foo.com

> But on the other hand, if there's anything that broken out there, you'd
> think that adding a backslash before the comma wouldn't help, either.

Sure.  Mail agents do not know what backslashes like to do.

> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>> But as for at least rfc2047-quote-*, it looks surely
>> needless.  With some brief tests, I verified the address-mime
>> encoder works without it.  Even if it is necessary for some cases,
>> it would probably be very rare, so I'd like to remove it now.
>> May I do that?

> Sure, go ahead.  If there's any interoperability problems here, we'll
> find out.  But I think it's pretty unlikely.

Done in master.  Thanks.  Ed, a patch for Emacs 26.2 is attached.

Regards,

[-- Attachment #2: remove-rfc2047-quote-special-characters-in-quoted-strings.patch.gz --]
[-- Type: application/x-gunzip, Size: 1374 bytes --]

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Done in master.  Thanks.  Ed, a patch for Emacs 26.2 is attached.

[...]

> -(defun rfc2047-quote-special-characters-in-quoted-strings (&optional
> -							   encodable-regexp)
> -  "Quote special characters with `\\'s in quoted strings.
> -Quoting will not be done in a quoted string if it contains characters
> -matching ENCODABLE-REGEXP or it is within parentheses."

I'm not sure removing the function is the right thing to do -- it may be
used by third-party code.  So we should just obsolete it and then remove
it in (say) Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Katsumi Yamaoka]

On Thu, 14 Nov 2019 09:10:52 +0100, Lars Ingebrigtsen wrote:
>> -(defun rfc2047-quote-special-characters-in-quoted-strings (&optional
>> -							   encodable-regexp)

> I'm not sure removing the function is the right thing to do -- it may be
> used by third-party code.  So we should just obsolete it and then remove
> it in (say) Emacs 29.

Done.  Thanks again.

bug#38200: 26.2; sendmail.el needlessly munging headers by inserting backslashes

[Ed Sabol via Bug reports for GNU Emacs, the Swiss army knife of text editors]

I have applied the patch to my installation of 26.2, and I can confirm
that the backslashes are no longer inserted into the To: header.

Thank you, Lars and Katsumi, for so quickly investigating and
resolving this!

Regards,
Ed