bug#53412: 27.1; sendmail does not wrap overlong lines

bug#53412: 27.1; sendmail does not wrap overlong lines

[Francesco Potortì]

I use sendmail-user-agent (rather than message-user-agent) to send
email.  I have mml-mode in the mail-mode-hook.  When I have overlong
lines (> 1000 characters) Emacs should wrap them, as they are not
allowed in SMTP.  This should be standard behaviour, but it is not.

Digging in the source, I found mm-use-ultra-safe-encoding, which should
do that (among other things).  By the way, its doc string says that long
lines are wrapped to prevent broken email servers from breaking; that's
inaccurate: overlong lines are prohibited.

However I tried to set it, mm-use-ultra-safe-encoding is ignored, I have
not understood how it should be set.  I worked around it by

(add-hook 'mail-send-hook
	  (lambda () (and (mm-long-lines-p 1000)
			  (mml-to-mime)))) ; break overlong lines

but breaking overlong lines should be the normal behaviour, not a user
option to be explicitly activated.




In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2021-11-27, modified by Debian built on x86-csail-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Debian GNU/Linux bookworm/sid

Recent messages:
Quit
 [7 times]
Auto-saving...done
Auto save file for draft message exists; consider M-x mail-recover
You can run the command ‘mml-validate’ with C-c RET v
Beginning of buffer
scroll-signal-boundary-error: Beginning of buffer
Auto-saving...
Type C-x 1 to delete the help window, C-M-v to scroll help.

Quit
Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-x=yes
 --with-x-toolkit=lucid --with-toolkit-scroll-bars --without-gsettings
 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/emacs-fQHadv/emacs-27.1+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL
LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

Important settings:
  value of $LC_COLLATE: it_IT.UTF-8
  value of $LC_CTYPE: it_IT.UTF-8
  value of $LC_NUMERIC: C
  value of $LANG: C.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  desktop-save-mode: t
  epa-global-mail-mode: t
  shell-dirtrack-mode: t
  openwith-mode: t
  xterm-mouse-mode: t
  display-time-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
~/elisp/bhl hides /usr/share/emacs/site-lisp/bhl
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-autoloads hides /usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/debian-autoloads
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.22.1/cmake-mode hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.22.1/cmake-mode
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.22.1/cmake-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.22.1/cmake-mode-pkg
/usr/share/emacs/site-lisp/elpa/cmake-mode-3.22.1/cmake-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/cmake-mode-3.22.1/cmake-mode-autoloads
/usr/share/emacs/site-lisp/elpa/csv-mode-1.18/csv-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.18/csv-mode-pkg
/usr/share/emacs/site-lisp/elpa/csv-mode-1.18/csv-mode hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.18/csv-mode
/usr/share/emacs/site-lisp/elpa/csv-mode-1.18/csv-mode-tests hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.18/csv-mode-tests
/usr/share/emacs/site-lisp/elpa/csv-mode-1.18/csv-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/csv-mode-1.18/csv-mode-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-el hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/debian-el
/usr/share/emacs/site-lisp/elpa/debian-el-37/gnus-BTS hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/gnus-BTS
/usr/share/emacs/site-lisp/elpa/debian-el-37/preseed hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/preseed
/usr/share/emacs/site-lisp/elpa/debian-el-37/deb-view hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/deb-view
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-el-autoloads hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/debian-el-autoloads
/usr/share/emacs/site-lisp/elpa/debian-el-37/apt-utils hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/apt-utils
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-bug hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/debian-bug
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-el-pkg hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/debian-el-pkg
/usr/share/emacs/site-lisp/elpa/debian-el-37/apt-sources hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/apt-sources
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-autoloads hides /usr/share/emacs/site-lisp/elpa-src/debian-el-37/debian-autoloads
/usr/share/emacs/site-lisp/elpa/dictionary-1.10/dictionary hides /usr/share/emacs/site-lisp/elpa-src/dictionary-1.10/dictionary
/usr/share/emacs/site-lisp/elpa/dictionary-1.10/link hides /usr/share/emacs/site-lisp/elpa-src/dictionary-1.10/link
/usr/share/emacs/site-lisp/elpa/dictionary-1.10/dictionary-pkg hides /usr/share/emacs/site-lisp/elpa-src/dictionary-1.10/dictionary-pkg
/usr/share/emacs/site-lisp/elpa/dictionary-1.10/dictionary-autoloads hides /usr/share/emacs/site-lisp/elpa-src/dictionary-1.10/dictionary-autoloads
/usr/share/emacs/site-lisp/elpa/dictionary-1.10/connection hides /usr/share/emacs/site-lisp/elpa-src/dictionary-1.10/connection
/usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/gnuplot hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/gnuplot
/usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/gnuplot-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/gnuplot-mode-pkg
/usr/share/emacs/site-lisp/elpa/debian-el-37/debian-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/debian-autoloads
/usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/gnuplot-context hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/gnuplot-context
/usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/gnuplot-gui hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/gnuplot-gui
/usr/share/emacs/site-lisp/elpa/gnuplot-mode-20141231/gnuplot-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/gnuplot-mode-20141231/gnuplot-mode-autoloads
/usr/share/emacs/site-lisp/elpa/markdown-mode-2.4/markdown-mode-autoloads hides /usr/share/emacs/site-lisp/elpa-src/markdown-mode-2.4/markdown-mode-autoloads
/usr/share/emacs/site-lisp/elpa/markdown-mode-2.4/markdown-mode hides /usr/share/emacs/site-lisp/elpa-src/markdown-mode-2.4/markdown-mode
/usr/share/emacs/site-lisp/elpa/markdown-mode-2.4/markdown-mode-pkg hides /usr/share/emacs/site-lisp/elpa-src/markdown-mode-2.4/markdown-mode-pkg
~/elisp/bibtex hides /usr/share/emacs/27.1/lisp/textmodes/bibtex
~/elisp/octave hides /usr/share/emacs/27.1/lisp/progmodes/octave
/usr/share/emacs/site-lisp/flim/sasl hides /usr/share/emacs/27.1/lisp/net/sasl

Features:
(shadow emacsbug pp cl-print debug backtrace repeat grep eieio-opt
speedbar sb-image ezimage dframe find-func help-fns radix-tree dabbrev
iso-transl rmailsort wdired w3m-filter w3m-form w3m-cookie w3m-bookmark
w3m-tabmenu w3m-session w3m-search tabify warnings mailalias dired-aux
tramp-cache misearch multi-isearch rmailedit rmailout rmailkwd shr-color
time-stamp cl-extra server qp cal-julian solar cal-dst conf-mode python
tramp-sh vc-dir ewoc vc mhtml-mode css-mode eww mm-url gnus nnheader
url-queue url url-proxy url-privacy url-expand url-methods url-history
shr url-cookie url-domsuf url-util svg xml js imenu sgml-mode dom bibtex
generic sh-script executable make-mode gud bug-reference cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs tex-mode compile vc-dispatcher vc-svn vc-filewise vc-rcs octave
texinfo pcase smie vc-git diff-mode rmailmm message rmc puny rfc822 mml
mml-sec gnus-util text-property-search mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader mail-parse rfc2231 markdown-mode rx
color noutline outline easy-mmode desktop frameset pot skeleton rmailsum
rmail rmail-loaddefs w3m mailcap doc-view image-mode exif timezone
w3m-hist w3m-fb bookmark-w3m w3m-ems wid-edit w3m-favicon w3m-image
tab-line w3m-proc w3m-util ffap thingatpt ange-ftp sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr epa-mail mail-utils epa derived
epg epg-config view mule-util holidays hol-loaddefs appt diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete comint
ring parse-time iso8601 time-date ls-lisp format-spec bhl
visual-fill-column openwith switch-to-shell hi-lock anything-config
anything advice woman man cl locate xt-mouse scroll-in-place filladapt
ansi-color time jka-compr quail help-mode dired-x dired dired-loaddefs
generic-x shell-apt disp-table finder-inf w3m-load info debian-el
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 717143 274029)
 (symbols 48 42375 1)
 (strings 32 201218 11566)
 (string-bytes 1 5486048)
 (vectors 16 54011)
 (vector-slots 8 1886696 294498)
 (floats 8 735 643)
 (intervals 56 114279 81)
 (buffers 1000 172))

bug#53412: 27.1; sendmail does not wrap overlong lines

[Eli Zaretskii]

> From: Francesco Potortì <pot@gnu.org>
> Date: Fri, 21 Jan 2022 16:16:15 +0100
> 
> I use sendmail-user-agent (rather than message-user-agent) to send
> email.  I have mml-mode in the mail-mode-hook.  When I have overlong
> lines (> 1000 characters) Emacs should wrap them, as they are not
> allowed in SMTP.  This should be standard behaviour, but it is not.

Feel free to submit patches to that effect.

Thanks.

bug#53412: 27.1; sendmail does not wrap overlong lines

[Lars Ingebrigtsen]

Francesco Potortì <pot@gnu.org> writes:

> However I tried to set it, mm-use-ultra-safe-encoding is ignored, I have
> not understood how it should be set.  I worked around it by
>
> (add-hook 'mail-send-hook
> 	  (lambda () (and (mm-long-lines-p 1000)
> 			  (mml-to-mime)))) ; break overlong lines
>
> but breaking overlong lines should be the normal behaviour, not a user
> option to be explicitly activated.

I've now done something similar in `mail-send' in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Francesco Potortì]

I just discovered that checking overlong text lines is not enough.

Apparently calling mml-to-mime is not enough when the overlong line is a header.

If there is an overlong header, other headers which are utf-8 escaped are not sent correctly, and the recipent sees something like this:

Subject: test =?utf-8?Q?=C3=A0?= b c
 =?utf-8?Q?=C3=AC?=

RFC 5322 mandates header line lengths less than 998, including CRLF, and this is apparently not enforced by mml-to-mime.  Sending an identical email with long headers split into several lines makes the problem disappear.

I do not know where the culprit is and I do not have a workaround :(

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Francesco Potortì]

I just discovered that checking overlong text lines is not enough.

Apparently calling mml-to-mime does not wrap overlong headers.

If there is an overlong header, other headers which are utf-8 escaped are not sent correctly, and the recipient sees something like this:

Subject: test =?utf-8?Q?=C3=A0?= b c
 =?utf-8?Q?=C3=AC?=

Sending an identical email with long headers split into several lines makes the problem disappear.

RFC 5322 mandates mail line lengths less than 998 (including CRLF), and this is apparently not enforced by mml-to-mime for headers, I suppose it should.

My current workaround is:

(setq overlength 500)	; should by 998 including CRLF
(setq overlong-header-message "Overlong header at point: abort sending email.")
(add-hook 'mail-send-hook
	  ;; should become useless with Emacs 29
	  (lambda ()
	    (and (mm-long-lines-p overlength)
		 (mml-to-mime))	 ; break overlong text lines
	    ;; Now check that no overlong headers exist
	    (save-restriction
	      (message-narrow-to-headers)
	      (goto-char (point-min))
	      (and (search-forward-regexp (make-string overlength ?.) nil t)
		   (error overlong-header-message)))))

	  90)

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Lars Ingebrigtsen]

Francesco Potortì <pot@gnu.org> writes:

> If there is an overlong header, other headers which are utf-8 escaped are not sent correctly, and the recipient sees something like this:
>
> Subject: test =?utf-8?Q?=C3=A0?= b c
>  =?utf-8?Q?=C3=AC?=

That looks correct to me?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Francesco Potortì]

>> If there is an overlong header, other headers which are utf-8 escaped are not sent correctly, and the recipient sees something like this:
>>
>> Subject: test =?utf-8?Q?=C3=A0?= b c
>>  =?utf-8?Q?=C3=AC?=
>
>That looks correct to me?

No, if that's what the recipent sees in their inbox.  I don't know why, but mail sent by Emacs with an overlong header (whichever header) ends up with recipients seeing that string in their MUA.

This was true of some tens of recipients, I suppose with several different MUAs.  By the way, that happens also when the MUA is M-x rmail.

I can send mail to you or this list to demonstrate the problem, if you think that would be useful

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Lars Ingebrigtsen]

Francesco Potortì <pot@gnu.org> writes:

>>> If there is an overlong header, other headers which are utf-8
>>> escaped are not sent correctly, and the recipient sees something
>>> like this:
>>>
>>> Subject: test =?utf-8?Q?=C3=A0?= b c
>>>  =?utf-8?Q?=C3=AC?=
>>
>>That looks correct to me?
>
> No, if that's what the recipent sees in their inbox.  I don't know
> why, but mail sent by Emacs with an overlong header (whichever header)
> ends up with recipients seeing that string in their MUA.

They see literally that displayed?  Then it sounds like the mail you're
sending out has been double-rfc2047-encoded.

> I can send mail to you or this list to demonstrate the problem, if you
> think that would be useful

Sure.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Andreas Schwab]

On Mär 07 2022, Lars Ingebrigtsen wrote:

> Francesco Potortì <pot@gnu.org> writes:
>
>>>> If there is an overlong header, other headers which are utf-8
>>>> escaped are not sent correctly, and the recipient sees something
>>>> like this:
>>>>
>>>> Subject: test =?utf-8?Q?=C3=A0?= b c
>>>>  =?utf-8?Q?=C3=AC?=
>>>
>>>That looks correct to me?
>>
>> No, if that's what the recipent sees in their inbox.  I don't know
>> why, but mail sent by Emacs with an overlong header (whichever header)
>> ends up with recipients seeing that string in their MUA.
>
> They see literally that displayed?  Then it sounds like the mail you're
> sending out has been double-rfc2047-encoded.

I would think this happens because of the call to mml-to-mime in the
mail-send-hook he uses.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> I would think this happens because of the call to mml-to-mime in the
> mail-send-hook he uses.

Ah, makes sense.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53412: wrappin text lines is not enough, headers shoould be split too

[Francesco Potortì]

FP:
>>>>> If there is an overlong header, other headers which are utf-8
>>>>> escaped are not sent correctly, and the recipient sees something
>>>>> like this:
>>>>>
>>>>> Subject: test =?utf-8?Q?=C3=A0?= b c
>>>>>  =?utf-8?Q?=C3=AC?=

LI:
>>>>That looks correct to me?

FP:
>>> No, if that's what the recipent sees in their inbox.  I don't know
>>> why, but mail sent by Emacs with an overlong header (whichever header)
>>> ends up with recipients seeing that string in their MUA.

LI:
>> They see literally that displayed?  Then it sounds like the mail you're
>> sending out has been double-rfc2047-encoded.

AS:
>I would think this happens because of the call to mml-to-mime in the
>mail-send-hook he uses.

Sure, that's it.  This happens when there is an overlong header line and mml-to-mime is in the mail-send-hook.

mml-to-mime in the mail-send-hook is there because without it overlong text lines would go unbroken