bug#7589: smtpmail: bad line endings in queued mails on Windows

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Vida Gábor]

Hi!

I've already sent this bug to help-emacs-windows (as advised by M-x
report-emacs-bug in Windows port of "GNU Emacs 23.1.50.1
(i386-mingw-nt6.0.6001) of 2009-11-04 on LENNART-69DE564 (patched)"),
but I think that is not the best place, so do I report it here.

[h-e-w] 23.1.50; smtpmail: bad line endings in queued mails
From: Gábor Vida
Subject: [h-e-w] 23.1.50; smtpmail: bad line endings in queued mails
Date: Sun, 26 Sep 2010 00:20:31 +0200

Hi!

This problem comes only if `smtpmail-queue-mail' is set to `t'.

When I'm ready with the mail, smtpmail-send-it is called and it tries to
save the mail to disk. It is ok, but if I open the file with
find-file-literally, then I see an additional ^M character at the end of
each line. This causes problem (at least with MS Exchange) when I send
it: the mail received by the recipient has empty body. This is sure that
this additional ^M causes the problem (if I set smtpmail-queue-mail to
nil, the smtpmail trace shows only one ^M at the end of each line).

I tried to understand what and where is the problem in the code, but the
only conclusion was that if smtpmail reads the saved mail without
conversion (coding-system-for-read set to 'no-conversion), then it
should save it using Unix-style line endings.

I run Emacs from Cygwin.

Thanks and regards,
Gábor

Here is the bugous smtpmail trace:

[...]
250-SIZE 10485760^M
250-PIPELINING^M
250-DSN^M
250-ENHANCEDSTATUSCODES^M
250-AUTH GSSAPI NTLM LOGIN^M
250-8BITMIME^M
250-BINARYMIME^M
250 CHUNKING^M
[...]
235 2.7.0 Authentication successful^M
MAIL FROM:<address@hidden> SIZE=426^M
250 2.1.0 Sender OK^M
RCPT TO:<address@hidden>^M
250 2.1.5 Recipient OK^M
DATA^M
354 Start mail input; end with <CRLF>.<CRLF>^M
From: address@hidden (=?utf-8?Q?G=C3=A1bor?= VIDA)^M^M
To: Vida =?utf-8?Q?G=C3=A1bor?= <address@hidden>^M^M
Subject: aa^M^M
Date: Sat, 25 Sep 2010 22:52:36 +0200^M^M
Message-ID: <address@hidden>^M^M
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (windows-nt)^M^M
MIME-Version: 1.0^M^M
Content-Type: text/plain; charset=utf-8^M^M
Content-Transfer-Encoding: base64^M^M
^M^M
DQotLSANCsOcZHYsDQpHw6Fib3INCg==^M^M
.^M
250 2.6.0 <address@hidden> Queued mail for delivery^M
QUIT^M
221 2.0.0 Service closing transmission channel^M



And the generated part:

This is a bug report for the PATCHED Emacs+EmacsW32.
Please test the unpatched version of Emacs+EmacsW32 before reporting
if you can. If the bug is also in the unpatched version then
report from the unpatched version else report here.

If it is very inconvenient for you to try the unpatched version,
but you believe the bug is not related to the patches then you
may change the address above to address@hidden

For easy to read information about the patches see EmacsW32 home
page.


Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the address@hidden mailing list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
c:/Program Files/Emacs/emacs/etc/DEBUG.


In GNU Emacs 23.1.50.1 (i386-mingw-nt6.0.6001)
 of 2009-11-04 on LENNART-69DE564 (patched)
Windowing system distributor `Microsoft Corp.', version 6.0.6001
configured using `configure --with-gcc (3.4) --cflags -Ic:/g/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: C.UTF-8

  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  minibuffer-depth-indicate-mode: t
  icicle-mode: t
  display-time-mode: t
  pretty-control-l-mode: t
  delete-selection-mode: t
  shell-dirtrack-mode: t
  which-function-mode: t
  show-paren-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  noticeable-minibuffer-prompts-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  global-auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-n C-n C-n C-n C-n C-n C-n # # # # # # # # # # # # 
# # # # C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-x b * M <backspace> <tab> m <tab> 
<return> C-x k <return> y C-x b * t r <tab> <return> 
M-> M-< C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-p C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f 
C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f 
C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-n 
C-n C-n C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n M-< 
C-s v i d a g C-s C-n C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-s b u g s C-s C-n C-p C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p C-p C-p C-p 
C-p C-p C-p C-s m a i l i n g M-b M-b M-b M-b M-b M-b 
C-SPC M-f M-f M-f M-w C-p C-p C-p M-x M-p M-p <ret
urn>

Recent messages:
No more articles [2 times]
Computing completion candidates... [2 times]
Buffer *mail* modified; kill anyway? (y or n) 
Computing completion candidates...
Mark set [2 times]
Auto-saving...done
Mark set
Mark saved where search started [3 times]
Mark set
Auto-saving...done

Load-path shadows:
~/config/emacs.d/elisp/apel/timezone hides c:/Program 
Files/Emacs/emacs/lisp/timezone
~/config/emacs.d/elisp/apel/env hides c:/Program Files/Emacs/emacs/lisp/env
c:/Program Files/Emacs/EmacsW32/nxhtml/util/chart hides c:/Program 
Files/Emacs/emacs/lisp/emacs-lisp/chart

Features:
(shadow emacsbug debug smiley gnus-async gnus-bcklg goto-addr vc-cvs
noutline outline cl-specs edebug mule-diag newcomment multi-isearch
windmove time-stamp mailalias sort bbdb-gui gnus-cite face-remap
parse-time gnus-ml gnus-topic nnfolder utf-7 utf7 nnimap nndraft nnmh
bbdb-gnus bbdb-snarf mail-extr bbdb-com nnagent nnml gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 pgg pgg-parse pgg-def mm-view smime dig gnus-cache gnus-sum
gnus-demon nntp gnus-group gnus-undo nnmail mail-source nnoo gnus-start
gnus-spec gnus-int gnus-range gnus-win gnus gnus-ems mb-depth two-column
bookmark pp icicles icicles-mode icicles-cmd2 yow cookie1 etags
icicles-cmd1 icicles-mcmd help-mode view icicles-fn icicles-mac
icicles-var icicles-opt icicles-face hexrgb time google-maps
google-maps-static google-maps-geocode google-maps-base json
bbdb-autoloads bbdb hidesearch hide-lines jabber-autoloads tramp-imap
epa epg epg-config imap-hash imap message ecomplete rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 qp
ietf-drums mailabbrev nnheader gmm-utils mailheader canlock sha1
hex-util hashcash mail-utils assoc tramp-gw tramp-fish tramp-cache
tramp-ftp tramp-cmds tramp auth-source gnus-util netrc time-date
password-cache format-spec tramp-compat trampver traverselisp grep-a-lot
grep+ compile+ compile- uniquify byte-opt hexview-mode htmlize
auto-install thingatpt bytecomp byte-compile smtpmail sendmail
modeline-posn shebang pp-c-l buff-menu+ delsel irfc muse-html
muse-xml-common pcomplete cus-edit cus-start muse-publish muse-project
muse-protocols info muse-regexps derived muse muse-nested-tags muse-mode
google winner filecache multi-term term ehelp elscreen-server
elscreen-w3m mule-util alist elscreen multi-shell edmacro kmacro
ansi-color shell type-break sunrise-commander advice help-fns
advice-preload dired-x dired-aux dired w3m-load url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-util
url-parse timezone pym static apel-ver product url-vars mm-util
mail-prsvr mailcap esense-start erlang regexp-opt erlang-start
whitespace which-func imenu paren hl-line+ hl-line avoid ffap memo-pop
fixme-mode saveplace vc-dispatcher vc-svn emacsw32 find-func w32-meta
menuacc tmm electric w32-integ w32shell w32-grep grep compile comint
ring rebind ourcomments-widgets new-key-seq-widget wid-edit hfyview
easy-mmode easymenu htmlfontify cl cl-19 noprint nxhtml-loader
emacsw32-eol tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32
disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev button minibuffer faces cus-face text-properties overlay md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process multi-tty emacs)

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Eli Zaretskii]

> Date: Wed, 08 Dec 2010 00:26:15 +0100
> From: Vida Gábor <vidagabor@gmail.com>
> 
> This problem comes only if `smtpmail-queue-mail' is set to `t'.
> 
> When I'm ready with the mail, smtpmail-send-it is called and it tries to
> save the mail to disk. It is ok, but if I open the file with
> find-file-literally, then I see an additional ^M character at the end of
> each line. This causes problem (at least with MS Exchange) when I send
> it: the mail received by the recipient has empty body. This is sure that
> this additional ^M causes the problem (if I set smtpmail-queue-mail to
> nil, the smtpmail trace shows only one ^M at the end of each line).
> 
> I tried to understand what and where is the problem in the code, but the
> only conclusion was that if smtpmail reads the saved mail without
> conversion (coding-system-for-read set to 'no-conversion), then it
> should save it using Unix-style line endings.

I think your analysis is correct.  Could you please try the patch
below?  If it works for you, I will install it on the Emacs 23 branch,
so that it will be available in the upcoming Emacs 23.3 release.

Thanks.

=== modified file 'lisp/mail/smtpmail.el'
--- lisp/mail/smtpmail.el	2010-12-09 01:46:03 +0000
+++ lisp/mail/smtpmail.el	2010-12-10 14:32:28 +0000
@@ -392,7 +392,13 @@ The list is in preference order.")
 		(make-directory smtpmail-queue-dir t))
 	      (with-current-buffer buffer-data
 		(erase-buffer)
-		(set-buffer-file-coding-system smtpmail-code-conv-from nil t)
+		(set-buffer-file-coding-system
+		 ;; We will be reading the file with no-conversion in
+		 ;; smtpmail-send-queued-mail below, so write it out
+		 ;; with Unix EOLs.
+		 (coding-system-change-eol-conversion smtpmail-code-conv-from
+						      'unix)
+		 nil t)
 		(insert-buffer-substring tembuf)
 		(write-file file-data)
 		(set-buffer buffer-elisp)

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Vida Gábor]

>  I think your analysis is correct.  Could you please try the patch
>  below?  If it works for you, I will install it on the Emacs 23 branch,
>  so that it will be available in the upcoming Emacs 23.3 release.

Still doesn't work. I put some (message ...) to see the value of some
variables during the execution of smtpmail-send-it and I think the
problem is possibly somewhere else.

smtpmail-code-conv-from is nil and the return value of
(coding-system-change-eol-conversion ...) is also nil right after the
function call of (set-buffer-file-coding-system ...) that you modified
in your patch.

In the body of the very first let in smtpmail-send-it, just after the
varlist, the enable-multibyte-characters is nil, and thus
smtpmail-code-conv-from is also nil. But the global vaule of
enable-multibyte-characters is t. Do you think it helps anything if I
investigate where enable-multibyte-characters is set to nil?

Here is the mail I wanted to send, the saved mail file and the trace of
SMTP:

To: vidagabor@gmail.com
Subject: probe mail
Date: Sat, 11 Dec 2010 10:02:47 +0100
Message-ID: <kod0mxocof60.fsf@ericsson.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (windows-nt)
Gcc: nnimap+nedu:incoming
From: "Vida Gábor" <vidagabor@gmail.com>

a
b
c

-- 
Üdv,
Gábor

------------------------------------------------------------------------------

From: =?utf-8?Q?Vida_G=C3=A1bor?= <vidagabor@gmail.com>^M
To: vidagabor@gmail.com^M
Subject: probe mail^M
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (windows-nt)^M
Date: Sat, 11 Dec 2010 10:03:37 +0100^M
Message-ID: <kod0lj3wof4m.fsf@gmail.com>^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: base64^M
^M
YQ0KYg0KYw0KDQotLSANCsOcZHYsDQpHw6Fib3INCg==^M

------------------------------------------------------------------------------

Process SMTP deleted
220 nedu.dyndns.org ESMTP Postfix^M
EHLO EV001A4B593FDA^M
250-nedu.dyndns.org^M
250-PIPELINING^M
250-SIZE 50000000^M
250-VRFY^M
250-ETRN^M
250-ENHANCEDSTATUSCODES^M
250-8BITMIME^M
250 DSN^M
MAIL FROM:<gabor.v.vida@ericsson.com> SIZE=401^M
250 2.1.0 Ok^M
RCPT TO:<vidagabor@gmail.com>^M
250 2.1.5 Ok^M
DATA^M
354 End data with <CR><LF>.<CR><LF>^M
From: =?utf-8?Q?Vida_G=C3=A1bor?= <vidagabor@gmail.com>^M^M
To: vidagabor@gmail.com^M^M
Subject: probe mail^M^M
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (windows-nt)^M^M
Date: Sat, 11 Dec 2010 10:03:37 +0100^M^M
Message-ID: <kod0lj3wof4m.fsf@gmail.com>^M^M
MIME-Version: 1.0^M^M
Content-Type: text/plain; charset=utf-8^M^M
Content-Transfer-Encoding: base64^M^M
^M^M
YQ0KYg0KYw0KDQotLSANCsOcZHYsDQpHw6Fib3INCg==^M^M
.^M
250 2.0.0 Ok: queued as A2FB814F727^M
QUIT^M
221 2.0.0 Bye^M

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Eli Zaretskii]

> Date: Sat, 11 Dec 2010 10:43:47 +0100
> From: Vida Gábor <vidagabor@gmail.com>
> Cc: 7589@debbugs.gnu.org
> 
> smtpmail-code-conv-from is nil and the return value of
> (coding-system-change-eol-conversion ...) is also nil right after the
> function call of (set-buffer-file-coding-system ...) that you modified
> in your patch.

Sorry, missed that use-case.  How about the patch below?  (It is
relative to the original version of smtpmail.el.)

=== modified file 'lisp/mail/smtpmail.el'
--- lisp/mail/smtpmail.el	2010-12-09 01:46:03 +0000
+++ lisp/mail/smtpmail.el	2010-12-11 10:03:04 +0000
@@ -392,7 +392,14 @@ The list is in preference order.")
 		(make-directory smtpmail-queue-dir t))
 	      (with-current-buffer buffer-data
 		(erase-buffer)
-		(set-buffer-file-coding-system smtpmail-code-conv-from nil t)
+		(set-buffer-file-coding-system
+		 ;; We will be reading the file with no-conversion in
+		 ;; smtpmail-send-queued-mail below, so write it out
+		 ;; with Unix EOLs.
+		 (coding-system-change-eol-conversion
+		  (or smtpmail-code-conv-from 'undecided)
+		  'unix)
+		 nil t)
 		(insert-buffer-substring tembuf)
 		(write-file file-data)
 		(set-buffer buffer-elisp)

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Vida Gábor]

Hi Eli!

The correction works fine now according to the SMTP trace. Tomorrow I
will try it with an Exchange server (Postfix eated the previous, invalid
^M^M line ending), and notify you about the result, but I'm sure there
won't be any problem).

Thanks and regards,
Gábor

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Eli Zaretskii]

> Date: Sun, 12 Dec 2010 20:40:45 +0100
> From: Vida Gábor <vidagabor@gmail.com>
> Cc: 7589@debbugs.gnu.org
> 
> The correction works fine now according to the SMTP trace. Tomorrow I
> will try it with an Exchange server (Postfix eated the previous, invalid
> ^M^M line ending), and notify you about the result, but I'm sure there
> won't be any problem).

Thanks for testing.  I will commit the change as soon as I hear from
you that it works with Exchange.

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Vida Gábor]

Eli Zaretskii <eliz@gnu.org> writes:
> Thanks for testing.  I will commit the change as soon as I hear from
> you that it works with Exchange.

I can riport that the patched smtpmail.el works fine Exchange. Thanks
for the correction.

bug#7589: smtpmail: bad line endings in queued mails on Windows

[Eli Zaretskii]

> Date: Mon, 13 Dec 2010 09:59:55 +0100
> From: Vida Gábor <vidagabor@gmail.com>
> Cc: 7589@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Thanks for testing.  I will commit the change as soon as I hear from
> > you that it works with Exchange.
> 
> I can riport that the patched smtpmail.el works fine Exchange. Thanks
> for the correction.

Thanks; installed on the emacs-23 branch for Emacs 23.3.