bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Christoph]

I use the following setup:

Emacs 24.0.50 r103346
GnuTLS 2.10.1 (from http://josefsson.org/gnutls4win/)
GnuGPG 1.4.11

running on Windows 7.

My mail setup is as follows:

(require 'smtpmail)
(setq send-mail-function 'smtpmail-send-it
      message-send-mail-function 'smtpmail-send-it
      smtpmail-starttls-credentials '(("smtp.gmail.com" 587 nil nil))
      smtpmail-auth-credentials (expand-file-name "~/.authinfo.gpg")
      smtpmail-default-smtp-server "smtp.gmail.com"
      smtpmail-smtp-server "smtp.gmail.com"
      smtpmail-smtp-service 587
      user-mail-address "xxx@gmail.com"
      smtpmail-debug-info t)

Sending a mail fails with the following error:

Sending via mail...
220 mx.google.com ESMTP ca7sm2512icb.12
250-mx.google.com at your service, [71.208.192.11]
250-SIZE 35651584
250-8BITMIME
250-STARTTLS
250 ENHANCEDSTATUSCODES
c:/Users/Christoph/A: 0% (0/105)
c:/Users/Christoph/A: 100% (105/105)
530 5.7.0 Must issue a STARTTLS command first. ca7sm2512icb.12
221 2.0.0 closing connection ca7sm2512icb.12
smtpmail-send-it: Sending failed; SMTP protocol error


The SMTP trace buffer contains this:

Process SMTP deleted
220 mx.google.com ESMTP ca7sm2512icb.12
EHLO MARVIN
250-mx.google.com at your service, [71.208.192.11]
250-SIZE 35651584
250-8BITMIME
250-STARTTLS
250 ENHANCEDSTATUSCODES
MAIL FROM:<xxx@gmail.com> SIZE=278
530 5.7.0 Must issue a STARTTLS command first. ca7sm2512icb.12
QUIT
221 2.0.0 closing connection ca7sm2512icb.12



In GNU Emacs 24.0.50.1 (i386-mingw-nt6.1.7600)
 of 2011-02-18 on MARVIN
Windowing system distributor `Microsoft Corp.', version 6.1.7600
configured using `configure --with-gcc (4.5) --cflags -IC:/Progra~2/GnuWin32/include -ID:/devel/emacs/libXpm-3.5.8/include -ID:/devel/emacs/libXpm-3.5.8/src'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  autopair-mode: t
  autopair-global-mode: t
  ido-everywhere: t
  yas/global-mode: t
  yas/minor-mode: t
  global-auto-revert-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t

Recent input:
C-x RET r e p o r t <tab> <return>

Recent messages:
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/windows.el (source)...done
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/plugins.el (source)...
Ido mode enabled
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/anything-c-adaptive-history...done
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/plugins.el (source)...done
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/custom.el (source)...done
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/gnus.el (source)...done
Loading c:/Users/Christoph/AppData/Roaming/.emacs.d/init-custom.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
~/.emacs.d/plugins/python hides d:/devel/emacs/emacs-bzr/trunk/lisp/progmodes/python

Features:
(shadow sort gnus-cite mail-extr gnus-msg gnus-art mm-uu mml2015
epg-config mm-view mml-smime smime dig mailcap nnir gnus-sum macroexp
nnoo gnus-group gnus-undo nnmail mail-source format-spec gnus-start
gnus-spec gnus-int gnus-range gnus-win message rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev gmm-utils mailheader gnus gnus-ems nnheader
mail-utils wid-edit emacsbug url-util url-parse auth-source netrc
gnus-util time-date password-cache url-vars mm-util mail-prsvr
js2-mode-autoloads package zenburn-theme sha1 hex-util re-builder
ibuffer nav nav-tags nav-bufs uniquify anything-config warnings
browse-url semantic/util-modes semantic/util semantic semantic/tag
semantic/lex semantic/fw loaddefs eieio byte-opt bytecomp byte-compile
mode-local cedet imenu bookmark pp dired ffap thingatpt anything python
rx noutline outline ansi-color autopair google-c-style cc-mode cc-fonts
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
grep-o-matic grep compile comint regexp-opt browse-kill-ring+
browse-kill-ring second-sel ido yasnippet dropdown-list derived
easy-mmode easymenu assoc cl etags-table edmacro kmacro etags ring
org-install server advice help-fns advice-preload autorevert delsel
tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32 disp-table
ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe lisp-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
button minibuffer faces cus-face files text-properties overlay md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process multi-tty emacs)

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Claudio Bley]

At Fri, 18 Feb 2011 23:23:35 -0700,
Christoph wrote:
> Sending a mail fails with the following error:
> 
> Sending via mail...
> 220 mx.google.com ESMTP ca7sm2512icb.12
> 250-mx.google.com at your service, [71.208.192.11]
> 250-SIZE 35651584
> 250-8BITMIME
> 250-STARTTLS
> 250 ENHANCEDSTATUSCODES
> c:/Users/Christoph/A: 0% (0/105)
> c:/Users/Christoph/A: 100% (105/105)
> 530 5.7.0 Must issue a STARTTLS command first. ca7sm2512icb.12

As discussed in bug#7789 you cannot use the native gnutls-cli from
within emacs. Although it seems smtpmail didn't even try to use it as
the starttls feature is not loaded?!

You would need to use gnutls-cli from Cygwin and hack starttls.el. Or
wait for me to submit my GnuTLS changes for Woe32 and build Emacs with
gnutls support... ;)

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Christoph]

claudio.bley@gmail.com (Claudio Bley) writes:

> As discussed in bug#7789 you cannot use the native gnutls-cli from
> within emacs. Although it seems smtpmail didn't even try to use it as
> the starttls feature is not loaded?!

OK. Sorry for the dupe. This seems to be a similar problem.

> You would need to use gnutls-cli from Cygwin and hack starttls.el. Or
> wait for me to submit my GnuTLS changes for Woe32 and build Emacs with
> gnutls support... ;)

I can send mail using msmtp from Cygwin, but I was just wondering why
the gnutls-cli stuff didn't work. 

How soon can we expect these changes you talk about?

Christoph

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Claudio Bley]

At Tue, 22 Feb 2011 17:20:33 -0700,
Christoph wrote:
> 
> claudio.bley@gmail.com (Claudio Bley) writes:
> > You would need to use gnutls-cli from Cygwin and hack starttls.el. Or
> > wait for me to submit my GnuTLS changes for Woe32 and build Emacs with
> > gnutls support... ;)
> 
> I can send mail using msmtp from Cygwin, but I was just wondering why
> the gnutls-cli stuff didn't work. 
> 
> How soon can we expect these changes you talk about?

Until now I was hunting for some spurious bug which looked like a race
condition. (note to self: disable any arcane security software before
starting up the debugger next time)

Since I experimented with a few things the code needs some cleanup /
polishing now. I'll try to submit a patch next weekend.

-- 
Claudio

Re: bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Ted Zlatanov]

On Wed, 23 Feb 2011 22:13:16 +0100 claudio.bley@gmail.com (Claudio Bley) wrote: 

CB> At Tue, 22 Feb 2011 17:20:33 -0700,
CB> Christoph wrote:
>> 
>> claudio.bley@gmail.com (Claudio Bley) writes:
>> > You would need to use gnutls-cli from Cygwin and hack starttls.el. Or
>> > wait for me to submit my GnuTLS changes for Woe32 and build Emacs with
>> > gnutls support... ;)
>> 
>> I can send mail using msmtp from Cygwin, but I was just wondering why
>> the gnutls-cli stuff didn't work. 
>> 
>> How soon can we expect these changes you talk about?

CB> Until now I was hunting for some spurious bug which looked like a race
CB> condition. (note to self: disable any arcane security software before
CB> starting up the debugger next time)

CB> Since I experimented with a few things the code needs some cleanup /
CB> polishing now. I'll try to submit a patch next weekend.

That's terrific.  Thank you for working on that.

Are you interested in adding support for the more recent versions of
GnuTLS as well (2.10+)?  I stopped with the one generally available in
Debian at the time (see
http://permalink.gmane.org/gmane.emacs.gnus.general/75001).  It's on my
TODO list to upgrade to 2.10 and add callbacks for certificate
verification, but if you want to try it I would be grateful.

Ted

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Christoph]

claudio.bley@gmail.com (Claudio Bley) writes:

> Until now I was hunting for some spurious bug which looked like a race
> condition. (note to self: disable any arcane security software before
> starting up the debugger next time)

Now that sounds like fun times. ;)

> Since I experimented with a few things the code needs some cleanup /
> polishing now. I'll try to submit a patch next weekend.

Awesome. Much appreciated. I will give that a try then.

Christoph

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Stefan Monnier]

> condition. (note to self: disable any arcane security software before
> starting up the debugger next time)

Last I looked, all those "security" software are better off disabled all
the time.


        Stefan

bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Claudio Bley]

At Wed, 23 Feb 2011 15:47:05 -0600,
Ted Zlatanov wrote:
> 
> On Wed, 23 Feb 2011 22:13:16 +0100 claudio.bley@gmail.com (Claudio Bley) wrote: 
> 
> CB> At Tue, 22 Feb 2011 17:20:33 -0700,
> CB> Christoph wrote:
> >> 
> >> How soon can we expect these changes you talk about?
> 
> CB> Since I experimented with a few things the code needs some cleanup /
> CB> polishing now. I'll try to submit a patch next weekend.
> 
> That's terrific.  Thank you for working on that.
> 
> Are you interested in adding support for the more recent versions of
> GnuTLS as well (2.10+)?  I stopped with the one generally available in
> Debian at the time (see
> http://permalink.gmane.org/gmane.emacs.gnus.general/75001).  It's on my
> TODO list to upgrade to 2.10 and add callbacks for certificate
> verification, but if you want to try it I would be grateful.

Actually, I'm using GnuTLS 2.10 (2.10.1 and 2.10.4) right now. What
are the differences between 2.8 and 2.10? I would be glad to help out
implementing the missing callback support.

Incidently, I had a car crash yesterday. Nothing serious, still
feeling a bit dizzy, though. That probably means I won't have any time
preparing a patch this weekend. So, don't hold your breath on it...

Claudio

Re: bug#8080: 24.0.50; Sending mail failed with Gmail/GnuTLS on Windows

[Ted Zlatanov]

On Fri, 25 Feb 2011 19:42:14 +0100 claudio.bley@gmail.com (Claudio Bley) wrote: 

CB> At Wed, 23 Feb 2011 15:47:05 -0600,
CB> Ted Zlatanov wrote:

>> Are you interested in adding support for the more recent versions of
>> GnuTLS as well (2.10+)?  I stopped with the one generally available in
>> Debian at the time (see
>> http://permalink.gmane.org/gmane.emacs.gnus.general/75001).  It's on my
>> TODO list to upgrade to 2.10 and add callbacks for certificate
>> verification, but if you want to try it I would be grateful.

CB> Actually, I'm using GnuTLS 2.10 (2.10.1 and 2.10.4) right now. What
CB> are the differences between 2.8 and 2.10? I would be glad to help out
CB> implementing the missing callback support.

Thanks for your interest.  W32 users are the biggest reason for GnuTLS
support in Emacs: you've sen how hard it is to wrap the command-line
tools.  So your experience will be valuable.

2.10 is backwards compatible so everything works as you'd expect with
2.8.  That's good news.

The big difference is that 2.10 and higher have callbacks on credential
verification.  See gnutls_certificate_set_verify_function() (see
http://lists.gnu.org/archive/html/gnutls-devel/2010-06/msg00042.html for
the list of API/ABI changes in 2.10.0).

Note you have to use gnutls_malloc() and gnutls_free() in callbacks.

Also I think we should support (as in
http://www.gnu.org/software/gnutls/manual/html_node/Using-a-callback-to-select-the-certificate-to-use.html
which works in 2.8 as well) using
gnutls_certificate_client_set_retrieve_function() to set the callback to
cert_callback().  That way the client can decide which certificate to
present at connection time.

The reason is that if we write C-level verification code as in
http://www.gnu.org/software/gnutls/manual/html_node/Verifying-peer_0027s-certificate.html#Verifying-peer_0027s-certificate
it will be harder to manage the certificates and do fancy verification
on them.  That's my recollection anyhow, it's been a while since I
worked on that.  Take a look at `gnutls-negotiate' in net/gnutls.el for
the current state of things; it has a :callbacks key which is currently
unused.  I was going to make it an alist but let's continue the
discussion on emacs-devel after you've looked at the 2.10.x API, the
current gnutls.el, and maybe have some ideas about how to proceed.

I'm OK with changing the gnutls.el interface as long as it keeps working
with Gnus' proto-stream.el.  Lars Magne Ingebrigtsen can help us proceed
on that front.

CB> Incidently, I had a car crash yesterday. Nothing serious, still
CB> feeling a bit dizzy, though. That probably means I won't have any time
CB> preparing a patch this weekend. So, don't hold your breath on it...

Sorry to hear that.  Better take time to get better, of course.  If I
get something done before you, I'll be surprised :) 

Ted