bug#7789: cannot send smtpmail using gmail & tls on woe32

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790)
 of 2010-05-08 on G41R2F1
--with-gcc (3.4) --no-opt --cflags -Ic:/xpm/include

I followed the instructions on how to send smtp mail using emacs and gmail:
http://obfuscatedcode.wordpress.com/2007/04/26/configuring-emacs-for-gmails-smtp/
it works with the latest bzr emacs on ubuntu just fine, but does not
work on woe32.
first, it hangs forever with the message
Opening STARTTLS connection to `smtp.gmail.com:587'...done
and then it dies when the process times out:

Debugger entered--Lisp error: (error "Process SMTP not running")
  process-send-string(#<process SMTP> "EHLO SDS.SDS")
  smtpmail-send-command(#<process SMTP> "EHLO SDS.SDS")
  byte-code("....")
  smtpmail-via-smtp(("sds@gnu.org" "sds@gnu.org") #<buffer  smtpmail temp>)
  smtpmail-send-it()
  sds-send-mail()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
  call-interactively(message-send-and-exit nil nil)

(process-status process)
==> exit
-- 
Sam Steingold <http://sds.podval.org>

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> Date: Wed, 5 Jan 2011 10:05:59 -0500
> From: Sam Steingold <sds@gnu.org>
> Cc: 
> 
> GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790)
>  of 2010-05-08 on G41R2F1
> --with-gcc (3.4) --no-opt --cflags -Ic:/xpm/include
> 
> I followed the instructions on how to send smtp mail using emacs and gmail:
> http://obfuscatedcode.wordpress.com/2007/04/26/configuring-emacs-for-gmails-smtp/
> it works with the latest bzr emacs on ubuntu just fine, but does not
> work on woe32.

smtpmail works perfectly for me on MS-Windows, but I don't use tls.

What port of gnutls did you use?  If it's a Cygwin port, perhaps try a
MinGW port instead.

Also, if you set smtpmail-debug-info non-nil, what do you see in the
buffer named "*trace of SMTP session to <somewhere>*" after the
connection attempt times out?

Maybe also post here the setup on your .emacs for using smtpmail
(although I doubt that this is the culprit).

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

On Wed, Jan 5, 2011 at 11:16 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 5 Jan 2011 10:05:59 -0500
>> From: Sam Steingold <sds@gnu.org>
>>
>> GNU Emacs 23.2.1 (i386-mingw-nt5.2.3790)
>>  of 2010-05-08 on G41R2F1
>> --with-gcc (3.4) --no-opt --cflags -Ic:/xpm/include
>>
>> I followed the instructions on how to send smtp mail using emacs and gmail:
>> http://obfuscatedcode.wordpress.com/2007/04/26/configuring-emacs-for-gmails-smtp/
>> it works with the latest bzr emacs on ubuntu just fine, but does not
>> work on woe32.
>
> smtpmail works perfectly for me on MS-Windows, but I don't use tls.
>
> What port of gnutls did you use?

native win32 port.

> Also, if you set smtpmail-debug-info non-nil, what do you see in the
> buffer named "*trace of SMTP session to <somewhere>*" after the
> connection attempt times out?

Resolving 'smtp.gmail.com'...
Connecting to '74.125.113.109:587'...

- Simple Client Mode:

NOTE: you should run 'diskperf -y' to enable the disk statistics
NOTE: you should run 'diskperf -y' to enable the disk statistics
NOTE: you should run 'diskperf -y' to enable the disk statistics
NOTE: you should run 'diskperf -y' to enable the disk statistics
NOTE: you should run 'diskperf -y' to enable the disk statistics
220 mx.google.com ESMTP e10sm2834564vch.19
451 4.4.2 Timeout - closing connection. q5sm4981221vcr.39
- Peer has closed the GnuTLS connection

Process SMTP finished


> Maybe also post here the setup on your .emacs for using smtpmail
> (although I doubt that this is the culprit).

smtpmail-auth-credentials
=> "~/.netrc"

(custom-set-variables
 '(smtpmail-smtp-service 587)
 '(smtpmail-default-smtp-server "smtp.gmail.com")
 '(smtpmail-starttls-credentials
   `((,smtpmail-default-smtp-server ,smtpmail-smtp-service nil nil))))


-- 
Sam Steingold <http://sds.podval.org>

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

same behavior with the emacs built from the latets bzr sources

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> Date: Wed, 5 Jan 2011 15:06:27 -0500
> From: Sam Steingold <sds@gnu.org>
> Cc: 7789@debbugs.gnu.org
> 
> same behavior with the emacs built from the latets bzr sources

As expected: smtpmail didn't change a bit in this respect.

I don't know TLS enough to suggest how to debug this.  Can you connect
to the server from telnet using TLS?  If so, see if you at all get the
server's prompt in response to the login.

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Jason Rumney]

Sam Steingold <sds@gnu.org> writes:

>> smtpmail works perfectly for me on MS-Windows, but I don't use tls.
>>
>> What port of gnutls did you use?
>
> native win32 port.

Try the Cygwin port. I did get openssl working once, and I think it was
with the Cygwin port, but it wasn't easy.

Windows changes its I/O buffering when commands are run with I/O
redirected, and if the commands are not designed to be used in a pipe or
are direct Unix ports input is often buffered indefinitely.  We have
done all we can at the Emacs end to improve this. Unfortunately Cygwin
programs often use a Cygwin specific solution for this which results in
an error message like "no tty found" when they are run from native
Emacs, but it is still worth trying.

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

On Wed, Jan 5, 2011 at 6:09 PM, Jason Rumney <jasonr@gnu.org> wrote:
> Sam Steingold <sds@gnu.org> writes:
>
>>> smtpmail works perfectly for me on MS-Windows, but I don't use tls.
>>>
>>> What port of gnutls did you use?
>>
>> native win32 port.
>
> Try the Cygwin port. I did get openssl working once, and I think it was
> with the Cygwin port, but it wasn't easy.

I _started_ with the cygwin port.
I switched to the native port only under Eli's urging.

-- 
Sam Steingold <http://sds.podval.org>

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> Date: Wed, 5 Jan 2011 18:31:06 -0500
> From: Sam Steingold <sds@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 7789@debbugs.gnu.org
> 
> I _started_ with the cygwin port.
> I switched to the native port only under Eli's urging.

I'm sorry that I got you into extra effort that turned out to be
futile.

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

On Thu, Jan 6, 2011 at 4:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 5 Jan 2011 18:31:06 -0500
>> From: Sam Steingold <sds@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, 7789@debbugs.gnu.org
>>
>> I _started_ with the cygwin port.
>> I switched to the native port only under Eli's urging.
>
> I'm sorry that I got you into extra effort that turned out to be
> futile.

that's OK.
I am still having this problem (native bzr tip emacs; both cygwin &
mingw gnutls-cli).
any suggestions are welcome.

-- 
Sam Steingold <http://sds.podval.org>

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> Date: Thu, 6 Jan 2011 18:13:40 -0500
> From: Sam Steingold <sds@gnu.org>
> Cc: jasonr@gnu.org, 7789@debbugs.gnu.org
> 
> I am still having this problem (native bzr tip emacs; both cygwin &
> mingw gnutls-cli).
> any suggestions are welcome.

You posted the contents of the trace buffer on Windows.  What does
that buffer show on Ubuntu, where you say it works?  The differences
might give a clue for what's wrong on Windows.

Another idea would be to ask on gnu.emacs.help if anyone has got this
set up successfully on Windows, and how.

Failing all that, I'd suggest to run the related functions under
Edebug and see what fails and why (I'm guessing that it hangs
indefinitely somewhere waiting for something, until the timeout hits).
From a casual look in the sources, I'd say the relevant functions are
smtpmail-open-stream, smtpmail-via-smtp, and starttls-open-stream and
its subroutines.

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Sam Steingold]

On Fri, Jan 7, 2011 at 2:43 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 6 Jan 2011 18:13:40 -0500
>> From: Sam Steingold <sds@gnu.org>
>> Cc: jasonr@gnu.org, 7789@debbugs.gnu.org
>>
>> I am still having this problem (native bzr tip emacs; both cygwin &
>> mingw gnutls-cli).
>> any suggestions are welcome.
>
> You posted the contents of the trace buffer on Windows.  What does
> that buffer show on Ubuntu, where you say it works?  The differences
> might give a clue for what's wrong on Windows.

the major difference is the conspicuous presence of ^M ("\r") chars in
the smtp trace buffer on windows.
when I use cygwin gnutls-cli, _all_ lines contain it.
when I use the native port, only _some_ lines do.
I wonder if the unconditional setting of coding-system in smtpmail.el
is the culprit.

-- 
Sam Steingold <http://sds.podval.org>

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> Date: Fri, 7 Jan 2011 13:55:25 -0500
> From: Sam Steingold <sds@gnu.org>
> Cc: jasonr@gnu.org, 7789@debbugs.gnu.org
> 
> the major difference is the conspicuous presence of ^M ("\r") chars in
> the smtp trace buffer on windows.
> when I use cygwin gnutls-cli, _all_ lines contain it.
> when I use the native port, only _some_ lines do.
> I wonder if the unconditional setting of coding-system in smtpmail.el
> is the culprit.

What unconditional setting? where in smtpmail.el do you see it?

In general, the encoding of incoming and outgoing data for
communications with subprocesses can be tuned with
process-coding-system-alist (unless smtpmail overrides that).

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Claudio Bley]

Hi.

At Thu, 6 Jan 2011 18:13:40 -0500,
Sam Steingold wrote:
> I am still having this problem (native bzr tip emacs; both cygwin &
> mingw gnutls-cli).
> any suggestions are welcome.

The problem is that Woe32 does not support signals. So, Emacs "Only
handle(s) signals that will result in the process dying" (see
src/w32proc.c:1427 sys_kill).

gnutls-cli waits for a SIGALRM to initiate the STARTTLS handshake --
which Emacs isn't able to send -- or, alternatively, an EOF -- which
doesn't work because communication is done over a pipe instead of a
PTY.

I'm using cygwin's gnutls-cli and have hacked ssl.el in order to
replace the signal-process calls with (call-process "kill.exe" nil nil
nil "-ALRM" PID). This works because cygwin provides its own layer of
signal handling and is able to send / receive the SIGALRM signal.

Make sure to call cygwin's kill.exe, not the one from MinGW.

It would be nice if one would be able to use starttls natively under
Woe32 by using some other means instead of signals to communicate with
it, e.g. just using windows messages (WM_COPYDATA)?

- Claudio

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> From: claudio.bley@gmail.com (Claudio Bley)
> Date: Mon, 17 Jan 2011 09:09:10 +0100
> 
> gnutls-cli waits for a SIGALRM to initiate the STARTTLS handshake --
> which Emacs isn't able to send -- or, alternatively, an EOF -- which
> doesn't work because communication is done over a pipe instead of a
> PTY.

Is this a bug in the ported gnutls, in Emacs, or in both?

> I'm using cygwin's gnutls-cli and have hacked ssl.el in order to
> replace the signal-process calls with (call-process "kill.exe" nil nil
> nil "-ALRM" PID). This works because cygwin provides its own layer of
> signal handling and is able to send / receive the SIGALRM signal.

How about making that hack part of Emacs?  It could be conditioned on
running on Windows.

TIA

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Claudio Bley]

At Mon, 17 Jan 2011 17:47:28 -0500,
Eli Zaretskii wrote:
> 
> > From: claudio.bley@gmail.com (Claudio Bley)
> > Date: Mon, 17 Jan 2011 09:09:10 +0100
> > 
> > gnutls-cli waits for a SIGALRM to initiate the STARTTLS handshake --
> > which Emacs isn't able to send -- or, alternatively, an EOF -- which
> > doesn't work because communication is done over a pipe instead of a
> > PTY.
> 
> Is this a bug in the ported gnutls, in Emacs, or in both?

I'd say it's a deficiency of the platform.

Woe32 has no signal and no PTY support. So, the signal support
has been ifdef'ed out in gnutls and Emacs for Woe32.

> > I'm using cygwin's gnutls-cli and have hacked ssl.el in order to
> > replace the signal-process calls with (call-process "kill.exe" nil nil
> > nil "-ALRM" PID). This works because cygwin provides its own layer of
> > signal handling and is able to send / receive the SIGALRM signal.
> 
> How about making that hack part of Emacs?  It could be conditioned on
> running on Windows.

You mean to distribute cygwin's kill.exe with Emacs and just using it
instead of `signal-process' everywhere? Or to depend upon the user to
install a cygwin environment along with Emacs?

I'd be a bit reluctant to do that since it seems a bit awkward...

IMHO, it would be better to let the programs work together on all
platforms using different means of notification where necessary,
e.g. using events on windows instead of signals...?! But that would
indeed require an appropriate change on both sides.

-- 
Claudio

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Eli Zaretskii]

> From: claudio.bley@gmail.com (Claudio Bley)
> Date: Tue, 18 Jan 2011 16:33:41 +0100
> Cc: 
> 
> > > gnutls-cli waits for a SIGALRM to initiate the STARTTLS handshake --
> > > which Emacs isn't able to send -- or, alternatively, an EOF -- which
> > > doesn't work because communication is done over a pipe instead of a
> > > PTY.
> > 
> > Is this a bug in the ported gnutls, in Emacs, or in both?
> 
> I'd say it's a deficiency of the platform.

A port that doesn't take platform deficiencies into consideration is a
broken port.  I was asking where should the correction be: in gnutls
or in Emacs, or in both?

> Woe32 has no signal and no PTY support. So, the signal support
> has been ifdef'ed out in gnutls and Emacs for Woe32.

If it has been ifdefed out, how are users supposed to do on Windows
what they do on GNU/Linux by using signals?

> > > I'm using cygwin's gnutls-cli and have hacked ssl.el in order to
> > > replace the signal-process calls with (call-process "kill.exe" nil nil
> > > nil "-ALRM" PID). This works because cygwin provides its own layer of
> > > signal handling and is able to send / receive the SIGALRM signal.
> > 
> > How about making that hack part of Emacs?  It could be conditioned on
> > running on Windows.
> 
> You mean to distribute cygwin's kill.exe with Emacs and just using it
> instead of `signal-process' everywhere? Or to depend upon the user to
> install a cygwin environment along with Emacs?

The latter, and also that hack in ssl.el you need for that.

> I'd be a bit reluctant to do that since it seems a bit awkward...

Is there a better way that's practical?  It is more awkward to ask
users to change platforms, or tell them to fix gnutls by themselves,
no?

> IMHO, it would be better to let the programs work together on all
> platforms using different means of notification where necessary,
> e.g. using events on windows instead of signals...?! But that would
> indeed require an appropriate change on both sides.

Exactly.  I'm trying to establish whether there's a less painful way,
even if it's less elegant.

Thanks.

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Claudio Bley]

At Tue, 18 Jan 2011 17:45:42 -0500,
Eli Zaretskii wrote:
> 
> > From: claudio.bley@gmail.com (Claudio Bley)
> > Date: Tue, 18 Jan 2011 16:33:41 +0100
> > Cc: 
> > 
> > > > gnutls-cli waits for a SIGALRM to initiate the STARTTLS handshake --
> > > > which Emacs isn't able to send -- or, alternatively, an EOF -- which
> > > > doesn't work because communication is done over a pipe instead of a
> > > > PTY.
> > > 
> > > Is this a bug in the ported gnutls, in Emacs, or in both?
> > 
> > I'd say it's a deficiency of the platform.
> 
> A port that doesn't take platform deficiencies into consideration is a
> broken port.  I was asking where should the correction be: in gnutls
> or in Emacs, or in both?

Well, incorporate GNU TLS into Emacs and you're done. Any OS needs
cryptographic and security features nowadays, no?

> > Woe32 has no signal and no PTY support. So, the signal support
> > has been ifdef'ed out in gnutls and Emacs for Woe32.
> 
> If it has been ifdefed out, how are users supposed to do on Windows
> what they do on GNU/Linux by using signals?

Since you only need to signal gnutls-cli once that it should negotiate
the TLS connection, it would be sufficient to set a boolean flag of
some sort. One may use an event e.g. called "gnutls-cli-<PID>" and
provide an auxiliary program which is able to signal events by a given
name.

You could also use a socket. Since Emacs is already able to use
sockets that would only require a change in GNU TLS.

> > > > I'm using cygwin's gnutls-cli and have hacked ssl.el in order to
> > > > replace the signal-process calls with (call-process "kill.exe" nil nil
> > > > nil "-ALRM" PID). This works because cygwin provides its own layer of
> > > > signal handling and is able to send / receive the SIGALRM signal.
> > > 
> > > How about making that hack part of Emacs?  It could be conditioned on
> > > running on Windows.
> > 
> > You mean to distribute cygwin's kill.exe with Emacs and just using it
> > instead of `signal-process' everywhere? Or to depend upon the user to
> > install a cygwin environment along with Emacs?
> 
> The latter, and also that hack in ssl.el you need for that.

Sorry, I actually meant to say that I hacked starttls.el for that
matter.

> Is there a better way that's practical?  It is more awkward to ask
> users to change platforms, or tell them to fix gnutls by themselves,
> no?
> 
> > IMHO, it would be better to let the programs work together on all
> > platforms using different means of notification where necessary,
> > e.g. using events on windows instead of signals...?! But that would
> > indeed require an appropriate change on both sides.
> 
> Exactly.  I'm trying to establish whether there's a less painful way,
> even if it's less elegant.

I just discovered that gnutls-cli is barely usable compiled on native
Woe32 because it uses (must use) polling of stdin, stdout file
descriptors which means 100% CPU usage when idle and connected to a
pipe instead of a real console. Oh my...

So, it seems that the least painful way is to use Cygwin's gnutls-cli
and its kill command. It's still painful, though. And it's actually a
bit slow compared to running it under Linux -- maybe that's because of
the extra layer(s) of Cygwin...

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Claudio Bley]

At Fri, 21 Jan 2011 15:32:33 +0100,
Claudio Bley wrote:
> 
> At Tue, 18 Jan 2011 17:45:42 -0500,
> Eli Zaretskii wrote:
> > 
> > A port that doesn't take platform deficiencies into consideration is a
> > broken port.  I was asking where should the correction be: in gnutls
> > or in Emacs, or in both?
> 
> Well, incorporate GNU TLS into Emacs and you're done. Any OS needs
> cryptographic and security features nowadays, no?

Apparently, this has been already done. One could compile Emacs with
GNUTLS support. But, of course it doesn't work on Woe32... Go figure!

Long story short, I made it work somehow. I'm using it with POP3,
IMAP4 and SMTP using SSL/TLS and STARTTLS.

It's still a bit flaky when trying to connect to a server (probably
some kind of race condition), but once connected it works like a
charm.

I'll prepare a patch when I find the time after work.

-- 
Claudio

bug#7789: cannot send smtpmail using gmail & tls on woe32

[Glenn Morris]

Version: 24.1

IIUC, the MS Windows build can now use gnutls.c and so this can be closed.
(I might not UC though.)