Emacs and TLS support

Emacs and TLS support

[Angelo Graziosi]

Bootstrapping rev.101632, 'configuring' says:

[...]
checking for gnutls >= 2.2.4... yes
checking LIBGNUTLS_CFLAGS...
checking LIBGNUTLS_LIBS... -lgnutls -ltasn1
[...]
   Does Emacs use -lgnutls (BROKEN)?                       yes
[...]

Why '(BROKEN)'?

This occurs both on GNU/Linux K10.04 and Cygwin with gnutls-2.8.5 
(K10.04) and gnutls-2.8.6 (Cygwin) packages.

Ciao,
Angelo.

Re: Emacs and TLS support

[Lars Magne Ingebrigtsen]

Angelo Graziosi <angelo.graziosi@alice.it> writes:

>   Does Emacs use -lgnutls (BROKEN)?                       yes
> [...]
>
> Why '(BROKEN)'?

The tls support doesn't currently work.  So if you try to use it, you'll
find that it's broken.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Emacs and TLS support

[Ted Zlatanov]

On Sun, 26 Sep 2010 18:22:24 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Angelo Graziosi <angelo.graziosi@alice.it> writes:
>> Does Emacs use -lgnutls (BROKEN)?                       yes
>> [...]
>> 
>> Why '(BROKEN)'?

LMI> The tls support doesn't currently work.  So if you try to use it, you'll
LMI> find that it's broken.

Yeah, I couldn't find a better way of saying "it should work but Ted
couldn't figure out why handshaking fails" :)

On Sun, 26 Sep 2010 17:32:10 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> gnutls: handshake: handshaking
>> gnutls.el: (err=[-9] A TLS packet with unexpected length was received.) handshake: nil
>> Ouch, error return -9 (A TLS packet with unexpected length was received.)
>> nil
>> Mark set [2 times]

LMI> (open-ssl-stream "tls" "tls-buffer" "imap.gmail.com" "imaps")

LMI> Yes, I'm getting the same:

LMI> gnutls.el: (err=[gnutls-e-again] Resource temporarily unavailable, try again.) handshake: nil
LMI> gnutls: handshake: handshaking [2 times]
LMI> gnutls.el: (err=[-15] An unexpected TLS packet was received.) handshake: nil
LMI> Ouch, error return -15 (An unexpected TLS packet was received.)

I stepped through the GnuTLS function calls and couldn't find a problem
with the credential structures.  It's frustrating that the exact same
code works for the example client in GnuTLS but breaks in Emacs (most of
my time debugging this was spent double-checking that the same functions
are called with the same parameters in both cases).

On Sun, 26 Sep 2010 17:16:28 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> joakim@verona.se writes:

>> | gcc: @LIBGNUTLS_CFLAGS@: No such file or directory

LMI> Ok; I've now fixed this, and am checking in now.

Thanks!

Ted

Re: Emacs and TLS support

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I stepped through the GnuTLS function calls and couldn't find a problem
> with the credential structures.  It's frustrating that the exact same
> code works for the example client in GnuTLS but breaks in Emacs (most of
> my time debugging this was spent double-checking that the same functions
> are called with the same parameters in both cases).

I know nothing about tls, but have you confirmed that the library calls
really work in non-blocking mode?  It's not uncommon to write libraries
that assume that socket connections are blocking...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Emacs and TLS support

[Ted Zlatanov]

On Sun, 26 Sep 2010 23:06:46 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I stepped through the GnuTLS function calls and couldn't find a problem
>> with the credential structures.  It's frustrating that the exact same
>> code works for the example client in GnuTLS but breaks in Emacs (most of
>> my time debugging this was spent double-checking that the same functions
>> are called with the same parameters in both cases).

LMI> I know nothing about tls, but have you confirmed that the library calls
LMI> really work in non-blocking mode?  It's not uncommon to write libraries
LMI> that assume that socket connections are blocking...

It would be great if the GnuTLS developers could comment.  But these
references seem to specifically confirm that non-blocking sockets should
work the way I'm doing it:

http://lists.gnupg.org/pipermail/gnutls-dev/2005-March/000839.html
http://www.gnu.org/software/gnutls/manual/html_node/The-transport-layer.html

I tried setting the low water value to 0 in `Fgnutls_handshake' after
`gnutls_transport_set_ptr2' but it didn't make a difference:

    gnutls_transport_set_lowat (state, 0);

So I removed it in the patch.  I sort of suspect right now that
recv/send are not working correctly so I need to provide custom versions
with `gnutls_transport_set_pull_function' and
`gnutls_transport_set_push_function'.  But I don't know enough about the
Emacs internals that set up processes, which are ridiculously
complicated because of all the supported platforms.  And Simon Josefsson
said his patch worked when he first wrote it, so I assumed that this
kind of deep surgery would not be required.

Ted

Re: Emacs and TLS support

[dhruva]

> From: Ted Zlatanov <tzz@lifelogs.com>
> To: emacs-devel@gnu.org
> Date: Sun, 26 Sep 2010 16:33:04 -0500
> Subject: Re: Emacs and TLS support
> On Sun, 26 Sep 2010 23:06:46 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>
> It would be great if the GnuTLS developers could comment.  But these
> references seem to specifically confirm that non-blocking sockets should
> work the way I'm doing it:
>
> http://lists.gnupg.org/pipermail/gnutls-dev/2005-March/000839.html
> http://www.gnu.org/software/gnutls/manual/html_node/The-transport-layer.html
>
> I tried setting the low water value to 0 in `Fgnutls_handshake' after
> `gnutls_transport_set_ptr2' but it didn't make a difference:
>
>    gnutls_transport_set_lowat (state, 0);
>
> So I removed it in the patch.  I sort of suspect right now that
> recv/send are not working correctly so I need to provide custom versions
> with `gnutls_transport_set_pull_function' and
> `gnutls_transport_set_push_function'.  But I don't know enough about the
> Emacs internals that set up processes, which are ridiculously
> complicated because of all the supported platforms.  And Simon Josefsson
> said his patch worked when he first wrote it, so I assumed that this
> kind of deep surgery would not be required.
>

Not sure if this is related to the thread. I am having problems using
gnutls to access my company (M$ Exchange) mail using imap.
This used to work ~2 weeks back though. I had not done any specific
setting to use gnutls-client but now I see it barfing with error
(unable to handshake). I did a bit of troubleshooting. I executed the
same program "gnutls-cli" from command line and found same error. I
later installed starttls package and used that to connect from command
line and it worked!
Now, I try to cajole gnus to use starttls, it just refuses to use it
and keeps defaulting to gnutls-cli or openssl. This is when I stopped
and decided to take a break.

If someone can help me use starttls through gnus, I can try to capture
the packets with gnutls-cli and starttls and hope it can throw some
light. Let me try doing that with command line since starttls works
and gnutls-cli does not.

-dhruva

Re: Emacs and TLS support

[Lars Magne Ingebrigtsen]

dhruva <dhruvakm@gmail.com> writes:

> Now, I try to cajole gnus to use starttls, it just refuses to use it
> and keeps defaulting to gnutls-cli or openssl. This is when I stopped
> and decided to take a break.

This is controlled by `tls-program'.

Guessing from your description of the problem, I'd say the most likely
problem is an invalid certificate on the IMAP server, which means that
you want:

(setq tls-program '("gnutls-cli --insecure -p %p %h"))

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Emacs and TLS support

[Jason Earl]

On Mon, Sep 27 2010, Lars Magne Ingebrigtsen wrote:

> dhruva <dhruvakm@gmail.com> writes:
>
>> Now, I try to cajole gnus to use starttls, it just refuses to use it
>> and keeps defaulting to gnutls-cli or openssl. This is when I stopped
>> and decided to take a break.
>
> This is controlled by `tls-program'.
>
> Guessing from your description of the problem, I'd say the most likely
> problem is an invalid certificate on the IMAP server, which means that
> you want:
>
> (setq tls-program '("gnutls-cli --insecure -p %p %h"))

Thank you.  This advice allowed me to use the bzr version of Emacs to
read email.  I have my own certificate authority.  Does anyone know how
you teach gnutls-cli to trust a signing certificate?

Oh, and my apologies to Lars for just sending this to him the first
time.

Jason

Re: Emacs and TLS support

[Simon Josefsson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 26 Sep 2010 23:06:46 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> I stepped through the GnuTLS function calls and couldn't find a problem
>>> with the credential structures.  It's frustrating that the exact same
>>> code works for the example client in GnuTLS but breaks in Emacs (most of
>>> my time debugging this was spent double-checking that the same functions
>>> are called with the same parameters in both cases).
>
> LMI> I know nothing about tls, but have you confirmed that the library calls
> LMI> really work in non-blocking mode?  It's not uncommon to write libraries
> LMI> that assume that socket connections are blocking...
>
> It would be great if the GnuTLS developers could comment.  But these
> references seem to specifically confirm that non-blocking sockets should
> work the way I'm doing it:
>
> http://lists.gnupg.org/pipermail/gnutls-dev/2005-March/000839.html
> http://www.gnu.org/software/gnutls/manual/html_node/The-transport-layer.html

Yes, it should work both in blocking and non-blocking.

> I tried setting the low water value to 0 in `Fgnutls_handshake' after
> `gnutls_transport_set_ptr2' but it didn't make a difference:
>
>     gnutls_transport_set_lowat (state, 0);
>
> So I removed it in the patch.  I sort of suspect right now that
> recv/send are not working correctly so I need to provide custom versions
> with `gnutls_transport_set_pull_function' and
> `gnutls_transport_set_push_function'.  But I don't know enough about the
> Emacs internals that set up processes, which are ridiculously
> complicated because of all the supported platforms.  And Simon Josefsson
> said his patch worked when he first wrote it, so I assumed that this
> kind of deep surgery would not be required.

I don't know Emacs internals well enough, but it may be that replacing
the send/recv functions could make things more reliable...

I don't have a lot of time to help here alas, and when I tried building
Emacs from CVS the other day it just crashed...  maybe you could provide
simple step-by-step instructions to get something building that I can
test?  With some specific CVS revision that is known working.

/Simon

Re: Emacs and TLS support

[Ted Zlatanov]

On Thu, 30 Sep 2010 12:18:26 +0200 Simon Josefsson <simon@josefsson.org> wrote: 

SJ> I don't know Emacs internals well enough, but it may be that replacing
SJ> the send/recv functions could make things more reliable...

It turned out to be better to do the handshake at the C level.  It's not
ideal but at least it works now.

SJ> I don't have a lot of time to help here alas, and when I tried building
SJ> Emacs from CVS the other day it just crashed...  maybe you could provide
SJ> simple step-by-step instructions to get something building that I can
SJ> test?  With some specific CVS revision that is known working.

Sure.  See http://www.emacswiki.org/emacs/BzrForEmacsDevs for all the
details.

Thanks for your help.  I'll follow up to the other comments too.

Ted