Re: gnutls status

Re: gnutls status

[Ted Zlatanov]

On Fri, 26 Nov 2010 01:28:46 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Julien Danjou <julien@danjou.info> writes:
>> gnutls support has been added to Emacs 24. What's the status of it?

LMI> It seems to work.

It's missing some features.  See
http://thread.gmane.org/gmane.emacs.devel/131441/focus=131551 and the
rest of that thread for the details.  The biggest one is callbacks.

We need callbacks to implement host name verification and certificate
chain checking, which are both IMO essential to making the Emacs GnuTLS
support "official."  In GnuTLS 2.8.x you can't set a callback function,
so the C code would need (from doc/examples/ex-rfc2818.c):

  if (gnutls_x509_crt_get_expiration_time (cert) < time (0))
    {
      printf ("The certificate has expired\n");
      return;
    }

  if (gnutls_x509_crt_get_activation_time (cert) > time (0))
    {
      printf ("The certificate is not yet activated\n");
      return;
    }

  if (!gnutls_x509_crt_check_hostname (cert, hostname))
    {
      printf ("The certificate's owner does not match hostname '%s'\n",
              hostname);
      return;
    }

which is very inflexible compared to a callback function.  We'd need to
add custom API options for each of the three checks above plus another
for the certificate chain verification; in addition it would be harder
to interact with the user and store trusted certificates from C.

2.10.x and above let us set a callback function, which would make all of
the above easier and more convenient from ELisp-land.  The problem is
that 2.10.x hasn't been widely adopted in Debian and thus won't work by
default.

So we'd need to either 1) require 2.10.x, or 2) complicate the C code
and API using just 2.8.x features and maybe figure out how to set up our
own callback mechanism, or 3) use the 2.10.x features only when it's
available, using autoconf detection, which is twice as complicated as
(2).

(2) and (3) will require a lot of new code that is completely
unnecessary under 2.10.x.

This is essentially why I haven't worked on the GnuTLS support in a bit:
I don't know the best way forward.  If anyone can suggest a good way to
do it, I'm all ears.  I also don't know about 2.10.x's status in the
major distros and whether Emacs can require that version or higher
specifically.

>> Could we use it in Gnus? Or is this still an Emacs side problem to
>> resolve?

LMI> nnimap has support for using it, but it's probably the wrong place to
LMI> put the support.  It should be put into tls.el, probably, so that all
LMI> the users don't have to know about the stuff...

I think every package should explicitly choose to support gnutls.el, it
shouldn't be an Emacs-wide choice.  There's too many configuration
options that depend on the purpose.  For instance IMAP and HTTPS have
really different security needs.  Also, GnuTLS is much more configurable
than the older command-line interfaces so normalizing the API is not so
easy.  This is why I removed the old tls.el compatibility code from
gnutls.el and it only has gnutls-* functions now (`open-gnutls-stream'
and `gnutls-negotiate' being the main entry points from the ELisp side,
while `gnutls-boot' is the main entry point from the C side).

IMO tls.el and friends should be made deprecated as soon as gnutls.el is
capable to verify a certificate chain, the expiration date, and the
hostname.  They are insecure and the cause of many bug reports over the
years, especially on the W32 platform.

As with the C issues above, if you have opinions or suggestions on how
to do all this better, including Gnus but also for Emacs in general,
tell me.  I'm CC-ing to emacs-devel as well.

Ted

Re: gnutls status

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1977 bytes --]

On Fri, Nov 26 2010, Ted Zlatanov wrote:

> 2.10.x and above let us set a callback function, which would make all of
> the above easier and more convenient from ELisp-land.  The problem is
> that 2.10.x hasn't been widely adopted in Debian and thus won't work by
> default.

It's in experimental. It won't go in sid until squeeze is released,
which should be very soon now. So there's nothing to wait IMHO.
At the time Emacs 24 will be released, gnutls 2.10 would have been
since a very long in Debian. :)

> This is essentially why I haven't worked on the GnuTLS support in a bit:
> I don't know the best way forward.  If anyone can suggest a good way to
> do it, I'm all ears.  I also don't know about 2.10.x's status in the
> major distros and whether Emacs can require that version or higher
> specifically.

My opinion is go with 2.10.

Some distro do not update often packages if they're is no urgent need
to, anyhow. So you will just be pushing them a bit, which is not a bad
thing. :)

Gentoo, OpenBSD, FreeBSD already have it FWIW. And Debian in
experimental, so it will end up in Debian and Ubuntu in a couple of
month top[1].

This is the development model I used for the last years in various
project like the awesome window manager, often requiring recent version
of libraries it uses. I consider I don't have enough time to write
compatibility code for older library release, while smart people write
better interface for us to use.
The only downside is for people using old OS for whatever reason. But
they *will have* to upgrade other time anyhow, so why not doing sooner
than later? Since we're talking a user program (not a production Web
server), there's no big counter-argument to this, IMHO.

Of course, YMMV, as your amount of spare time to be used writing
compatibility code. ;)

[1]  Depends if squeeze is released soon, of course.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: gnutls status

[Stefan Monnier]

>> 2.10.x and above let us set a callback function, which would make all of
>> the above easier and more convenient from ELisp-land.  The problem is
>> that 2.10.x hasn't been widely adopted in Debian and thus won't work by
>> default.

> It's in experimental. It won't go in sid until squeeze is released,
> which should be very soon now. So there's nothing to wait IMHO.
> At the time Emacs 24 will be released, gnutls 2.10 would have been
> since a very long in Debian. :)

Many people use Debian stable, so until it's not in Debian stable, it's
not really in Debian.


        Stefan

Re: gnutls status

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

On Fri, Nov 26 2010, Stefan Monnier wrote:

> Many people use Debian stable, so until it's not in Debian stable, it's
> not really in Debian.

But does many people use Debian stable with emacs development version?

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: gnutls status

[Stefan Monnier]

>> Many people use Debian stable, so until it's not in Debian stable, it's
>> not really in Debian.
> But does many people use Debian stable with emacs development version?

That's not the relevant question: the relevant question is whether 2.10
will be in Debian stable by the time Emacs-24 is released.  It might,
but I'm not completely sure.


        Stefan

Re: gnutls status

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

On Fri, Nov 26 2010, Stefan Monnier wrote:

> That's not the relevant question: the relevant question is whether 2.10
> will be in Debian stable by the time Emacs-24 is released.  It might,
> but I'm not completely sure.

Depends on how you think about it. Emacs 24 will be probably released
while Squeeze is stable, which does not have gnutls 2.10 (but may have
it through backports).

Next Debian stable would have Emacs 24, and gnutls 2.10.

So gnutls 2.10 will not be in Debian stable when Emacs 24 get released
(unless Emacs 24 is released after mid-2012). But most users won't try
to install Emacs 24 on it anyway.

OTOH Emacs 24 and gnutls 2.10 will be in the next Debian stable. :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: gnutls status

[Stefan Monnier]

> So gnutls 2.10 will not be in Debian stable when Emacs 24 get released
> (unless Emacs 24 is released after mid-2012). But most users won't try
> to install Emacs 24 on it anyway.

That's a good point.  Especially since gnutls is not indispensable for
Emacs, so as long as we properly handle the case where the gnutls library
to too old (either by not using it, or by using some fallback code),
then I'm OK with it.


        Stefan

Re: gnutls status

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Next Debian stable would have Emacs 24, and gnutls 2.10.
>
> So gnutls 2.10 will not be in Debian stable when Emacs 24 get released
> (unless Emacs 24 is released after mid-2012). But most users won't try
> to install Emacs 24 on it anyway.

True.  And people who use Emacs 24 on older machines can use the
external gnutls-cli/openssl solutions.  They're slower and slightly less
secure, but people have been living with that solution for decades, so
they can do so a year more.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnutls status

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> So we'd need to either 1) require 2.10.x, or 2) complicate the C code
> and API using just 2.8.x features and maybe figure out how to set up our
> own callback mechanism, or 3) use the 2.10.x features only when it's
> available, using autoconf detection, which is twice as complicated as
> (2).

Is 2.10.x at least backwards-compatible, so that if we do implement the
complicated 2.8.x features, it'll continue to work in the future, too?

> I think every package should explicitly choose to support gnutls.el, it
> shouldn't be an Emacs-wide choice.  There's too many configuration
> options that depend on the purpose.  For instance IMAP and HTTPS have
> really different security needs.

True.  On the other hand, look at `nnimap-open-connection' and weep.  It
started off as a simple function and grew into a monster.  That's why I
haven't adapted nntp/pop3/etc to use gnutls yet -- I don't want to have
to repeat the same code all over the place.

And the STARTTLS situation, in particular, is a nightmare, since the
sane gnutls way of doing it and the really awkward starttls.el way of
doing it require different code paths.

And we're going to have to support gnutls/tls.el/starttls.el in all the
connections for the unforeseeable future.  I kinda want to write a new
connection framework that'll just separate out all this cruft into a
separate package, but I can't really see a sensible unified interface.
Especially with the STARTTLS stuff.

I mean, ideally, in the future whenever you do a
pop3/imap/nntp/smtp/etc connection, Emacs should opportunistically
switch on STARTTLS if the server supports it.  Encryption is good.
Switching on STARTTLS is virtually free if you have gnutls.  It's really
expensive if you don't.  I mean, time-wise for the user.

But I don't really see an easy way to do that.  You'd need
protocol-specific callbacks and stuff.  And I hate frameworks.  And
it'll be brittle, anyway, since starttls.el is kinda brittle.

But I do think we need some kinda of compat library to avoid going
totally insane.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnutls status

[Ted Zlatanov]

On Fri, 26 Nov 2010 15:10:39 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Is 2.10.x at least backwards-compatible, so that if we do implement the
LMI> complicated 2.8.x features, it'll continue to work in the future, too?

Yes.  They try really hard to keep backwards compatibility.  I'd guess
for all 2.x releases we'll be OK unless there's newer features we simply
must have :)

>> I think every package should explicitly choose to support gnutls.el, it
>> shouldn't be an Emacs-wide choice.  There's too many configuration
>> options that depend on the purpose.  For instance IMAP and HTTPS have
>> really different security needs.

LMI> True.  On the other hand, look at `nnimap-open-connection' and weep.  It
LMI> started off as a simple function and grew into a monster.  That's why I
LMI> haven't adapted nntp/pop3/etc to use gnutls yet -- I don't want to have
LMI> to repeat the same code all over the place.

...thanks for working on proto-stream.el, it's made all this much simpler.

On Fri, 26 Nov 2010 13:51:09 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> My opinion is go with 2.10.

On Sat, 27 Nov 2010 09:28:11 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> So gnutls 2.10 will not be in Debian stable when Emacs 24 get released
>> (unless Emacs 24 is released after mid-2012). But most users won't try
>> to install Emacs 24 on it anyway.

SM> That's a good point.  Especially since gnutls is not indispensable for
SM> Emacs, so as long as we properly handle the case where the gnutls library
SM> to too old (either by not using it, or by using some fallback code),
SM> then I'm OK with it.

OK, I will put upgrading Emacs' support of GnuTLS to 2.10 on my TODO
list.  Which means it won't happen soon, so if anyone else is feeling
frisky, go ahead and do it.  I listed what's necessary in my original
e-mail here; basically it's callbacks but there's other features in the
2.10 API we should use if possible.

I think that supporting 2.8 is counterproductive so I'll happily target
only 2.10.  I think that's reasonable and doesn't put a big burden on
the users and distros.

Ted

Re: gnutls status

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

On Tue, 14 Dec 2010 16:59:34 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Fri, 26 Nov 2010 15:10:39 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
LMI> Is 2.10.x at least backwards-compatible, so that if we do implement the
LMI> complicated 2.8.x features, it'll continue to work in the future, too?

TZ> Yes.  They try really hard to keep backwards compatibility.  I'd guess
TZ> for all 2.x releases we'll be OK unless there's newer features we simply
TZ> must have :)

Argh, GnuTLS 2.8.x is still standard on Ubuntu 10.10, so practically we
should support it.  Below is my first (untested) patch to generate the
HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE and
HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY definitions in the configure.in
using AC_CHECK_FUNCS and then use them (currently just #ifdef
placeholders) in gnutls.c.  I plan to retrieve them from the :callbacks
alist parameter to `gnutls-boot'.

Regenerating "configure" failed for me.  I get this error at the end:

./configure: line 12620: gl_ASSERT_NO_GNULIB_POSIXCHECK: command not found
./configure: line 12621: gl_ASSERT_NO_GNULIB_TESTS: command not found
./configure: line 12622: gl_INIT: command not found
checking for lstat... yes
./configure: line 12648: syntax error near unexpected token `lstat'
./configure: line 12648: `gl_SYS_STAT_MODULE_INDICATOR(lstat)'

at the end.  But it gets far enough that I can tell the tests are being
run.  This is why the patch is untested; I'll see if I can figure out
why that's happening.  It may be an Ubuntu oddity.

Please let me know if the proposed approach is reasonable and if you
have any comments.  In theory this should be pretty trivial.

Thanks
Ted



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: callbacks.patch --]
[-- Type: text/x-diff, Size: 2684 bytes --]

=== modified file 'configure.in'
--- configure.in	2011-02-24 04:28:17 +0000
+++ configure.in	2011-03-01 21:39:23 +0000
@@ -1972,12 +1972,26 @@
 AC_SUBST(LIBSELINUX_LIBS)
 
 HAVE_GNUTLS=no
+HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY=no
+HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE=no
 if test "${with_gnutls}" = "yes" ; then
   PKG_CHECK_MODULES([LIBGNUTLS], [gnutls >= 2.2.4], HAVE_GNUTLS=yes, HAVE_GNUTLS=no)
   if test "${HAVE_GNUTLS}" = "yes"; then
     AC_DEFINE(HAVE_GNUTLS, 1, [Define if using GnuTLS.])
   fi
+
+  AC_CHECK_FUNCS(gnutls_certificate_set_verify_function, HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY=yes)
+  AC_CHECK_FUNCS(gnutls_certificate_client_set_retrieve_function, HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE=yes)
+
+  if test "${HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE}" = "yes"; then
+    AC_DEFINE(HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE, 1, [Define if using GnuTLS certificate retrieval callbacks.])
+  fi
+
+  if test "${HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY}" = "yes"; then
+    AC_DEFINE(HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY, 1, [Define if using GnuTLS certificate verification callbacks.])
+  fi
 fi
+
 AC_SUBST(LIBGNUTLS_LIBS)
 AC_SUBST(LIBGNUTLS_CFLAGS)
 
@@ -3667,6 +3681,8 @@
 echo "  Does Emacs use -lgconf?                                 ${HAVE_GCONF}"
 echo "  Does Emacs use -lselinux?                               ${HAVE_LIBSELINUX}"
 echo "  Does Emacs use -lgnutls?                                ${HAVE_GNUTLS}"
+echo "  Does Emacs use -lgnutls certificate verify callbacks?   ${HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY}"
+echo "  Does Emacs use -lgnutls certificate retrieve callbacks? ${HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE}"
 echo "  Does Emacs use -lxml2?                                  ${HAVE_LIBXML2}"
 
 echo "  Does Emacs use -lfreetype?                              ${HAVE_FREETYPE}"

=== modified file 'src/gnutls.c'
--- src/gnutls.c	2011-01-25 04:08:28 +0000
+++ src/gnutls.c	2011-03-01 21:41:36 +0000
@@ -484,6 +484,16 @@
 
   GNUTLS_INITSTAGE (proc) = GNUTLS_STAGE_FILES;
 
+  GNUTLS_LOG (1, max_log_level, "gnutls callbacks");
+
+  GNUTLS_INITSTAGE (proc) = GNUTLS_STAGE_CALLBACKS;
+
+#ifdef HAVE_GNUTLS_CALLBACK_CERTIFICATE_VERIFY
+#endif
+
+#ifdef HAVE_GNUTLS_CALLBACK_CERTIFICATE_RETRIEVE
+#endif
+
   GNUTLS_LOG (1, max_log_level, "gnutls_init");
 
   ret = gnutls_init (&state, GNUTLS_CLIENT);

=== modified file 'src/gnutls.h'
--- src/gnutls.h	2011-01-25 04:08:28 +0000
+++ src/gnutls.h	2011-03-01 21:32:17 +0000
@@ -28,6 +28,7 @@
   GNUTLS_STAGE_EMPTY = 0,
   GNUTLS_STAGE_CRED_ALLOC,
   GNUTLS_STAGE_FILES,
+  GNUTLS_STAGE_CALLBACKS,
   GNUTLS_STAGE_INIT,
   GNUTLS_STAGE_PRIORITY,
   GNUTLS_STAGE_CRED_SET,

Re: gnutls status

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Regenerating "configure" failed for me.  I get this error at the end:
>
> ./configure: line 12620: gl_ASSERT_NO_GNULIB_POSIXCHECK: command not found
> ./configure: line 12621: gl_ASSERT_NO_GNULIB_TESTS: command not found
> ./configure: line 12622: gl_INIT: command not found
> checking for lstat... yes
> ./configure: line 12648: syntax error near unexpected token `lstat'
> ./configure: line 12648: `gl_SYS_STAT_MODULE_INDICATOR(lstat)'

I tried applying your patch to configure.in, and it worked for me.

Here's the output from around the lstat check:

checking for working GNU getopt function... yes
checking whether getenv is declared... yes
checking for lstat... yes
checking for alarm... yes


-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnutls status

[Ted Zlatanov]

On Sat, 05 Mar 2011 12:01:39 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Regenerating "configure" failed for me.  I get this error at the end:
>> 
>> ./configure: line 12620: gl_ASSERT_NO_GNULIB_POSIXCHECK: command not found
>> ./configure: line 12621: gl_ASSERT_NO_GNULIB_TESTS: command not found
>> ./configure: line 12622: gl_INIT: command not found
>> checking for lstat... yes
>> ./configure: line 12648: syntax error near unexpected token `lstat'
>> ./configure: line 12648: `gl_SYS_STAT_MODULE_INDICATOR(lstat)'

LMI> I tried applying your patch to configure.in, and it worked for me.

LMI> Here's the output from around the lstat check:

LMI> checking for working GNU getopt function... yes
LMI> checking whether getenv is declared... yes
LMI> checking for lstat... yes
LMI> checking for alarm... yes

Something is broken in my autoconf setup at home, because the exact same
configure.in worked on another machine (same Ubuntu 10.10 setup!)
yesterday.  Weird.

Anyhow, since GnuTLS 2.10 is not available even in Ubuntu 10.10, I'm
working on a way to control the verification with the GnuTLS
verification flags.  It's not as good as a verification callback but it
will be more useful for the vast majority of people right now.  I hope
to have something ready this weekend.

Ted