bug#10904: 24.0.93; Infinite loop in GnuTLS code during Gnus nnimap-initiated SSL handshake

[Thomas Fitzsimmons <fitzsim@fitzsim.org>]

[-- Attachment #1: Type: text/plain, Size: 1634 bytes --]

I rechecked this against bzr Emacs and GnuTLS 3.0.17 and it's still
there.  I tried to create a smaller test case and came up with this:

$ gnutls-serv --http &
$ emacs -Q

(progn
  (setq gnutls-log-level 5
        message-log-max t)
  (open-protocol-stream
        "*nnimap*" (current-buffer) "localhost"
        5556
        :type 'ssl
        :return-list t
        :shell-command "ssh %s imapd"
        :capability-command "1 CAPABILITY\r\n"
        :end-of-command "\r\n"
        :success " OK "
        :starttls-function
        (lambda (capabilities)
          (when (gnus-string-match-p "STARTTLS" capabilities)
            "1 STARTTLS\r\n"))))

The open-protocol-stream call is how nnimap-open-connection-1 in
lisp/gnus/nnimap.el creates the IMAP network process.

The loop happens when the GnuTLS handshake fails for some reason, within
a network process.  I use the attached patch to limit the number of
iterations.  I'm not familiar enough with the Emacs process code to
suggest a fix though.

It would be nice if one of you could try against gnutls-serv and confirm
you see this -- I think the problem is general enough that this proves
it and will allow a fix that I can test against my IMAP server.

However, if need-be I can try to arrange a tunnel to the IMAP server I'm
trying to connect to.

As for why this hasn't been seen before, it doesn't affect the GnuTLS
backend standalone (as Ted pointed out) -- it seems to only happen when
an SSL-using process is created and the GnuTLS handshake fails, so it
will be rare that it happens.  Still, an infinite loop is not a nice
failure mode, even if it's rare.

Thomas

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: emacs-gnutls-limit-loop.patch --]
[-- Type: text/x-patch, Size: 651 bytes --]

--- src/gnutls.c~	2012-02-13 15:46:01.000000000 -0500
+++ src/gnutls.c	2012-02-23 14:21:22.000000000 -0500
@@ -378,6 +378,8 @@
   return (bytes_written);
 }
 
+static int error_count = 0;
+
 EMACS_INT
 emacs_gnutls_read (struct Lisp_Process *proc, char *buf, EMACS_INT nbyte)
 {
@@ -386,8 +388,17 @@
 
   if (proc->gnutls_initstage != GNUTLS_STAGE_READY)
     {
+      if (error_count < 100)
+        {
+      error_count++;
       emacs_gnutls_handshake (proc);
       return -1;
+        }
+      else
+        {
+      error_count = 0;
+        return 0;
+        }
     }
   rtnval = fn_gnutls_record_recv (state, buf, nbyte);
   if (rtnval >= 0)