Request for advice on GNUS internals. GSSAPI progress report

Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

I've now spent a few days figuring out how to do GSSAPI authentication with
IMAP, and I have now managed to complete a full handshake from GNUS with a
Microsoft Exchange server using GSSAPI.

Immediately after authentication is complete, the connection switches to
GSSAPI mode where each packet needs to pass through a call to ‘gss-unwrap’,
and the data sent to the server also needs to be split into packets which
are wrapped using a call to ‘gss-wrap’. The situation is further
complicated by the fact that the server can limit the maximum packet size
during initial handshake.

I need some advice from someone who is well-versed in the internals of GNUS
to explain where I should add the code to handle this.

At first I was looking at create a new ‘nnimap-stream’ type to represent
this, but the connection is already inside a TLS connection which already
uses ‘ssl’ or ‘starttls’ here. The GSSAPI authentication is independent of
the actual connection type, but it wraps all the IMAP commands that are
transmitted over it.

What approach should I take here?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1255 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On 15 February 2017 at 12:37, Elias Mårtenson <lokedhs@gmail.com> wrote:

What approach should I take here?
>

I have come up with a solution that uses a process filter to deal with
this. It probably works, but I can't really test it since my server doesn't
use GSS encryption, only the authentication part.

Other than that, it seems to work now, and I'm working on cleaning up the
code.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 848 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 11735 bytes --]

I am now able to read my email using a GSSAPI-authenticated connection.

If anyone is willing to try it, build the native module here:
https://github.com/lokedhs/emacs-gssapi

Then apply the below patch to nnimap.el. To enable the use of GSSAPI for
authentication, set ‘nnimap-authenticator’ to ‘gssapi’.

Regards,
Elias

*** /home/emartenson/src/emacs/lisp/gnus/nnimap.el 2017-01-31
10:04:34.692388829 +0800
--- /usr/local/share/emacs/26.0.50/lisp/gnus/nnimap.el 2017-02-16
18:08:44.883875947 +0800
***************
*** 331,336 ****
--- 331,337 ----
      (set (make-local-variable 'nnimap-object)
  (make-nnimap :server (nnoo-current-server 'nnimap)
       :initial-resync 0))
+     (set (make-local-variable 'nnimap-send-string-function)
'nnimap--unencoded-send-string-to-server)
      (push (list buffer (current-buffer)) nnimap-connection-alist)
      (push (current-buffer) nnimap-process-buffers)
      (current-buffer)))
***************
*** 513,520 ****
--- 514,652 ----

  (autoload 'rfc2104-hash "rfc2104")

+ (defun nnimap--send-string-to-server (string)
+   (message "Sending string using %S: %S" nnimap-send-string-function
string)
+   (funcall nnimap-send-string-function string))
+
+ (defun nnimap--unencoded-send-string-to-server (string)
+   (process-send-string (get-buffer-process (current-buffer)) string))
+
+ (defun nnimap--gss-send-string-to-server (string)
+   (cond ((eq nnimap-gss-protection-type :none)
+          (nnimap--unencoded-send-string-to-server string))
+         (t
+          ;; TODO: We should check the maximum block size here.
+          (destructuring-bind (encoded conf)
+              (gss-wrap nnimap-gss-context string :conf (eq
nnimap-gss-protection-type :priv))
+            (let ((size (length encoded)))
+              (nnimap--unencoded-send-string-to-server (format "%c%c%c%c%s"
+                                                               (logand
(ash size -24) #xff)
+                                                               (logand
(ash size -16) #xff)
+                                                               (logand
(ash size -8) #xff)
+                                                               (logand
size #xff)
+
encoded)))))))
+
+ (defun nnimap--gss-read-filter (proc string)
+   (message "Read block using type %S: %S" nnimap-gss-protection-type
string)
+   (cond ((eq nnimap-gss-protection-type :none)
+          (internal-default-process-filter proc string))
+         (t
+          ;; TODO: This function is currently unimplemented. Although
+          ;; its implementation isn't particularly complicated, I've
+          ;; left it blank for now since I don't have access to a mail
+          ;; server which uses wrapped streams.
+          ;;
+          ;; Implementing this should be reasonably simple. ‘string’
+          ;; contains a sequence of bytes that has been received from
+          ;; the IMAP server. The first 4 bytes of the stream is the
+          ;; length of the packet in network byte-order. The bytes in
+          ;; the packet should be passed to ‘gss-unwrap’. The ‘:conf’
+          ;; parameter to this call should be nil if
+          ;; ‘nnimap-gss-protection-type’ is ‘:integ’ or t if it is
+          ;; ‘:priv’. The resulting string should then be passed to
+          ;; ‘internal-default-process-filter’ in a similar way as what
+          ;; is done in the non-encrypted case.
+          (error "Wrapped streams are not implemented. Please see comments
in the function ‘nnimap--gss-read-filter’"))))
+
+ (defun nnimap--strip-cr (string)
+   (let ((length (length string)))
+     (if (and (plusp length)
+              (eql (aref string (1- length)) ?\r))
+         (subseq string 0 (1- length))
+       string)))
+
+ (defun nnimap--decode-token (decrypted)
+   ;; The first octet is the bitmask
+   (let ((flags (aref decrypted 0))
+         (max-message-size (logior (ash (aref decrypted 1) 16)
+                                   (ash (aref decrypted 2) 8)
+                                   (aref decrypted 3))))
+     (list flags max-message-size)))
+
+ ;;; GSSAPI authentication documented at:
https://tools.ietf.org/html/rfc1731
+ (defun nnimap-auth-gssapi (user)
+   (require 'gss)
+   (let ((sequence (nnimap-send-command "AUTHENTICATE GSSAPI")))
+     (let ((resp (nnimap--strip-cr (nnimap-wait-for-line
"^\\+\\(.*\\)\n"))))
+       (assert (equal resp ""))
+       (let ((context nil))
+         (loop with authenticated-p = nil
+               with input-token = nil
+               with name = (gss-make-name (format "imap@%s" (downcase
nnimap-address)))
+               while (not authenticated-p)
+               do (erase-buffer)
+               do (multiple-value-bind (continue-needed context-result
token flags)
+                      (gss-init-sec-context name :context context
:input-token input-token :flags '(:replay :conf :integ :mutual))
+                    (setq context context-result)
+                    (if token
+                        (nnimap--send-string-to-server (concat
(base64-encode-string token) "\r\n"))
+                      ;; ELSE: The spec says that if there is no output
token, there should be a response with no data
+                      (nnimap--send-string-to-server "\r\n"))
+                    (if continue-needed
+                        (let ((server-message (nnimap-wait-for-line
"^\\+\\(.*\\)\n")))
+                          (setq input-token (base64-decode-string
server-message)))
+                      ;; ELSE: We now have a valid context
+                      (setq authenticated-p t))))
+         ;; After initial GSS handshake, the server is expected to send a
blank response.
+         (let ((token (nnimap-wait-for-line "^\\+\\(.*\\)\n")))
+           (destructuring-bind (decrypted conf-p)
+               (gss-unwrap context (base64-decode-string token))
+             (destructuring-bind (flags max-server-size)
+                 (nnimap--decode-token decrypted)
+               ;; Select the strictest protection mechanism that the
+               ;; server supports.
+               (let ((protection-type (cond ((not (zerop (logand flags
#x4)))
+                                             :priv)
+                                            ((not (zerop (logand flags
#x2)))
+                                             :integ)
+                                            ((not (zerop (logand flags
#x1)))
+                                             :none)
+                                            (t
+                                             (error "Unsupported privacy
type: %d" flags)))))
+                 ;; I don't know what the actual max message size should
+                 ;; be, so for now let's just use whatever the server is
+                 ;; using.
+                 (let* ((max-client-size max-server-size))
+                   (erase-buffer)
+                   (destructuring-bind (encoded conf)
+                       (gss-wrap context (format "%c%c%c%c%s"
+                                                 (ecase protection-type
+                                                   (:priv #x4)
+                                                   (:integ #x2)
+                                                   (:none #x1))
+                                                 (logand (ash
max-client-size -16) #xff)
+                                                 (logand (ash
max-client-size -8) #xff)
+                                                 (logand max-client-size
#xff)
+                                                 user))
+                     (nnimap--send-string-to-server (concat
(base64-encode-string encoded) "\r\n")))
+                   (nnimap-wait-for-response sequence "OK")
+                   (cond ((equal (caar (nnimap-get-response sequence))
"OK")
+                          (setq-local nnimap-gss-context context)
+                          (setq-local nnimap-gss-expected-size nil)
+                          (setq-local nnimap-gss-read-buf "")
+                          (setq-local nnimap-gss-protection-type
protection-type)
+                          (setq-local nnimap-send-string-function
#'nnimap--gss-send-string-to-server)
+                          (set-process-filter (get-buffer-process
(current-buffer)) #'nnimap--gss-read-filter)
+                          (cons user nil))
+                         (t
+                          (error "Authentication error"))))))))))))
+
  (defun nnimap-login (user password)
+   (set-process-filter (get-buffer-process (current-buffer)) (lambda (proc
string) (message "Input: %S" string) (insert string)))
    (cond
+    ((and (nnimap-capability "AUTH=GSSAPI")
+          (eq nnimap-authenticator 'gssapi))
+     (nnimap-auth-gssapi user))
     ;; Prefer plain LOGIN if it's enabled (since it requires fewer
     ;; round trips than CRAM-MD5, and it's less likely to be buggy),
     ;; and we're using an encrypted connection.
***************
*** 529,536 ****
      (erase-buffer)
      (let ((sequence (nnimap-send-command "AUTHENTICATE CRAM-MD5"))
   (challenge (nnimap-wait-for-line "^\\+\\(.*\\)\n")))
!       (process-send-string
!        (get-buffer-process (current-buffer))
         (concat
  (base64-encode-string
  (concat user " "
--- 661,667 ----
      (erase-buffer)
      (let ((sequence (nnimap-send-command "AUTHENTICATE CRAM-MD5"))
   (challenge (nnimap-wait-for-line "^\\+\\(.*\\)\n")))
!       (nnimap--send-string-to-server
         (concat
  (base64-encode-string
  (concat user " "
***************
*** 1192,1202 ****
   (length message)))
   (unless nnimap-streaming
     (nnimap-wait-for-connection "^[+]"))
!  (process-send-string (get-buffer-process (current-buffer)) message)
!  (process-send-string (get-buffer-process (current-buffer))
!       (if (nnimap-newlinep nnimap-object)
!   "\n"
! "\r\n"))
   (let ((result (nnimap-get-response sequence)))
     (if (not (nnimap-ok-p result))
  (progn
--- 1323,1332 ----
   (length message)))
   (unless nnimap-streaming
     (nnimap-wait-for-connection "^[+]"))
!  (nnimap--send-string-to-server message)
!  (nnimap--send-string-to-server (if (nnimap-newlinep nnimap-object)
!                                              "\n"
!                                            "\r\n"))
   (let ((result (nnimap-get-response sequence)))
     (if (not (nnimap-ok-p result))
  (progn
***************
*** 1864,1871 ****

  (defun nnimap-send-command (&rest args)
    (setf (nnimap-last-command-time nnimap-object) (current-time))
!   (process-send-string
!    (get-buffer-process (current-buffer))
     (nnimap-log-command
      (format "%d %s%s\n"
     (incf nnimap-sequence)
--- 1994,2000 ----

  (defun nnimap-send-command (&rest args)
    (setf (nnimap-last-command-time nnimap-object) (current-time))
!   (nnimap--send-string-to-server
     (nnimap-log-command
      (format "%d %s%s\n"
     (incf nnimap-sequence)


On 15 February 2017 at 18:13, Elias Mårtenson <lokedhs@gmail.com> wrote:

> On 15 February 2017 at 12:37, Elias Mårtenson <lokedhs@gmail.com> wrote:
>
> What approach should I take here?
>>
>
> I have come up with a solution that uses a process filter to deal with
> this. It probably works, but I can't really test it since my server doesn't
> use GSS encryption, only the authentication part.
>
> Other than that, it seems to work now, and I'm working on cleaning up the
> code.
>
> Regards,
> Elias
>

[-- Attachment #2: Type: text/html, Size: 18539 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Lars Ingebrigtsen]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I am now able to read my email using a GSSAPI-authenticated connection.
>
> If anyone is willing to try it, build the native module here:
> https://github.com/lokedhs/emacs-gssapi
>
> Then apply the below patch to nnimap.el. To enable the use of GSSAPI for
> authentication, set ‘nnimap-authenticator’ to ‘gssapi’.

I've skimmed the patch, and it looks reasonable to me, but I know
nothing about GSSAPI.  :-)  I wonder whether that's also used for other
networking protocols, like pop3 or SMTP, or is it IMAP only?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

On 21 February 2017 at 00:10, Lars Ingebrigtsen <larsi@gnus.org> wrote:

I've skimmed the patch, and it looks reasonable to me, but I know
> nothing about GSSAPI.  :-)  I wonder whether that's also used for other
> networking protocols, like pop3 or SMTP, or is it IMAP only?
>

I don't know about POP3, does anyone still use that? It's definitely
supported for SMTP, and now that you mention it, I have to implement that
support too. It would be a pretty useless feature if you can read your mail
without requiring a stored password, but still needing it to send them. :-)

Seems I'll have to dig into the Gnus SMTP implementation before this is
ready. :-)

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1099 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Tue, 21 Feb 2017 10:50:38 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> Seems I'll have to dig into the Gnus SMTP implementation before this is ready. :-)

Isn't that on smtpmail.el?

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

On 21 February 2017 at 11:42, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Elias Mårtenson <lokedhs@gmail.com>
> > Date: Tue, 21 Feb 2017 10:50:38 +0800
> > Cc: emacs-devel <emacs-devel@gnu.org>
> >
> > Seems I'll have to dig into the Gnus SMTP implementation before this is
> ready. :-)
>
> Isn't that on smtpmail.el?
>

Possibly. I haven't started working on this yet. Thanks for the pointer.

[-- Attachment #2: Type: text/html, Size: 871 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Michael Albinus]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I don't know about POP3, does anyone still use that?

Me. I need it on one machine @work, for dumb reasons.

> Regards,
> Elias

Best regards, Michael.

Re: Request for advice on GNUS internals. GSSAPI progress report

[Lars Ingebrigtsen]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I don't know about POP3, does anyone still use that? It's definitely
> supported for SMTP, and now that you mention it, I have to implement
> that support too. It would be a pretty useless feature if you can read
> your mail without requiring a stored password, but still needing it to
> send them. :-)

If this is relevant to a lot of the different protocols, perhaps it
would make more sense to put this into the Emacs core like the TLS
support?  Then each protocol wouldn't have to be modified this much to
support it across Emacs...

Today they pretty much all say

  (make-network-process ... :tls-parameters ...)

and that's all they have to know about whether the socket uses TLS or
not.  If you'd add a :gssapi-parameters thing to that function, that
might make sense.

But like I said, I know naaathing about GSSAPI, so I have no idea
whether this makes sense.  :-)  I just thought I'd mention it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 2520 bytes --]

On 28 February 2017 at 00:29, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Elias Mårtenson <lokedhs@gmail.com> writes:
>
> > I don't know about POP3, does anyone still use that? It's definitely
> > supported for SMTP, and now that you mention it, I have to implement
> > that support too. It would be a pretty useless feature if you can read
> > your mail without requiring a stored password, but still needing it to
> > send them. :-)
>
> If this is relevant to a lot of the different protocols, perhaps it
> would make more sense to put this into the Emacs core like the TLS
> support?  Then each protocol wouldn't have to be modified this much to
> support it across Emacs...


Unfortunately, that's not possible. Every protocol has a different idea how
to perform a GSSAPI handshake. GSSAPI itself only returns a binary blong
that is to be sent to the remote side, that side can then send another blob
back. After ping-ponging a few times, you get a validated name object
representing the remote principal, and context that can be used to encrypt
and decrypt other binary blobs. There docs literally says something along
the lines of: “Send the binary output to the remote server and pass it to
gss_accept_sec_context()”.

This results in plenty of different specs how to apply GSSAPI
authentication to various protocols. A few examples:

  - IMAP: https://tools.ietf.org/html/rfc1731
  - SMTP: https://tools.ietf.org/html/rfc4954
  - POP3: https://tools.ietf.org/html/rfc5034
  - LDAP: https://tools.ietf.org/html/rfc4752

Etc, etc.

Currently, when using GNUS, LDAP authentication works with GSSAPI thanks to
fact that Emacs leverages the ‘ldapsearch’ external program. IMAP4 works
now because I've implemented it. I don't use POP3, and I don't think there
is much demand for it.

That leaves SMTP, which really should have support in Gnus proper, but as
it turns out, we're not using authenticated SMTP at my workplace so I'll
have a hard time testing it. The same goes for encrypted IMAP (using GSS
encryption, rather than tunnelling over TLS). To support it, one would have
to implement a very simple function but I left that empty since I have no
way of testing it.

So, that's the situation as it stands. I've restarted the process with my
employer's legal team to make sure I can get the copyright assignments
done. It seems to actually be happening this time (which means that we'll
be able to get gnu-apl-mode into ELPA soon).

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 3379 bytes --]

Re: Request for advice on GNUS internals. GSSAPI progress report

[Live System User]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I am now able to read my email using a GSSAPI-authenticated connection.
>
> If anyone is willing to try it,

  I know that you are still testing but don't forget to remove your
  "message" statements in nnimap.el -- it prints out passwords.

  Not sure where it is but I believe Gnus has facilities to inhibit the
  display of passwords as well as only messaging when debugging is
  turned on (and at different loglevels).
  
  Also. can you consider handling the error when either module support
  is not enabled or the emacs-gssapi module is unavailable?

  Bonus points if the authenicator falls back to using the "gsasl" or
  "imtest" program, if configured:


(defcustom gssapi-program (list
                           (concat "gsasl %s %p "
                                   "--mechanism GSSAPI "
                                   "--authentication-id %l")
                           "imtest -m gssapi -u %l -p %p %s")


  Thanks.

Re: Request for advice on GNUS internals. GSSAPI progress report

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

On 16 Feb 2017 11:42 PM, "Live System User" <nyc4bos@aol.com> wrote:

Elias Mårtenson <lokedhs@gmail.com> writes:

> I am now able to read my email using a GSSAPI-authenticated connection.
>
> If anyone is willing to try it,

  I know that you are still testing but don't forget to remove your
  "message" statements in nnimap.el -- it prints out passwords.


You are right indeed that there's a lot of debug messages there. They will
be removed of course.


  Also. can you consider handling the error when either module support
  is not enabled or the emacs-gssapi module is unavailable?


Good point. Perhaps it would make sense to let the authenticator parameter
be a list, and have the function try them all in order?

Bonus points if the authenicator falls back to using the "gsasl" or
  "imtest" program, if configured:


I don't know. The reason I started all of this was because I could never
get gsasl to work properly, so even if I do it I wouldn't be able to test
it.

That said, patch does not change any existing behaviour so if gsasl works
for you right now, nothing will change.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2446 bytes --]