bug#70645: Reliable HTTPS networking

bug#70645: Reliable HTTPS networking

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

For years now I've been trying to work out how to do reliable HTTPS
networking with Guile, where reliable just means that it can't hang
indefinitely.

After a few wrong turns, I believe the way to do this is use
non-blocking ports as that combined with suspendable ports in Guile
allows you to provide current-read-waiter/current-write-waiter
procedures that will timeout at some point.

I think the final hurdle is to get tls-wrap in (web client) to support
Asynchronous operation with GnuTLS [1] and I think there are only a
couple of things missing. make-session needs passing
connection-flag/nonblock and error/again plus error/interrupted
exceptions need handling for the handshake using the information from
record-get-direction about whether Guile should wait to write or read.

1: https://gnutls.org/manual/html_node/Asynchronous-operation.html

I think I forgot to move things forward after guile-gnutls 4.0.0
released with record-get-direction, so I'm opening this bug to try and
keep track of things.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri.

[Christopher Baines]

Since this allows specifying additional behaviours for the socket
through using SOCK_CLOEXEC and/or SOCK_NONBLOCK (when bitwise or'ed with
SOCK_STREAM).

Note that Guile/guile-gnutls currently doesn't support performing the
TLS handshake on a non-blocking socket, so this currently won't work.

* module/web/client.scm (open-socket-for-uri): Allow specifying the
socket style.
---
 module/web/client.scm | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/module/web/client.scm b/module/web/client.scm
index 6c54c5021..f26b5d259 100644
--- a/module/web/client.scm
+++ b/module/web/client.scm
@@ -317,9 +317,12 @@ host name without trailing dot."
   (read-response port))
 
 (define* (open-socket-for-uri uri-or-string
-                              #:key (verify-certificate? #t))
+                              #:key (verify-certificate? #t)
+                              (socket-style SOCK_STREAM))
   "Return an open input/output port for a connection to URI-OR-STRING.
-When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates."
+When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates.
+SOCKET-STYLE defaults to SOCK_STREAM, and can be bitwise or'ed with
+options like SOCK_CLOEXEC or SOCK_NONBLOCK."
   (define uri
     (ensure-uri-reference uri-or-string))
   (define https?
@@ -346,7 +349,9 @@ When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates."
       (let* ((ai (car addresses))
              (s  (with-fluids ((%default-port-encoding #f))
                    ;; Restrict ourselves to TCP.
-                   (socket (addrinfo:fam ai) SOCK_STREAM IPPROTO_IP))))
+                   (socket (addrinfo:fam ai)
+                           socket-style
+                           IPPROTO_IP))))
         (catch 'system-error
           (lambda ()
             (connect s (addrinfo:addr ai))
-- 
2.41.0

bug#70645: [PATCH 2/2] web: Handle non-blocking ports in tls-wrap.

[Christopher Baines]

As described in the GnuTLS documentation on Asynchronous operation,
GNUTLS_NONBLOCK should be passed to gnutls_init, and the Guile
equivalent is passing connection-flag/nonblock to make-session.

Additionally, error/again or error/interrupted should lead to a retry of
the handshake, after waiting for the appropriate I/O on the port.  As
record-get-direction is new in Guile-GnuTLS, specifically check if this
is defined.

* module/web/client.scm (tls-wrap): Call make-session with
connection-flag/nonblock if the port is non-blocking, and handle waiting
for I/O when performing the handshake.
---
 module/web/client.scm | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/module/web/client.scm b/module/web/client.scm
index f26b5d259..caf8e5f35 100644
--- a/module/web/client.scm
+++ b/module/web/client.scm
@@ -33,6 +33,7 @@
 
 (define-module (web client)
   #:use-module (rnrs bytevectors)
+  #:use-module (ice-9 suspendable-ports)
   #:use-module (ice-9 binary-ports)
   #:use-module (ice-9 copy-tree)
   #:use-module (ice-9 iconv)
@@ -225,7 +226,14 @@ host name without trailing dot."
 
   (load-gnutls)
 
-  (let ((session  (make-session connection-end/client))
+  (let ((session
+         (apply
+          make-session
+          (cons connection-end/client
+                (if (zero? (logand O_NONBLOCK (fcntl port F_GETFL)))
+                    '()
+                    ;; If the port is non-blocking, tell GnuTLS
+                    (list connection-flag/nonblock)))))
         (ca-certs (x509-certificate-directory)))
     ;; Some servers such as 'cloud.github.com' require the client to support
     ;; the 'SERVER NAME' extension.  However, 'set-session-server-name!' is
@@ -261,7 +269,19 @@ host name without trailing dot."
         (lambda ()
           (handshake session))
         (lambda (key err proc . rest)
-          (cond ((eq? err error/warning-alert-received)
+          (cond ((and
+                  (or (eq? err error/again)
+                      (eq? err error/interrupted))
+                  (module-defined? (resolve-interface '(gnutls))
+                                   'record-get-direction)) ; Guile-GnuTLS >= 4.0.0
+                 (if (= 0 (record-get-direction session))
+                     ((current-read-waiter) port)
+                     ((current-write-waiter) port))
+
+                 ;; These errors are expected and just signal that
+                 ;; GnuTLS was interrupted, so don't count the retry
+                 (loop retries))
+                ((eq? err error/warning-alert-received)
                  ;; Like Wget, do no stop upon non-fatal alerts such as
                  ;; 'alert-description/unrecognized-name'.
                  (format (current-error-port)
-- 
2.41.0

bug#70645: Reliable HTTPS networking

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> For years now I've been trying to work out how to do reliable HTTPS
> networking with Guile, where reliable just means that it can't hang
> indefinitely.
>
> After a few wrong turns, I believe the way to do this is use
> non-blocking ports as that combined with suspendable ports in Guile
> allows you to provide current-read-waiter/current-write-waiter
> procedures that will timeout at some point.
>
> I think the final hurdle is to get tls-wrap in (web client) to support
> Asynchronous operation with GnuTLS [1] and I think there are only a
> couple of things missing. make-session needs passing
> connection-flag/nonblock and error/again plus error/interrupted
> exceptions need handling for the handshake using the information from
> record-get-direction about whether Guile should wait to write or read.
>
> 1: https://gnutls.org/manual/html_node/Asynchronous-operation.html
>
> I think I forgot to move things forward after guile-gnutls 4.0.0
> released with record-get-direction, so I'm opening this bug to try and
> keep track of things.

I've now sent a couple of patches.

The first is a re-send of [2], but with some docstring improvements. I
can't find any reference in the Guile docs at least to the bitwise
or'ing of options with the socket style, so while it seems to work, I'm
a bit unsure about that.

2: https://lists.gnu.org/archive/html/guile-devel/2023-07/msg00025.html

The second patch makes the changes inside of tls-wrap.

There's also this patch [3] here to make get-bytevector-all
non-blocking, and that's relevant here as it's used in
read-response-body.

3: https://lists.gnu.org/archive/html/guile-devel/2023-07/msg00023.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]