* bug#70645: Reliable HTTPS networking @ 2024-04-29 10:51 Christopher Baines 2024-04-29 11:57 ` bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri Christopher Baines 2024-04-29 12:05 ` bug#70645: Reliable HTTPS networking Christopher Baines 0 siblings, 2 replies; 4+ messages in thread From: Christopher Baines @ 2024-04-29 10:51 UTC (permalink / raw) To: 70645 [-- Attachment #1: Type: text/plain, Size: 1012 bytes --] For years now I've been trying to work out how to do reliable HTTPS networking with Guile, where reliable just means that it can't hang indefinitely. After a few wrong turns, I believe the way to do this is use non-blocking ports as that combined with suspendable ports in Guile allows you to provide current-read-waiter/current-write-waiter procedures that will timeout at some point. I think the final hurdle is to get tls-wrap in (web client) to support Asynchronous operation with GnuTLS [1] and I think there are only a couple of things missing. make-session needs passing connection-flag/nonblock and error/again plus error/interrupted exceptions need handling for the handshake using the information from record-get-direction about whether Guile should wait to write or read. 1: https://gnutls.org/manual/html_node/Asynchronous-operation.html I think I forgot to move things forward after guile-gnutls 4.0.0 released with record-get-direction, so I'm opening this bug to try and keep track of things. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri. 2024-04-29 10:51 bug#70645: Reliable HTTPS networking Christopher Baines @ 2024-04-29 11:57 ` Christopher Baines 2024-04-29 11:57 ` bug#70645: [PATCH 2/2] web: Handle non-blocking ports in tls-wrap Christopher Baines 2024-04-29 12:05 ` bug#70645: Reliable HTTPS networking Christopher Baines 1 sibling, 1 reply; 4+ messages in thread From: Christopher Baines @ 2024-04-29 11:57 UTC (permalink / raw) To: 70645 Since this allows specifying additional behaviours for the socket through using SOCK_CLOEXEC and/or SOCK_NONBLOCK (when bitwise or'ed with SOCK_STREAM). Note that Guile/guile-gnutls currently doesn't support performing the TLS handshake on a non-blocking socket, so this currently won't work. * module/web/client.scm (open-socket-for-uri): Allow specifying the socket style. --- module/web/client.scm | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/module/web/client.scm b/module/web/client.scm index 6c54c5021..f26b5d259 100644 --- a/module/web/client.scm +++ b/module/web/client.scm @@ -317,9 +317,12 @@ host name without trailing dot." (read-response port)) (define* (open-socket-for-uri uri-or-string - #:key (verify-certificate? #t)) + #:key (verify-certificate? #t) + (socket-style SOCK_STREAM)) "Return an open input/output port for a connection to URI-OR-STRING. -When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates." +When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates. +SOCKET-STYLE defaults to SOCK_STREAM, and can be bitwise or'ed with +options like SOCK_CLOEXEC or SOCK_NONBLOCK." (define uri (ensure-uri-reference uri-or-string)) (define https? @@ -346,7 +349,9 @@ When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates." (let* ((ai (car addresses)) (s (with-fluids ((%default-port-encoding #f)) ;; Restrict ourselves to TCP. - (socket (addrinfo:fam ai) SOCK_STREAM IPPROTO_IP)))) + (socket (addrinfo:fam ai) + socket-style + IPPROTO_IP)))) (catch 'system-error (lambda () (connect s (addrinfo:addr ai)) -- 2.41.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* bug#70645: [PATCH 2/2] web: Handle non-blocking ports in tls-wrap. 2024-04-29 11:57 ` bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri Christopher Baines @ 2024-04-29 11:57 ` Christopher Baines 0 siblings, 0 replies; 4+ messages in thread From: Christopher Baines @ 2024-04-29 11:57 UTC (permalink / raw) To: 70645 As described in the GnuTLS documentation on Asynchronous operation, GNUTLS_NONBLOCK should be passed to gnutls_init, and the Guile equivalent is passing connection-flag/nonblock to make-session. Additionally, error/again or error/interrupted should lead to a retry of the handshake, after waiting for the appropriate I/O on the port. As record-get-direction is new in Guile-GnuTLS, specifically check if this is defined. * module/web/client.scm (tls-wrap): Call make-session with connection-flag/nonblock if the port is non-blocking, and handle waiting for I/O when performing the handshake. --- module/web/client.scm | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/module/web/client.scm b/module/web/client.scm index f26b5d259..caf8e5f35 100644 --- a/module/web/client.scm +++ b/module/web/client.scm @@ -33,6 +33,7 @@ (define-module (web client) #:use-module (rnrs bytevectors) + #:use-module (ice-9 suspendable-ports) #:use-module (ice-9 binary-ports) #:use-module (ice-9 copy-tree) #:use-module (ice-9 iconv) @@ -225,7 +226,14 @@ host name without trailing dot." (load-gnutls) - (let ((session (make-session connection-end/client)) + (let ((session + (apply + make-session + (cons connection-end/client + (if (zero? (logand O_NONBLOCK (fcntl port F_GETFL))) + '() + ;; If the port is non-blocking, tell GnuTLS + (list connection-flag/nonblock))))) (ca-certs (x509-certificate-directory))) ;; Some servers such as 'cloud.github.com' require the client to support ;; the 'SERVER NAME' extension. However, 'set-session-server-name!' is @@ -261,7 +269,19 @@ host name without trailing dot." (lambda () (handshake session)) (lambda (key err proc . rest) - (cond ((eq? err error/warning-alert-received) + (cond ((and + (or (eq? err error/again) + (eq? err error/interrupted)) + (module-defined? (resolve-interface '(gnutls)) + 'record-get-direction)) ; Guile-GnuTLS >= 4.0.0 + (if (= 0 (record-get-direction session)) + ((current-read-waiter) port) + ((current-write-waiter) port)) + + ;; These errors are expected and just signal that + ;; GnuTLS was interrupted, so don't count the retry + (loop retries)) + ((eq? err error/warning-alert-received) ;; Like Wget, do no stop upon non-fatal alerts such as ;; 'alert-description/unrecognized-name'. (format (current-error-port) -- 2.41.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* bug#70645: Reliable HTTPS networking 2024-04-29 10:51 bug#70645: Reliable HTTPS networking Christopher Baines 2024-04-29 11:57 ` bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri Christopher Baines @ 2024-04-29 12:05 ` Christopher Baines 1 sibling, 0 replies; 4+ messages in thread From: Christopher Baines @ 2024-04-29 12:05 UTC (permalink / raw) To: 70645 [-- Attachment #1: Type: text/plain, Size: 1707 bytes --] Christopher Baines <mail@cbaines.net> writes: > For years now I've been trying to work out how to do reliable HTTPS > networking with Guile, where reliable just means that it can't hang > indefinitely. > > After a few wrong turns, I believe the way to do this is use > non-blocking ports as that combined with suspendable ports in Guile > allows you to provide current-read-waiter/current-write-waiter > procedures that will timeout at some point. > > I think the final hurdle is to get tls-wrap in (web client) to support > Asynchronous operation with GnuTLS [1] and I think there are only a > couple of things missing. make-session needs passing > connection-flag/nonblock and error/again plus error/interrupted > exceptions need handling for the handshake using the information from > record-get-direction about whether Guile should wait to write or read. > > 1: https://gnutls.org/manual/html_node/Asynchronous-operation.html > > I think I forgot to move things forward after guile-gnutls 4.0.0 > released with record-get-direction, so I'm opening this bug to try and > keep track of things. I've now sent a couple of patches. The first is a re-send of [2], but with some docstring improvements. I can't find any reference in the Guile docs at least to the bitwise or'ing of options with the socket style, so while it seems to work, I'm a bit unsure about that. 2: https://lists.gnu.org/archive/html/guile-devel/2023-07/msg00025.html The second patch makes the changes inside of tls-wrap. There's also this patch [3] here to make get-bytevector-all non-blocking, and that's relevant here as it's used in read-response-body. 3: https://lists.gnu.org/archive/html/guile-devel/2023-07/msg00023.html [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 987 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-04-29 12:05 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-04-29 10:51 bug#70645: Reliable HTTPS networking Christopher Baines 2024-04-29 11:57 ` bug#70645: [PATCH 1/2] Allow specifying the socket style for open-socket-for-uri Christopher Baines 2024-04-29 11:57 ` bug#70645: [PATCH 2/2] web: Handle non-blocking ports in tls-wrap Christopher Baines 2024-04-29 12:05 ` bug#70645: Reliable HTTPS networking Christopher Baines
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).