From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 5AdFAeXYJ2AtXgAA0tVLHw (envelope-from ) for ; Sat, 13 Feb 2021 13:49:25 +0000 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id uJb7N+TYJ2DUHQAAbx9fmQ (envelope-from ) for ; Sat, 13 Feb 2021 13:49:24 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 756D228144 for ; Sat, 13 Feb 2021 14:49:24 +0100 (CET) Received: from localhost ([::1]:46910 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAvIZ-0003eI-If for larch@yhetil.org; Sat, 13 Feb 2021 08:49:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55166) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAvHH-0002hb-GI for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:50511) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lAvHH-00076d-8D for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lAvHH-0002wx-4B for guix-patches@gnu.org; Sat, 13 Feb 2021 08:48:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v5 05/14] http-client: Add error handling to http-multiple-get. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 13 Feb 2021 13:48:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161322404711194 (code B ref 45409); Sat, 13 Feb 2021 13:48:03 +0000 Received: (at 45409) by debbugs.gnu.org; 13 Feb 2021 13:47:27 +0000 Received: from localhost ([127.0.0.1]:33795 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGh-0002uN-8f for submit@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:27 -0500 Received: from mira.cbaines.net ([212.71.252.8]:48194) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lAvGb-0002so-AC for 45409@debbugs.gnu.org; Sat, 13 Feb 2021 08:47:22 -0500 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id A34D927BC47 for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:20 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 81cf0d6a for <45409@debbugs.gnu.org>; Sat, 13 Feb 2021 13:47:19 +0000 (UTC) From: Christopher Baines Date: Sat, 13 Feb 2021 13:47:10 +0000 Message-Id: <20210213134719.19625-5-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210213134719.19625-1-mail@cbaines.net> References: <20210213134719.19625-1-mail@cbaines.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 2.64 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 756D228144 X-Spam-Score: 2.64 X-Migadu-Scanner: scn0.migadu.com X-TUID: DZ+sM2yw4rwt Making sure to close the port if it looks to be unusable. This closing of the port will allow for caching connections, without caching broken connections, as the cache can avoid handing out closed ports. * guix/http-client.scm (http-multiple-get): Try to catch exceptions that happen if the port is unusable, this is a adaptation of code within the (guix scripts substitute) module. --- guix/http-client.scm | 74 +++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 7ead493633..3aba3b28c1 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -38,6 +38,7 @@ #:use-module (guix utils) #:use-module (guix base64) #:autoload (gcrypt hash) (sha256) + #:autoload (gnutls) (error/invalid-session) #:use-module ((guix build utils) #:select (mkdir-p dump-port)) #:use-module ((guix build download) @@ -180,10 +181,25 @@ returning." ;; Inherit the HTTP proxying property from P. (set-http-proxy-port?! buffer (http-proxy-port? p)) - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) + (catch #t + (lambda () + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + (lambda (key . args) + ;; If PORT becomes unusable, open a fresh connection and + ;; retry. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session))) + (begin + (close-port p) ; close the broken port + (connect #f + requests + result)) + (apply throw key args))))) ;; Now start processing responses. (let loop ((sent batch) @@ -199,20 +215,42 @@ returning." (remainder (connect p remainder result)))) ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going + (catch #t + (lambda () + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, + ;; in which case we have to try again. Check whether + ;; that is the case. Note that even upon "Connection: + ;; close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result))))) ;keep going + (lambda (key . args) + ;; If PORT was cached and the server closed the connection + ;; in the meantime, we get EPIPE. In that case, open a + ;; fresh connection and retry. We might also get + ;; 'bad-response or a similar exception from (web response) + ;; later on, once we've sent the request, or a + ;; ERROR/INVALID-SESSION from GnuTLS. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session)) + (memq key + '(bad-response bad-header bad-header-component))) + (begin + (close-port p) + (connect #f ; try again + (drop requests (+ 1 processed)) + result)) + (apply throw key args)))))))))) ;;; -- 2.30.0