From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id ILgXFQTyAmAOMQAA0tVLHw (envelope-from ) for ; Sat, 16 Jan 2021 14:02:44 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id KE/mEATyAmAeQAAA1q6Kng (envelope-from ) for ; Sat, 16 Jan 2021 14:02:44 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DF8D6940503 for ; Sat, 16 Jan 2021 14:02:43 +0000 (UTC) Received: from localhost ([::1]:37334 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l0mA6-00016F-SP for larch@yhetil.org; Sat, 16 Jan 2021 09:02:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0007BG-LQ for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:59296) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l0m6Y-0004ur-6W for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l0m6Y-0005wH-4s for guix-patches@gnu.org; Sat, 16 Jan 2021 08:59:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45409] [PATCH v4 05/13] http-client: Add error handling to http-multiple-get. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 16 Jan 2021 13:59:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45409 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45409@debbugs.gnu.org Received: via spool by 45409-submit@debbugs.gnu.org id=B45409.161080549122665 (code B ref 45409); Sat, 16 Jan 2021 13:59:02 +0000 Received: (at 45409) by debbugs.gnu.org; 16 Jan 2021 13:58:11 +0000 Received: from localhost ([127.0.0.1]:42581 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5i-0005sz-6Y for submit@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:11 -0500 Received: from mira.cbaines.net ([212.71.252.8]:49420) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l0m5g-0005sO-1F for 45409@debbugs.gnu.org; Sat, 16 Jan 2021 08:58:09 -0500 Received: from localhost (188.29.101.63.threembb.co.uk [188.29.101.63]) by mira.cbaines.net (Postfix) with ESMTPSA id 3CD5927BC17 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:07 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 78ff9e06 for <45409@debbugs.gnu.org>; Sat, 16 Jan 2021 13:58:03 +0000 (UTC) From: Christopher Baines Date: Sat, 16 Jan 2021 13:57:55 +0000 Message-Id: <20210116135803.21955-5-mail@cbaines.net> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20210116135803.21955-1-mail@cbaines.net> References: <20210116135803.21955-1-mail@cbaines.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 2.65 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: DF8D6940503 X-Spam-Score: 2.65 X-Migadu-Scanner: scn0.migadu.com X-TUID: aZR7w5orCrRe Making sure to close the port if it looks to be unusable. This closing of the port will allow for caching connections, without caching broken connections, as the cache can avoid handing out closed ports. * guix/http-client.scm (http-multiple-get): Try to catch exceptions that happen if the port is unusable, this is a adaptation of code within the (guix scripts substitute) module. --- guix/http-client.scm | 74 +++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 18 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 7ead493633..3aba3b28c1 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -38,6 +38,7 @@ #:use-module (guix utils) #:use-module (guix base64) #:autoload (gcrypt hash) (sha256) + #:autoload (gnutls) (error/invalid-session) #:use-module ((guix build utils) #:select (mkdir-p dump-port)) #:use-module ((guix build download) @@ -180,10 +181,25 @@ returning." ;; Inherit the HTTP proxying property from P. (set-http-proxy-port?! buffer (http-proxy-port? p)) - (for-each (cut write-request <> buffer) - batch) - (put-bytevector p (get)) - (force-output p)) + (catch #t + (lambda () + (for-each (cut write-request <> buffer) + batch) + (put-bytevector p (get)) + (force-output p)) + (lambda (key . args) + ;; If PORT becomes unusable, open a fresh connection and + ;; retry. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session))) + (begin + (close-port p) ; close the broken port + (connect #f + requests + result)) + (apply throw key args))))) ;; Now start processing responses. (let loop ((sent batch) @@ -199,20 +215,42 @@ returning." (remainder (connect p remainder result)))) ((head tail ...) - (let* ((resp (read-response p)) - (body (response-body-port resp)) - (result (proc head resp body result))) - ;; The server can choose to stop responding at any time, in which - ;; case we have to try again. Check whether that is the case. - ;; Note that even upon "Connection: close", we can read from BODY. - (match (assq 'connection (response-headers resp)) - (('connection 'close) - (close-port p) - (connect #f ;try again - (drop requests (+ 1 processed)) - result)) - (_ - (loop tail (+ 1 processed) result)))))))))) ;keep going + (catch #t + (lambda () + (let* ((resp (read-response p)) + (body (response-body-port resp)) + (result (proc head resp body result))) + ;; The server can choose to stop responding at any time, + ;; in which case we have to try again. Check whether + ;; that is the case. Note that even upon "Connection: + ;; close", we can read from BODY. + (match (assq 'connection (response-headers resp)) + (('connection 'close) + (close-port p) + (connect #f ;try again + (drop requests (+ 1 processed)) + result)) + (_ + (loop tail (+ 1 processed) result))))) ;keep going + (lambda (key . args) + ;; If PORT was cached and the server closed the connection + ;; in the meantime, we get EPIPE. In that case, open a + ;; fresh connection and retry. We might also get + ;; 'bad-response or a similar exception from (web response) + ;; later on, once we've sent the request, or a + ;; ERROR/INVALID-SESSION from GnuTLS. + (if (or (and (eq? key 'system-error) + (= EPIPE (system-error-errno `(,key ,@args)))) + (and (eq? key 'gnutls-error) + (eq? (first args) error/invalid-session)) + (memq key + '(bad-response bad-header bad-header-component))) + (begin + (close-port p) + (connect #f ; try again + (drop requests (+ 1 processed)) + result)) + (apply throw key args)))))))))) ;;; -- 2.30.0