From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qODaKJFvXGAp+QAAgWs5BA (envelope-from ) for ; Thu, 25 Mar 2021 12:10:09 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 3/ULGZFvXGAbYgAA1q6Kng (envelope-from ) for ; Thu, 25 Mar 2021 11:10:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1B51E9CA8 for ; Thu, 25 Mar 2021 12:10:09 +0100 (CET) Received: from localhost ([::1]:60618 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lPNsO-0007dn-8D for larch@yhetil.org; Thu, 25 Mar 2021 07:10:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58238) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPNsI-0007dh-88 for guix-patches@gnu.org; Thu, 25 Mar 2021 07:10:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:54032) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lPNsH-00036r-WB for guix-patches@gnu.org; Thu, 25 Mar 2021 07:10:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lPNsH-0001gC-QK for guix-patches@gnu.org; Thu, 25 Mar 2021 07:10:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47288] [PATCH] guix: http-client: Tweak http-multiple-get error handling. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 25 Mar 2021 11:10:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47288 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 47288@debbugs.gnu.org Received: via spool by 47288-submit@debbugs.gnu.org id=B47288.16166705506376 (code B ref 47288); Thu, 25 Mar 2021 11:10:01 +0000 Received: (at 47288) by debbugs.gnu.org; 25 Mar 2021 11:09:10 +0000 Received: from localhost ([127.0.0.1]:37345 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lPNrR-0001el-WE for submit@debbugs.gnu.org; Thu, 25 Mar 2021 07:09:10 -0400 Received: from mira.cbaines.net ([212.71.252.8]:50984) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lPNrP-0001ed-Vh for 47288@debbugs.gnu.org; Thu, 25 Mar 2021 07:09:08 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id 2854127BC5C; Thu, 25 Mar 2021 11:09:07 +0000 (GMT) Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 445f76e4; Thu, 25 Mar 2021 11:09:06 +0000 (UTC) References: <20210321004338.31867-1-mail@cbaines.net> <20210321005600.12551-1-mail@cbaines.net> <87pmzoeh3p.fsf_-_@gnu.org> User-agent: mu4e 1.4.15; emacs 27.1 From: Christopher Baines In-reply-to: <87pmzoeh3p.fsf_-_@gnu.org> Message-ID: <87zgyra3sg.fsf@cbaines.net> Date: Thu, 25 Mar 2021 11:09:04 +0000 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616670609; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=4vRvVGkl1s9lJKsHRlr6ObaHJAMtixvHA6lVHFT5jF0=; b=YhUGKO9d1M1eivJ8+ZKN6Czca+7cwpJjoqdpLAmNzAwLtYf1mLQlkLaCKXEZjiXS3yP+SX hPZoGhn0sZlZFv6U9VxH2dm4lsTR/dYLXItT8eDlwy6tJAHW9ypniiP6ORXgnWMyBZADL2 UKc46GYtIesYYezWZI1PMPkdC0Tt9UK4OjweMkZkYt9iM36Wv5mug6NCDTU1hl41lYNy+e ONvCKP+goFQ9XVyEs7uEoeLadqhDuU5liJLHa9OO3CgM+8DObWM/qO7L0GidJvMT0ABQOH NgaRoDtlenwHdVwpm5o0zR2JrRfkpqR6kPiebL5EQ1cPsuwVYXuvv34GR9prOw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616670609; a=rsa-sha256; cv=none; b=I+Zvj7MSwPoGE5BcNprsrZIoYX40/dsoMZDfzGAfIg5A2aGi0vG13hysq+aKe1oo98OAVz 0dRu62ObtDAogqzxFwUTAyWUFRaJJzu92ly2MZ7pIiXLvxVGIJ1HfT3fi7Co+lJOHgMaU/ wuWuddVzRT8jBrhBAJDAY/r+0ZRReJJsxfW0v1ShH3aMFYM4fxfwEXL1P3MIr1PYLrSJtR ClSV4L7qCZpJ/FM638XcVo4K+0KUCT7TJZmJAFx/lIdLpjAIJdnKbpQKW9QVo+Q5tpXzKP AnPaeVYkHL3Qn75bFJRofs9gU+0TY1eKERF061WCUHyruh6v6GFGNnSMhOoInA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -4.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 1B51E9CA8 X-Spam-Score: -4.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: K2JaC+u/aJ7u --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > As discussed at , I=E2=80=99m still un= sure > these exceptions need to be caught within =E2=80=98http-multiple-get=E2= =80=99 and at > each iteration. > > Just minor comments: > > Christopher Baines skribis: > >> + (catch #t >> + (lambda () >> + (let* ((resp (read-response p)) >> + (body (response-body-port resp)) >> + (result (proc head resp body result))) >> + ;; The server can choose to stop responding at any= time, >> + ;; in which case we have to try again. Check whet= her >> + ;; that is the case. Note that even upon "Connect= ion: >> + ;; close", we can read from BODY. >> + (match (assq 'connection (response-headers resp)) >> + (('connection 'close) >> + (close-port p) >> + (list 'connect >> + #f >> (drop requests (+ 1 processed)) >> result)) >> - (apply throw key args)))))))))) >> + (_ >> + (list 'loop tail (+ 1 processed) result))))) >> + (lambda (key . args) >> + ;; If PORT was cached and the server closed the conn= ection >> + ;; in the meantime, we get EPIPE. In that case, ope= n a >> + ;; fresh connection and retry. We might also get >> + ;; 'bad-response or a similar exception from (web re= sponse) >> + ;; later on, once we've sent the request, or a >> + ;; ERROR/INVALID-SESSION from GnuTLS. >> + (if (or (and (eq? key 'system-error) >> + (=3D EPIPE (system-error-errno `(,key ,= @args)))) >> + (and (eq? key 'gnutls-error) >> + (eq? (first args) error/invalid-session= ))1 >> + (memq key >> + '(bad-response >> + bad-header >> + bad-header-component))) >> + (begin >> + (close-port p) >> + (list 'connect >> + #f >> + requests >> + result)) >> + (apply throw key args)))) >> + (('connect . args) >> + (apply connect args)) >> + (('loop . args) >> + (apply loop args))))))))) > > This is not new to this patch, but I think the whole exception handling > bit should be factorized and written in such a way that > =E2=80=98http-multiple-get=E2=80=99 still first on a horizontal screen (e= ven though mine > is actually vertical :-)). Otherwise one might easily overlook the core > logic of the function. I've sent a v3 now, which makes a few changes to the original patch, and includes a second patch for a potential refactoring. I tested three variants for performance, http-multiple-get with no error handling, the first v3 patch and the first and second v3 patches, and at least with the test I'm using, the performance seems similar. Assuming the performance is lower with error handling, the impact seems to be within the margin of error, at least for test I was using. --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=http-multiple-get-perf.scm (use-modules (ice-9 binary-ports) (srfi srfi-19) (web uri) (web request) (web response) (guix http-client)) (define (call-with-time-logging requests thunk) (let ((start (current-time time-utc))) (call-with-values thunk (lambda vals (let* ((end (current-time time-utc)) (elapsed (time-difference end start))) (display (format #f "~f seconds (~f microseconds per request)~%" (+ (time-second elapsed) (/ (time-nanosecond elapsed) 1e9)) (* (/ (+ (time-second elapsed) (/ (time-nanosecond elapsed) 1e9)) requests) 1e6))) (apply values vals)))))) (define requests (map (lambda _ (build-request (string->uri "http://localhost/does-not-exist") #:method 'GET)) (iota 200000))) (call-with-time-logging (length requests) (lambda () (http-multiple-get (string->uri "http://localhost/") (lambda (request response port result) (get-bytevector-n port (response-content-length response)) '()) '() requests))) --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBcb1BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcMYg//ZHjTZpEIBy3zvkUytf7DqxlMj6/e/fkR G5B8VLkRysYDJUckziRqN/bBykRRrjYR7mCt4qX1XHROZdUAkxZ++K+xkrtUxSPq k9ezSw+akd4+geCpfyw0/R8aiqHPMUVlrvuSheUEsb2jHRD/oikkoS9N7xRrRsQN 3O4+alydMmKd24SKBjAHY375JE+Q0ad69Rry8mXw8rcXPuX4wGGsMfwamI2n38pZ Lghc9sgH82RXP4aZmdGyS2jWt6m/Qmydu0Par159cG9Ru0CCbio0HoMZd4Vpnvy7 VcMzp8UM2pAffD3CfzpOgKgIKKkb4wWJ25EgPD2+sDbCNBInz3XbUX3qCKHhngmi 6SDFzBFe59G5D/VnLGcOACUBg6a2IhFBjKuGcvWSELUDRCuQp2Z91KcdDb5kxPl1 Aznk2IwetBRUjXJUs8MhKsoit7E7KYFihN9zXsvXRlbnRcsyGoatTV48BXZHQOpv pXFngQYroYnp4/UIFgTdx4Dm6Oxs+VWdSZgM6/bSUmVHQmTxw++83Ia+L8zRTgp6 TCtycdC7tcJxlA7fQDfp99VPki5p3QGDZkShDIm3TqHwvlf1VG504HnNTRcr7yY+ 2feCfA3kcA5EJNYl5iDlhHT4dWM4LxGVUUCO51vWxVLKbsxCeRlTfl+GM8ePRzzS 1ErFEpvTEKI= =Wa1I -----END PGP SIGNATURE----- --==-=-=--