From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id WJChFrAVcWIqkAAAbAwnHQ (envelope-from ) for ; Tue, 03 May 2022 13:44:48 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 0CObFrAVcWJopAAAauVa8A (envelope-from ) for ; Tue, 03 May 2022 13:44:48 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E39521F199 for ; Tue, 3 May 2022 13:44:47 +0200 (CEST) Received: from localhost ([::1]:46440 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nlqxS-0004I0-4P for larch@yhetil.org; Tue, 03 May 2022 07:44:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38602) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nlqwl-0003dv-VB for guix-patches@gnu.org; Tue, 03 May 2022 07:44:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44943) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nlqwl-0002Rc-M8 for guix-patches@gnu.org; Tue, 03 May 2022 07:44:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1nlqwl-0005QC-K3 for guix-patches@gnu.org; Tue, 03 May 2022 07:44:03 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#55242] [PATCH 06/10] guix: import: go: Add a local duplicate of http-fetch. Resent-From: Attila Lendvai Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 03 May 2022 11:44:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 55242 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 55242@debbugs.gnu.org Cc: Attila Lendvai Received: via spool by 55242-submit@debbugs.gnu.org id=B55242.165157821320737 (code B ref 55242); Tue, 03 May 2022 11:44:03 +0000 Received: (at 55242) by debbugs.gnu.org; 3 May 2022 11:43:33 +0000 Received: from localhost ([127.0.0.1]:38826 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nlqwH-0005OH-Af for submit@debbugs.gnu.org; Tue, 03 May 2022 07:43:33 -0400 Received: from mail-ed1-f44.google.com ([209.85.208.44]:39809) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nlqwE-0005N9-Gz for 55242@debbugs.gnu.org; Tue, 03 May 2022 07:43:31 -0400 Received: by mail-ed1-f44.google.com with SMTP id g20so19511897edw.6 for <55242@debbugs.gnu.org>; Tue, 03 May 2022 04:43:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gdXWK8w8gBi+z5S1dgCIitSmkMrxIP2VPhd8XbARarQ=; b=HyT0crT8mm3ccVqm4+LrFQX4+WiSxtSoDxXeiDCKMi0m0+yfRxgHbU335LD3UyA2qS faSoQOL1S7Ic4iyEgX8MymFdKKpsUSTMqMsy9BCttAhPXeYfJEAr7W4RkQm+YNCsa9SB o2u/iUFOjx3OOOVjJEmb4qTziWQQapiyf13ZizMbAAxUChmv28eos7AVXQ3ObP2WESNS uRc6tLh+MfV1WVZpyh1DLZgGc7EwctnBqnTVTNTywVfW+cJjUT+itpCoGytSnAwNLC4Z MPIGdy5m3WEcMeXjYAeGVL3olBC1pLNrUsf88UUT1nCjIV9MbxrdSu0iB/XkQPpfTdJY Z/UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=gdXWK8w8gBi+z5S1dgCIitSmkMrxIP2VPhd8XbARarQ=; b=Fn66oExjvWeDzNZ65FwRpVFfoi8CYxe9yIfoSz4hQ/wC2WezgBgbxltyjOg4SD5jgq D03JR5CurhRKeFmcI0EY+EN7zkrWQpfFRcCK3jCePWuhkcNGJ+rdSqoGD1yVfPll9/i6 RdiPIaatifV+Wh70PduLTFjBzkkr3MrONzbrxYKWOWDFaPueByFY55JC04eOtL5UPLcp eaMZLU8kciFXf9USIcWb+sT9tOL3hJ7uVJ8V/HEeDVD+cq7RRTL65NbpKD8Yrhh2TsA+ hSp0zxwax0C88DLL9Tq7YkGoz4DilZQA9R2+hMk2SGAjHwueimyLty5N52p9VD5zYdC9 j5NQ== X-Gm-Message-State: AOAM533iBh/q1QIpqtHKQKtmkqqikcVEBvESx7TJ0IvFPAz9SeMYlfDX ay6cJqPIHvr+Rfbe885u4P47E8OZu28= X-Google-Smtp-Source: ABdhPJw6GCQJ3tKo7W92ii6KOVMfj+EE/DhSOzgMXDjsmzwQY7qmRgluEc7ZYnlbxQhB/J6RQXdhMw== X-Received: by 2002:a05:6402:354a:b0:427:d0e1:8ef3 with SMTP id f10-20020a056402354a00b00427d0e18ef3mr7012237edd.66.1651578204809; Tue, 03 May 2022 04:43:24 -0700 (PDT) Received: from lelap.local (catv-89-132-245-188.catv.fixed.vodafone.hu. [89.132.245.188]) by smtp.gmail.com with ESMTPSA id k13-20020a05640212cd00b0042617ba63d6sm7766964edx.96.2022.05.03.04.43.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 May 2022 04:43:24 -0700 (PDT) From: Attila Lendvai Date: Tue, 3 May 2022 13:42:57 +0200 Message-Id: <20220503114301.9524-6-attila@lendvai.name> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220503114301.9524-2-attila@lendvai.name> References: <20220503114301.9524-2-attila@lendvai.name> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1651578288; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=gdXWK8w8gBi+z5S1dgCIitSmkMrxIP2VPhd8XbARarQ=; b=Hy/5BE89NEI1fAbBMcE0i83uyHLomBwSTf/xczzjKOXE888oZyBQ/bOfDYLwVBUIHhav7I /Wm1Wu+SL/qxHRHN0HPcS3WaQTXQwNTV3GWozpPk0SMTbpEJVjcnSWVGUdyxo1S6xotj6I CNd4qa6D1GTSRkIgK8n8+EZoSsB+Jz6Y8qXu1uifwpO2/u0QiY4LWfA412LZ6pTzEKj8Kj UgsPvsUzwo/5qUJUAT6wFczWdAhT8Zrhd7mrgTX6h0VRxrOJe44FbUjt9J9VZ2JvlsXpiF HLuWTkEH0tFOy1vMdmUtsWclF+rZisaxW0Bj3tfL3+v0bxPmSAQo1EdqU8G7aQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1651578288; a=rsa-sha256; cv=none; b=smite4Rm56Z+2tCQ+i0B6aBT+a3/eDz7Pa/pnf1UN8Tg2xyBIhSLtadJjB1WslKgRWuNHv ovDBbswlchNJyTc4J53i9Yd/4Gb9SaabKCRNcEG1bNzluuHIECB54PpQtkoWG4JEea2EOz KGixAJdvWlspjPAX1KbRZ7JodvHEJQGfmr06QezKdB60JXaRSAjX4Uzt4oCA1ly1f35l0U ZdK9awHiRWxrBjKyxnmCL672OYHpAXqsJYE4TX3jkCXHH02BMfh/JxsE38UnbYqPSACKxJ JR11mQKuCg97QHAnNTmw+2MR2XdmEbFWMrjDR67lAw1Tr1EHTlHvHSr+d0eivw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=HyT0crT8; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 2.02 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20210112 header.b=HyT0crT8; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: E39521F199 X-Spam-Score: 2.02 X-Migadu-Scanner: scn0.migadu.com X-TUID: 6hq/PEH08OL6 This is needed when dealing with golang packages, as per: https://golang.org/ref/mod#vcs-find A page may return 404, but at the same time also contain the sought after `go-import` meta tag. An example for such a project/page is: https://www.gonum.org/v1/gonum?go-get=1 It's not enough to just handle the thrown exception, because we need to be able to get hold of the fetched content, too. Discussion why it's duplicated here: https://issues.guix.gnu.org/54836 * guix/import/go.scm (http-fetch): Add a copy and extend it with the accept-all-response-codes? param. --- guix/import/go.scm | 107 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 105 insertions(+), 2 deletions(-) diff --git a/guix/import/go.scm b/guix/import/go.scm index 6b0fbaa8b6..a08005d090 100644 --- a/guix/import/go.scm +++ b/guix/import/go.scm @@ -33,13 +33,22 @@ (define-module (guix import go) #:use-module (guix import json) #:use-module (guix packages) #:use-module ((guix utils) #:select (string-replace-substring)) - #:use-module (guix http-client) + ;; FIXME? We use a local copy of http-fetch. + ;; See https://issues.guix.gnu.org/54836 + #:use-module ((guix http-client) #:hide (http-fetch)) + #:use-module (guix base64) + #:use-module (rnrs bytevectors) + #:use-module ((guix build download) + #:select (open-socket-for-uri + (open-connection-for-uri + . guix:open-connection-for-uri) + resolve-uri-reference)) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix memoization) #:autoload (htmlprag) (html->sxml) ;from Guile-Lib #:autoload (guix serialization) (write-file) #:autoload (guix base32) (bytevector->nix-base32-string) - #:autoload (guix build utils) (mkdir-p) + #:autoload (guix build utils) (mkdir-p dump-port) #:autoload (gcrypt hash) (hash-algorithm sha256) #:use-module (ice-9 format) #:use-module (ice-9 match) @@ -69,6 +78,100 @@ (define-module (guix import go) go-module->guix-package* go-module-recursive-import)) +;; This is a duplicate from (guix http-client) with the addition of a +;; #:accept-all-response-codes? param. See https://issues.guix.gnu.org/54836 +(define* (http-fetch uri #:key port (text? #f) (buffered? #t) + (open-connection guix:open-connection-for-uri) + (keep-alive? #f) + (verify-certificate? #t) + (headers '((user-agent . "GNU Guile"))) + (log-port (current-error-port)) + timeout + (accept-all-response-codes? #f)) + "Return an input port containing the data at URI, and the expected number of +bytes available or #f. If TEXT? is true, the data at URI is considered to be +textual. Follow any HTTP redirection. When BUFFERED? is #f, return an +unbuffered port, suitable for use in `filtered-port'. HEADERS is an alist of +extra HTTP headers. + +When KEEP-ALIVE? is true, the connection is marked as 'keep-alive' and PORT is +not closed upon completion. + +When VERIFY-CERTIFICATE? is true, verify HTTPS server certificates. + +TIMEOUT specifies the timeout in seconds for connection establishment; when +TIMEOUT is #f, connection establishment never times out. + +Write information about redirects to LOG-PORT. + +When ACCEPT-ALL-RESPONSE-CODES? is false then raise an '&http-get-error' +condition if downloading fails, otherwise return the response regardless +of the reponse code." + (define parsed-initial-uri + (if (string? uri) (string->uri uri) uri)) + + (define (open-connection* uri) + (open-connection uri + #:verify-certificate? verify-certificate? + #:timeout timeout)) + + (let loop ((current-uri parsed-initial-uri) + (current-port (or port (open-connection parsed-initial-uri)))) + (let ((headers (match (uri-userinfo current-uri) + ((? string? str) + (cons (cons 'Authorization + (string-append "Basic " + (base64-encode + (string->utf8 str)))) + headers)) + (_ headers)))) + (unless (or buffered? (not (file-port? current-port))) + (setvbuf current-port 'none)) + (let*-values (((resp data) + (http-get current-uri #:streaming? #t #:port current-port + #:keep-alive? keep-alive? + #:headers headers)) + ((code) + (response-code resp))) + (case code + ((200) + (values data (response-content-length resp))) + ((301 ; moved permanently + 302 ; found (redirection) + 303 ; see other + 307 ; temporary redirection + 308) ; permanent redirection + (let ((host (uri-host current-uri)) + (new-uri (resolve-uri-reference (response-location resp) + current-uri))) + (if keep-alive? + (dump-port data (%make-void-port "w0") + (response-content-length resp)) + (close-port current-port)) + (format log-port (G_ "following redirection to `~a'...~%") + (uri->string new-uri)) + (loop new-uri + (or (and keep-alive? + (or (not (uri-host new-uri)) + (string=? host (uri-host new-uri))) + current-port) + (open-connection* new-uri))))) + (else + (if accept-all-response-codes? + (values data (response-content-length resp)) + (raise (condition (&http-get-error + (uri current-uri) + (code code) + (reason (response-reason-phrase resp)) + (headers (response-headers resp))) + (&message + (message + (format + #f + (G_ "~a: HTTP download failed: ~a (~s)") + (uri->string current-uri) code + (response-reason-phrase resp))))))))))))) + ;;; Commentary: ;;; ;;; (guix import go) attempts to make it easier to create Guix package -- 2.35.1