From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id RFjKKDJ51GDwBgEAgWs5BA (envelope-from ) for ; Thu, 24 Jun 2021 14:23:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id QLD1IzJ51GCRMQAA1q6Kng (envelope-from ) for ; Thu, 24 Jun 2021 12:23:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C979F91D0 for ; Thu, 24 Jun 2021 14:23:13 +0200 (CEST) Received: from localhost ([::1]:36518 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lwOO0-0002tt-Fa for larch@yhetil.org; Thu, 24 Jun 2021 08:23:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lwONr-0002rS-PK for guix-patches@gnu.org; Thu, 24 Jun 2021 08:23:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:59278) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lwONq-0002hs-BQ for guix-patches@gnu.org; Thu, 24 Jun 2021 08:23:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lwONq-0008IH-4I for guix-patches@gnu.org; Thu, 24 Jun 2021 08:23:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#49196] [PATCH] import: utils: 'recursive-import' skips unfound packages Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 24 Jun 2021 12:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49196 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Sarah Morgensen Cc: 49196@debbugs.gnu.org Received: via spool by 49196-submit@debbugs.gnu.org id=B49196.162453736131838 (code B ref 49196); Thu, 24 Jun 2021 12:23:02 +0000 Received: (at 49196) by debbugs.gnu.org; 24 Jun 2021 12:22:41 +0000 Received: from localhost ([127.0.0.1]:42591 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwONN-0008HK-EI for submit@debbugs.gnu.org; Thu, 24 Jun 2021 08:22:41 -0400 Received: from mail-wm1-f43.google.com ([209.85.128.43]:54257) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lwONL-0008H4-JH for 49196@debbugs.gnu.org; Thu, 24 Jun 2021 08:22:32 -0400 Received: by mail-wm1-f43.google.com with SMTP id w13so3833069wmc.3 for <49196@debbugs.gnu.org>; Thu, 24 Jun 2021 05:22:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=lbGWsfwxQyBkXcUFnLP4I5PV6YgDveXoE7i4ODBN1IY=; b=bZQhLBJDABwcw3cK1oo5kX8vyu3txaWEduG46MUgM0Vyoh6XM7jGmDkz1tXNbxWR7h ad1asPVVb4uPHxMImVU3JZuon3HFDAOAI0FME/OnSSG2VXm9PkrtqxCar9vewceKeUXz rJN9KywAWRj2KgUSwT1YZxQLIRGb0y3pxscQC4Uy2D1+bnu2UElRMQsLBbqe5sA32p83 CwgWB9HLlqM64Fjhud0wpucuBnXcIvE7vbrQgvjB+VPr109j6ctjQ11bcD5RC5aR/gNw h2vfFg9pGtH5VzgmBddXbhHS0c7miaHs4WHcnz5Pp9W95YwZZscmPSi7mbG4zg9zGIbW vkcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=lbGWsfwxQyBkXcUFnLP4I5PV6YgDveXoE7i4ODBN1IY=; b=mrzdNS0Q7a/Lab/UerJXuA8rBOaINtOX0PQTBshCw2Kijz95PonXiZJ4fCwnEeOGcF 9Z9gJCeH1ClR5AwgfhjnxOi6SQOOo59t1PnBGjLXJg6al/3z5IQTJkNPOtDFPkym7BpO 62skuRKEO2hXhay1cJp0vzUWSnf265WrOe9/PbRcGIjOam+PpUJlBjHTr1CN6/hBo8dY d4+15/bQNPqZmTsdprQwl8I/O0htiRAA4ZI7OxfHBGWpTs1Yre+/FL2ZIjzcUpzMT0ED d1mnofP4U0i9Q2XgpWCztg3Ldxx9MxQwKZAQSjyUEhB8yniGm2VFiu6ezR5QZ4/+4IvS yBlg== X-Gm-Message-State: AOAM532gMWlt7wz9c1fa77cPYWzbSwLQZq4WxAGxMi/P0/VtLfi3mA/z zAKlEtPT/m9eq8Fe4KiGR+Vlpk3lxZ8= X-Google-Smtp-Source: ABdhPJzMpSumQomh9Idt/4QIw/04eYjEP0Vl3IKeN59RtUWWg+9VsZ0yFSWbUPASg1+HA0V/Jyq71g== X-Received: by 2002:a05:600c:4417:: with SMTP id u23mr4004646wmn.26.1624537345524; Thu, 24 Jun 2021 05:22:25 -0700 (PDT) Received: from bioinfomeary01-Precision-7820-Tower ([193.48.40.110]) by smtp.gmail.com with ESMTPSA id x25sm9162493wmj.23.2021.06.24.05.22.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 05:22:25 -0700 (PDT) From: zimoun References: <07d7149fc0f89f7f2d11fba47e1b0b2db5ceb809.1624479231.git.iskarian@mgsn.dev> Date: Thu, 24 Jun 2021 14:21:37 +0200 In-Reply-To: <07d7149fc0f89f7f2d11fba47e1b0b2db5ceb809.1624479231.git.iskarian@mgsn.dev> (Sarah Morgensen's message of "Wed, 23 Jun 2021 13:49:05 -0700") Message-ID: <854kdna1wu.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1624537394; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=lbGWsfwxQyBkXcUFnLP4I5PV6YgDveXoE7i4ODBN1IY=; b=lAUpWbPBGSEJdG48bDr/05BQ0FMhw1mwzQ2dx6rdgXag2hlelg+d3HWMpho66ywBTQv87d vt6ex8Y23S06vnSvlgPNPAoBJE6Iy0flf4J7D5mIHnVSoapaPbluQAwR0PpqjcCxqj2FZ0 +mSwmkDVcMDx7tu5kQZwLquh25VpHMysmwdRyHAMIHHPQoo65PSrHx8NKHsCmsR/NXMgLH aD7UYrsIj+f2xPstZpR2TfmROqr/b4iM7X/ITVT3h8dLq52fS8bq2UXGe5g+gsHQRVwcrI fpS5sJUxbRKDf8XNqPqiLGCS+aqmNDGwg1bUToPwuHDww4mGth4E8XQlhurWoQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1624537394; a=rsa-sha256; cv=none; b=hbG7KfJdd6G3GF77Uj+L6PEHLRsf6qz8aMWHFeItEVY8bhUZBpWh5Yjr1D5uFRAHJK0lPm gzJiUDFF2uI9uc3fVc3YuPBiKt5FeMjGigOYtjMubtzgWzKYB1mfFqgsySkSbWOV8A6kRX xa5mzLrjARwZPZdN6RPtnfp6y0zwbohFvB0wTSGNzQSQkAh4EPOhh69adO/+lkawh04S/A qGyHFc/dTQ5BJu3kPE6RizkciL+8PrhrnnLG2SHxZ/+0IMEG5TuEmwtmnDRDPvlDqvmHqV EpfL/S0USTEqER6+s/P3/DCsmG6EiViOVrAtYC+/yvKQRF4DB+A1tvTwsEZK3Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=bZQhLBJD; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.33 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=bZQhLBJD; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: C979F91D0 X-Spam-Score: -1.33 X-Migadu-Scanner: scn1.migadu.com X-TUID: HhPamSLovxMU Hi, Thanks for the patch. On mer., 23 juin 2021 at 13:49, Sarah Morgensen wrote: > It seems like several importers (gem, egg, go) expect to be able to retur= n #f or > '() in #:repo->guix-package when a package is not found (while printing a > warning to the user) but recursive-import doesn't look like it handles it= . I > suppose this worked before but was silently broken at some point. This pa= tch > (re-)enables this behavior. I added a test for this as well. Indeed, there is an inconsistency betweem all the recursive importers. An attempt to fix this is done by [1]. 1: > diff --git a/guix/import/go.scm b/guix/import/go.scm > index d110954664..c859098492 100644 > --- a/guix/import/go.scm > +++ b/guix/import/go.scm > @@ -5,6 +5,7 @@ > ;;; Copyright =C2=A9 2021 Maxim Cournoyer > ;;; Copyright =C2=A9 2021 Ludovic Court=C3=A8s > ;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Sarah Morgensen > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -651,7 +652,7 @@ This package and its dependencies won't be imported.~= %") > (uri->string (http-get-error-uri c)) > (http-get-error-code c) > (http-get-error-reason c)) > - (values '() '()))) > + (values #f '()))) Yes, there is an inconsistency... > (receive (package-sexp dependencies) > (go-module->guix-package* name #:goproxy goproxy > #:version version > diff --git a/guix/import/utils.scm b/guix/import/utils.scm > index d817318a91..49f38cfa2a 100644 > --- a/guix/import/utils.scm > +++ b/guix/import/utils.scm > @@ -8,6 +8,7 @@ > ;;; Copyright =C2=A9 2020 Helio Machado <0x2b3bfa0+guix@googlemail.com> > ;;; Copyright =C2=A9 2020 Martin Becze > ;;; Copyright =C2=A9 2021 Maxim Cournoyer > +;;; Copyright =C2=A9 2021 Sarah Morgensen > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -469,16 +470,17 @@ to obtain the Guix package name corresponding to th= e upstream name." > (normalized-deps (map (match-lambda > ((name version) (list name version)) > (name (list name #f))) dependencies))) > - (make-node name version package normalized-deps))) > + (and package > + (make-node name version package normalized-deps)))) > > (map node-package > (topological-sort (list (lookup-node package-name version)) > (lambda (node) > - (map (lambda (name-version) > - (apply lookup-node name-version)) > - (remove (lambda (name-version) > - (apply exists? name-version)) > - (node-dependencies node)))) > + (filter-map (lambda (name-version) > + (apply lookup-node name-version= )) > + (remove (lambda (name-version) > + (apply exists? name-ver= sion)) > + (node-dependencies node))= )) ...however, I am not convinced this fixes the issue. Compare: --8<---------------cut here---------------start------------->8--- $ guix import go do-not-exist -r guix import: warning: Failed to import package "do-not-exist". reason: "https://proxy.golang.org/do-not-exist/@v/list" could not be fetche= d: HTTP error 410 ("Gone"). This package and its dependencies won't be imported. Backtrace: 4 (primitive-load "/home/sitour/.config/guix/current/bin/guix") In guix/ui.scm: 2147:12 3 (run-guix-command _ . _) In guix/scripts/import.scm: 120:11 2 (guix-import . _) In srfi/srfi-1.scm: 586:17 1 (map1 (())) In guix/import/utils.scm: 280:2 0 (package->definition _ _) guix/import/utils.scm:280:2: In procedure package->definition: Throw to key `match-error' with args `("match" "no matching pattern" ())'. --8<---------------cut here---------------end--------------->8--- with: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix import go do-not-exist -r guix import: warning: Failed to import package "do-not-exist". reason: "https://proxy.golang.org/do-not-exist/@v/list" could not be fetche= d: HTTP error 410 ("Gone"). This package and its dependencies won't be imported. Backtrace: In ice-9/boot-9.scm: 1752:10 8 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) In unknown file: 7 (apply-smob/0 #) In ice-9/boot-9.scm: 724:2 6 (call-with-prompt _ _ #) In ice-9/eval.scm: 619:8 5 (_ #(#(#))) In guix/ui.scm: 2147:12 4 (run-guix-command _ . _) In guix/scripts/import.scm: 120:11 3 (guix-import . _) In guix/scripts/import/go.scm: 116:20 2 (guix-import-go . _) In guix/import/utils.scm: 440:34 1 (recursive-import _ #:repo->guix-package _ #:guix-name _ #:ver= sion _ #:repo _) 486:28 0 (_ #f) guix/import/utils.scm:486:28: In procedure struct-vtable: Wrong type argume= nt in position 1 (expecting struct): #f --8<---------------cut here---------------end--------------->8--- Then, the patch introduces issues against other importers, for instance: --8<---------------cut here---------------start------------->8--- r@bioinfomeary01-Precision-7820-Tower$ guix import gem do-not-exist -r #f $ ./pre-inst-env guix import gem do-not-exist -r Backtrace: In ice-9/boot-9.scm: 1752:10 8 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) In unknown file: 7 (apply-smob/0 #) In ice-9/boot-9.scm: 724:2 6 (call-with-prompt _ _ #) In ice-9/eval.scm: 619:8 5 (_ #(#(#))) In guix/ui.scm: 2147:12 4 (run-guix-command _ . _) In guix/scripts/import.scm: 120:11 3 (guix-import . _) In guix/scripts/import/gem.scm: 97:16 2 (guix-import-gem . _) In guix/import/utils.scm: 440:34 1 (recursive-import _ #:repo->guix-package _ #:guix-name _ #:ver= sion _ #:repo _) 486:28 0 (_ #f) guix/import/utils.scm:486:28: In procedure struct-vtable: Wrong type argume= nt in position 1 (expecting struct): #f --8<---------------cut here---------------end--------------->8--- Well, it is not worse for most of the importers. And perhaps this patch is worth. As explained in [1], all the recursive importers should be unified and the errors correctly handled, IMHO. With jeko, we are pair-programming to work on it... but we are really slow. ;-) > diff --git a/tests/import-utils.scm b/tests/import-utils.scm > index 874816442e..1b728104e0 100644 > --- a/tests/import-utils.scm > +++ b/tests/import-utils.scm Thanks for adding a test. :-) Cheers, simon