From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id wCNXHOBxWWCAXQAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 04:43:12 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id CFGFF+BxWWDxTgAAB5/wlQ (envelope-from ) for ; Tue, 23 Mar 2021 04:43:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9FE4D1B688 for ; Tue, 23 Mar 2021 05:43:11 +0100 (CET) Received: from localhost ([::1]:49702 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOYsn-00080X-JB for larch@yhetil.org; Tue, 23 Mar 2021 00:43:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59630) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYsg-00080F-Cu for bug-guix@gnu.org; Tue, 23 Mar 2021 00:43:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:47695) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOYsg-0008Jm-5J for bug-guix@gnu.org; Tue, 23 Mar 2021 00:43:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOYsg-0001qs-2B for bug-guix@gnu.org; Tue, 23 Mar 2021 00:43:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#47336: Disarchive as a fallback for downloads Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16164745447069 (code B ref -1); Tue, 23 Mar 2021 04:43:01 +0000 Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000 Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs3-0001pw-WB for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:51240) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs2-0001pp-0u for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYs1-0007wV-QR for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYry-0007rZ-7A for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D60141373 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/ YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:42:12 -0400 Message-ID: <87eeg6o50b.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@ngyro.com; helo=wout3-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616474592; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=lFT6ACaQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=R6eDg33HV39xkz3BzOmGoEMOEjwJr9c9SVT5KhN1iQNoODhxMfP/Iva9GtYRH0M+fAF0Ce O0C9avDHH3e35BIaQ6v9kbvFnG+Aa+XO7jM9XfXVHky04Y97DaB++y2NrxEkGNWmQe811T EHllfluoDDbgWVjnDI0PPd5P7eBZf2cEKj1R2Yp8KH5GK1QxGUvfXWK58+H769gHTeihOK QGYSekh5fdUKRWvq9iF+eNOWqFDcZV14yWCKMlPYHWa8O0sWilYbgDaUpKuKdNu496RXJ1 vj9VA4tPPCBOergxroA/lMSTJQ1diM66lu0AeqaxqPhxT8KDPjgkPzjtzt0HMg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616474592; a=rsa-sha256; cv=none; b=bEm4g/C3knAsr7a/sjg6B4g22WHGd32rMjgkgQQQpQ3oN2epzwIVaNv9slaCNNkFtyaT+X xIQ0ykMcD62Ih+fxSGWn2rhD4cOQnKvrNqsS+UnvOvlLSjLVx++01i0CJDRnss8KTGnqaO P/p6KOtoj6pSet6ISTBzEpC7xYvkmqjDfF2H8C1R6OyfTv9b4iW7MikqIAYuG65+WLWqac /6mP+ErjdSnE+a5L+Tu24soAFtWnXcPO+xmYUJIQ26QbCMvwaojHWVjrzMNNxeYWKhPuUO SFWtEASnKNJIN7bsPpjcmgR5OTziePDPvbRK8eJw1JNm7ibfUEJjQaUsbLZhyA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="X/5rSkWH"; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -1.42 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b="X/5rSkWH"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: 9FE4D1B688 X-Spam-Score: -1.42 X-Migadu-Scanner: scn0.migadu.com X-TUID: N1S48IOq9PDD Hello, This patch series adds Disarchive assembly (backed by SWH lookup) as a fallback for downloads. To try it, make sure you are running the daemon in an environment with Disarchive available: $ ./pre-inst-env guix environment --ad-hoc guile disarchive # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild Don=E2=80=99t forget to stop your existing Guix Daemon. :) You also need to make sure that regular downloads are unavailable. I do this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl= -fetch=E2=80=9D in =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of URL= s with =E2=80=98()=E2=80=99: (let try ((uri (append uri content-addressed-uris))) (match '() ; uri ...)) Now you can ask Guix for a recent .tar.gz source package: $ ./pre-inst-env guix build --no-substitutes -S python-httpretty You should see: Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v= diwlv19y9-httpretty-1.0.5.tar.gz Assembling the directory httpretty-1.0.5 Downloading from Software Heritage... 7903d608efc89c14afb4d692a3721156e31a43e2/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING [...] Checking httpretty-1.0.5 digest... ok Assembling the tarball httpretty-1.0.5.tar Checking httpretty-1.0.5.tar digest... ok Assembling the Gzip file httpretty-1.0.5.tar.gz Checking httpretty-1.0.5.tar.gz digest... ok Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty= -1.0.5.tar.gz successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett= y-1.0.5.tar.gz.drv There=E2=80=99s lots to talk about though.... First, it looks up the metadata on my server. This is fine for a demo, but not what we want forever. The patch series supports adding several mirrors for looking up the metadata. In the past, we talked about putting everything on one or a few of the big Git hosting platforms like GitHub or Gitlab. That way, it would be easily picked up by SWH and archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to bu= ild the metadata, and a little script that moves it from the build server to my Web server. It would be simple enough to adjust that script to push it to a remote Git repo. (Of course, the next step is to move this setup to Guix infrastructure.) Thoughts? On the code level, there were two things I couldn=E2=80=99t figure out for myself. I made the mirror list just simple strings. AIUI, the client and the daemon have to agree about the format of the mirror list. Given that running old daemons is common, changing the format is difficult. Is it worth it to copy the more flexible interface used by the content addressed mirrors? If yes, do I have to do the same =E2=80=98module-autolo= ad!=E2=80=99 dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probably = would have just copied it, but that part confused me a bit.) I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel= l, just =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to use= a bunch of host-side modules from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. This = is okay because =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. = It=E2=80=99s more like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2= =80=9D, which is why importing host-side modules is OK... right? Hopefully everything else is more-or-less fine. :) -- Tim