From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id UAQbJyC8WWCJMQAA0tVLHw (envelope-from ) for ; Tue, 23 Mar 2021 10:00:00 +0000 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id UEL/IiC8WWB5MQAAB5/wlQ (envelope-from ) for ; Tue, 23 Mar 2021 10:00:00 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CC3642CBE5 for ; Tue, 23 Mar 2021 10:59:59 +0100 (CET) Received: from localhost ([::1]:55660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lOdpO-0006AB-A3 for larch@yhetil.org; Tue, 23 Mar 2021 05:59:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59692) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOdbv-0001Xs-9f for guix-patches@gnu.org; Tue, 23 Mar 2021 05:46:05 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:47991) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lOdbt-00052N-Tn for guix-patches@gnu.org; Tue, 23 Mar 2021 05:46:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lOdbt-0003F9-Sb for guix-patches@gnu.org; Tue, 23 Mar 2021 05:46:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 23 Mar 2021 09:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample , 47336@debbugs.gnu.org, Mathieu Othacehe Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161649270212351 (code B ref 47336); Tue, 23 Mar 2021 09:46:01 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 09:45:02 +0000 Received: from localhost ([127.0.0.1]:59537 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOdav-0003Cz-Cr for submit@debbugs.gnu.org; Tue, 23 Mar 2021 05:45:01 -0400 Received: from mail-wm1-f50.google.com ([209.85.128.50]:55998) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOdat-0003Ck-Fq for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 05:44:59 -0400 Received: by mail-wm1-f50.google.com with SMTP id 12so10656916wmf.5 for <47336@debbugs.gnu.org>; Tue, 23 Mar 2021 02:44:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:in-reply-to:references:date:message-id:mime-version :content-transfer-encoding; bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=; b=a1kWhOty04rSkuZn+yOOy75QNSCxZU3tKaRoQzcBkwzUhLzQRGMidjhxwTCzmeQctS ujC55BPtDGxZVtFweOzRrWLjaea8ykqig+xmrybhktNCqYZZ9OwoS/Hx5fBtX7AKaHJs bSurhEv0RzAsjKnBE7drV8Hi2KxB7QCqXt0RfVdthLcy28j6WCVUSppEVi7ucmgMb2ph skLA9ocM+AYUU22B51RMDoyTuq4UwLQ7Ox9FMI2ApiZnNUunp+TRF+LWf6QxVKmXpgkB fUdTJFwHRzd6iuFDxCbUNBDAH2vITfyRpBbTFvGPJXcj1v3Vqm+xcmi2lixcnYGJmZsk nztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=; b=j1s02Fyb9xGS3D9uIpCmDT7+9dPrkxnTE2BtP3W6bowPwzQN5MrTntBrrV4Xgzyo7h Hqv2mSJyi7t5AggIjot1kn4HxsvvFjfrMu8SMA+pp8gKHtWTCyYcZo3g66mud9s/7OYm KPmRsdWM+Sh21aEL/tghxI8x9Z9NHPH15tqpHGFDxQTfIFVCm3C5TwifjapaC8z62hIW Rr+atHIUYVtrEa/TRjvF7eBB1BYIoEmQfqcrLEhRWCV3y6LLpHrgSItWqVd68JJcw5aN Ck9Pn3VetH+QPk9n3tEgrbq2/xkqNGMInzDZUW6DPQ/9QqirvJAK9aYKVnFbda0jvHXP nu7w== X-Gm-Message-State: AOAM531oRr/FTg6xo2XF6gFkoebVpwVtFhTSEAHbdnoCK9klky+yS3/9 JcIQxjFzAdChFx4cNLG6q7YAG9RZo2I= X-Google-Smtp-Source: ABdhPJzJrqraIVEvPjdWBzhyCgBvir/3yhgtfd13IxE/sysUfOFhuI40WjeXUCiXzi1VC5UM04lndw== X-Received: by 2002:a1c:66c4:: with SMTP id a187mr2474403wmc.164.1616492693386; Tue, 23 Mar 2021 02:44:53 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id s83sm2074886wmf.26.2021.03.23.02.44.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Mar 2021 02:44:53 -0700 (PDT) From: zimoun In-Reply-To: <87eeg6o50b.fsf@ngyro.com> References: <87eeg6o50b.fsf@ngyro.com> Date: Tue, 23 Mar 2021 10:35:53 +0100 Message-ID: <86sg4mnreu.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616493600; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=; b=dh0JVSzWTJRw9azm0ZpYf/t7OMkJYn3SrAmp4BWcbNqILa9vuAtDIl3sPi6tA3voXsRu3w qwOrsDw6VQl35hZ4ox4AUsWZ2EOhv+hwaJXezh8OpGPkvUKMn+i/0gW7byTKDqfd1RzodS 2HgBjhrEe6UUHGuk75f8SToB/vDWjGOIJ2Buye57P/WE885JchmOrRCOzApujn4vrGaDe7 Zh73BVFm0GQB4w4PXSd6texHUBkTShtoybP1u6OGyp3zz2BJcR3hEmGBrsvbRbWq/WOoYr Nov4ITyjavuCcbpHgyARodWWDUelJ8MU9O5z9rb1+LHcBiN2DtJiD0dTtWLPgw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616493600; a=rsa-sha256; cv=none; b=ko0zrPheOz5BM06ePoFXy8PIxH+IM6IZAVmNKumQ5Cp/3U/FJFvamcmMprSoimSsQuAM8M 2Sg2GfdSAz3e2FYWIjQeI5FJ/jKe1xUlvodPnpkxyw+OJsgZOQbCq9s+t4reyaDSJqgzQF lRScIYuS/8crdol0Ho5EpfRNLaOOWaaVWvqaosy2AMz1WmqA1E9s2kidC7GfGfAmqfRRtM Tcp6ciywkE2/Dv0ie1B0L4wmfU4uDaF3MKdqJi2PF8cRLslU3j7jD1cGAdmXHpDe+BCN+Z 8i4pe+aYvGmDHacINpKbykRAdwjm8lltT/DuQbdnhdb/3yNZgqqLt1UlNYhOcA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=a1kWhOty; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=a1kWhOty; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: CC3642CBE5 X-Spam-Score: -1.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: YRc8QRdaMzBF Hi Timothy, (CC Mathieu to advice if it could be a feature of Cuirass.) On Tue, 23 Mar 2021 at 00:42, Timothy Sample wrote: > This patch series adds Disarchive assembly (backed by SWH lookup) as a > fallback for downloads. Awesome! > You also need to make sure that regular downloads are unavailable. I do > this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Cu= rl-fetch=E2=80=9D in > =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of U= RLs with =E2=80=98()=E2=80=99: > > (let try ((uri (append uri content-addressed-uris))) > (match '() ; uri > ...)) > > Now you can ask Guix for a recent .tar.gz source package: > > $ ./pre-inst-env guix build --no-substitutes -S python-httpretty Neat! Now, there is a way to easily check the coverage, right? Since SWH is ingesting the tarball using , there is now a mean to report what Guix is able to rebuild. > Checking httpretty-1.0.5 digest... ok What happens if it is not ok? > Assembling the tarball httpretty-1.0.5.tar > Checking httpretty-1.0.5.tar digest... ok > Assembling the Gzip file httpretty-1.0.5.tar.gz > Checking httpretty-1.0.5.tar.gz digest... ok > Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpret= ty-1.0.5.tar.gz Where is the assembly done? In /tmp/, right? > successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpre= tty-1.0.5.tar.gz.drv Just to be sure, when does Guix check the integrity checksum? I mean, does Guix check the checksum after =E2=80=99disassemble=E2=80=99 re-assembl= ed the source? > First, it looks up the metadata on my server. This is fine for a demo, > but not what we want forever. The patch series supports adding > several As we talked before, how does the database scale? Do you have some numbers for the current demo? In order to try to extrapolate what does it mean for a server to =C2=ABstore the metadata=C2=BB. > mirrors for looking up the metadata. In the past, we talked about > putting everything on one or a few of the big Git hosting platforms like > GitHub or Gitlab. That way, it would be easily picked up by SWH and > archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to = build the > metadata, and a little script that moves it from the build server to my > Web server. It would be simple enough to adjust that script to push it > to a remote Git repo. (Of course, the next step is to move this setup > to Guix infrastructure.) Thoughts? Maybe this database could be a package, say =E2=80=9Cguix-tarball-db=E2=80= =9D, updated in agreement with the package =E2=80=9Cguix=E2=80=9D. The source of this =E2=80=9Cguix-tarball-db=E2=80=9D would be a remote big Git hosting platfor= ms like GitHub or whatever and not stored on Guix infrastructure, or maybe stored on Guix infra. Regularly, i.e., when the package =E2=80=9Cguix=E2=80=9D is updated, in the= same time, the package =E2=80=9Cguix-tarball-db=E2=80=9D is updated too. The =E2=80= =9Cguix lint -c archival=E2=80=9D sends the saving request to SWH. Even if this saving req= uest should be automated soon. :-) Then if Cuirass would have a feature to disassemble and update the Git repo. Last, a service should run as your demo. But for long-term, this service could disappear=E2=80=93=E2=80=93assuming SWH not :-). Therefore, = we could imagine installing =E2=80=9Cguix-tarball-db=E2=80=9D then tweak some parame= ters of the guix-daemon and =E2=80=9Cguix build =E2=80=9D. Both installing and bu= ilding would fetch from SWH if both upstream disappear. Or this =E2=80=9Cguix-tarball-db=E2=80=9D should not be a plain package but= only an input as origin for the package =E2=80=9Cguix=E2=80=9D. > Hopefully everything else is more-or-less fine. :) Thanks! That=E2=80=99s awesome! Cheers, simon