From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id WEFeEuLhrWNqewAAbAwnHQ (envelope-from ) for ; Thu, 29 Dec 2022 19:52:18 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id yPlYEuLhrWNn9QAA9RJhRA (envelope-from ) for ; Thu, 29 Dec 2022 19:52:18 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CB7663C1A3 for ; Thu, 29 Dec 2022 19:52:17 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQo-0000C2-BS; Thu, 29 Dec 2022 13:15:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000Ar-Vi for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQl-0004Cs-MZ for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQl-0007Rf-I4 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 4/8] WIP: substitute: Fetch substitutes using ERIS. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767728510 (code B ref 52555); Thu, 29 Dec 2022 18:15:03 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:37 +0000 Received: from localhost ([127.0.0.1]:32810 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQL-0007Pf-6l for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:37 -0500 Received: from mout01.posteo.de ([185.67.36.65]:54713) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQD-0007OK-34 for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 54D32240149 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337663; bh=HALOp6BmLbWkF/uCoyDNMO6MNjP132bCXeHMxPGwckE=; h=From:To:Cc:Subject:Date:From; b=M9s4m7gqOxDtn1oXZWShSq/dECdvB4UuAcJivvUt4Lgh+/JAMhn0rPDOXrILXnR6E akjsOL1WZGk4ob7R6pHkfkjdh9UCiWAzbW88qZqoE7WVTDG2OutBQEivnJjdkbQ5ox de0LfvMGr2myvEMwb8G2e2E+XQiMUjYt3ronDb0K0yw/G+8FNcwYqSU3zDTce74FQ5 zC1Jz9xVH8Kg8zALAGZr6gdJZhe/awXoBnbYY135OLf/uSNzF9FgaqOkNX4FcwulIx N55XqyozUIgaRn2T6RCDyy27zxYldju+PIjb2HYxhRHyVJ2sHQMd2SxSXXBveShYJ6 xpyP/PHugameQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3f5tmNz6tn8; Thu, 29 Dec 2022 19:14:22 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:23 +0000 Message-Id: <20221229181327.758-5-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1672339937; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=PRVGUA6C3+e48BYGRADNjkAvC5ST/fNBLC5k6yiFaQU=; b=YNcphCwRwn+a4FuAmr9RDrWUExirZ5+c8cB0yVRbNGMUml7oN+LZY8IX7Q00YAyxHGETFA 4nWx1rzF33N3nbv+ImeCz8Nky4uwy3u4N2WyXOjSQUN+EahxEtM4RcxC1ersm0ChHxQF6/ 09hUHLEUZJV+4pBOL5HIlRllbbvAxTiuC/nw3jayhcX7f1LEOib6OFmbUCZbUYvZfHdkJ8 HdFv/yDcquNwF+GX5CHQ1YYfw3AIW3iipa2Guxi9nPEHOCl1xY3/TpAXENj8EbJaOPb3QM M0PZC7Jik0ZJhZwkkbvnhzIA5nytaHERO6xpMjaTDpJOmLlKMg9vjRDMiFokIg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=M9s4m7gq; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1672339937; a=rsa-sha256; cv=none; b=aTOrHvqQPfTaqi+mxnzQS2di7B/PhJMlEgmsq63wxAiimGdCDPVinmF1tc7VYwfuqS1ITK Ka3Y83XV6cy7rIfKrgMHBfvNr7q4Vet4YTnwOIfdecnlQ1dLVIiOmojRmGh3yxEvF7+iFh 6o+gTFPWNRYBfur3HUYKN8wq4B6UPeXtUcyxMnDJt4NHVRKETEcYdA5MIxI3mKdWchVTZ2 LrOKCKaIJ1ttb4hvsrwViBjTsKJmI6mH6R+giofFTU7vH/n4YJAPDoxy/vDxYsTXtcN1XN 2i98ObqeadVU3GpjnAZEHeWYqiKXYaklciJ/Fm4Rd9XGQUnW3kbKLcmg3XFRQw== X-Spam: Yes X-Migadu-Queue-Id: CB7663C1A3 X-Migadu-Spam: Yes X-Spam-Score: 8.77 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=M9s4m7gq; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: 8.77 X-TUID: rAA2hNJpuGoS * guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS. * guix/eris.scm (call-with-eris-block-ref): New procedure. TODO: - When to set prefer-eris? in scripts/substitute.scm? --- guix/eris.scm | 7 +++++++ guix/scripts/substitute.scm | 42 +++++++++++++++++++++++++++++-------- 2 files changed, 40 insertions(+), 9 deletions(-) diff --git a/guix/eris.scm b/guix/eris.scm index 29d5e7b1db..d9a0914b67 100644 --- a/guix/eris.scm +++ b/guix/eris.scm @@ -23,6 +23,7 @@ (define-module (guix eris) #:use-module (guix eris fs-store) #:export (guix-eris-block-reducer + call-with-eris-block-ref %eris-block-store-directory)) @@ -34,3 +35,9 @@ (define %eris-block-store-directory (define (guix-eris-block-reducer) "Returns a block reducer that stores blocks of ERIS encoded content." (eris-fs-store-reducer (%eris-block-store-directory))) + +(define (call-with-eris-block-ref f) + (let ((fs-store-block-ref + (eris-fs-store-ref + (%eris-block-store-directory)))) + (f fs-store-block-ref))) diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index 0efa61b0d7..8cf011d7e6 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -55,12 +55,16 @@ (define-module (guix scripts substitute) #:use-module (ice-9 ftw) #:use-module (rnrs bytevectors) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-2) #:use-module (srfi srfi-19) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) #:use-module (srfi srfi-71) #:use-module (web uri) + #:use-module (eris) + #:use-module (eris read-capability) + #:use-module (guix eris) #:use-module (guix http-client) #:export (%allow-unauthenticated-substitutes? %reply-file-descriptor @@ -439,11 +443,12 @@ (define-syntax-rule (with-cached-connection uri port exp ...) (define* (download-nar narinfo destination #:key status-port - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-eris?) "Download the nar prescribed in NARINFO, which is assumed to be authentic and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and -if DESTINATION is in the store, deduplicate its files. Print a status line to -STATUS-PORT." +if DESTINATION is in the store, deduplicate its files. When PREFER-ERIS? is +true, attempt to ERIS to get the nar. Print a status line to STATUS-PORT." (define destination-in-store? (string-prefix? (string-append (%store-prefix) "/") destination)) @@ -474,14 +479,29 @@ (define (fetch uri) #:port port #:keep-alive? #t #:buffered? #f)))) + ((urn) + (let ((read-capability (->eris-read-capability uri))) + (if (eris-read-capability? read-capability) + (call-with-eris-block-ref + (lambda (block-ref) + (values (open-eris-input-port + read-capability + #:block-ref block-ref) + #f))) + (leave (G_ "unsupported substitute URI scheme: ~a~%") + (uri->string uri))))) (else (leave (G_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) (let ((uri compression file-size - (narinfo-best-uri narinfo - #:fast-decompression? - %prefer-fast-decompression?))) + (if (and prefer-eris? (narinfo-eris-urn narinfo)) + (values (narinfo-eris-urn narinfo) "zstd" #f) + (narinfo-best-uri narinfo + #:fast-decompression? + %prefer-fast-decompression?)))) + + (unless print-build-trace? (format (current-error-port) (G_ "Downloading ~a...~%") (uri->string uri))) @@ -631,7 +651,8 @@ (define* (process-substitution/fallback port narinfo destination (define* (process-substitution port store-item destination #:key cache-urls acl - deduplicate? print-build-trace?) + deduplicate? print-build-trace? + prefer-eris?) "Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to DESTINATION as a nar file. Verify the substitute against ACL, and verify its hash against what appears in the narinfo. When DEDUPLICATE? is true, and if @@ -660,7 +681,8 @@ (define narinfo (download-nar narinfo destination #:status-port port #:deduplicate? deduplicate? - #:print-build-trace? print-build-trace?))) + #:print-build-trace? print-build-trace? + #:prefer-eris? prefer-eris?))) ;;; @@ -858,7 +880,9 @@ (define reply-port #:acl (current-acl) #:deduplicate? deduplicate? #:print-build-trace? - print-build-trace?) + print-build-trace? + ;; TODO when to prefer ERIS? + #:prefer-eris? #t) (loop)))))) (opts (leave (G_ "~a: unrecognized options~%") opts)))))) -- 2.38.1