From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id kKObHJPmrWPhrAAAbAwnHQ (envelope-from ) for ; Thu, 29 Dec 2022 20:12:19 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id 2Et+HJPmrWOVxgAAauVa8A (envelope-from ) for ; Thu, 29 Dec 2022 20:12:19 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 19C311B068 for ; Thu, 29 Dec 2022 20:12:19 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQn-0000BP-W8; Thu, 29 Dec 2022 13:15:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQk-0000A0-W3 for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQk-00042u-2J for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQj-0007R7-UN for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 1/8] publish: Add ERIS URN to narinfo. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767028459 (code B ref 52555); Thu, 29 Dec 2022 18:15:01 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:30 +0000 Received: from localhost ([127.0.0.1]:32798 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQD-0007Og-BY for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500 Received: from mout02.posteo.de ([185.67.36.66]:34393) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQB-0007OG-Mo for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:28 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 1F77C2401A9 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337662; bh=lrwWE98dyL4vG9CQTOKjY0FM2je3h7b72QM1GOagd7I=; h=From:To:Cc:Subject:Date:From; b=rY/0OHumeRqmQHm9ycrIyREYwhfVM4WFZHzDCG5fb7wY7E7FLI9NjGqFeCIRkxTFP 7opk0CVeyfn9FQc3ITWWnwdXqAe3yfNkL6CQTpxqzckPElDbPWni1mkx+ffjW7pWXo qL7tCFc7e7rjpyxh7ZVNn3iiUyMyhLJ+BgyhQecUCsVbSKWsvLLvecmC48xZjMa+Ma BboQRiziZ1Pqt/Kebv5WIKp8m+Yb0LmQ3Xet8DLxQx+p9/yL8j3Dj5wn8DkCc9zenx E9UAbBOqX1FQxCKW/5nkBesBl2uQ9t/kZxBIN7ZsHTa9+qQh1U+pfTI6PkYEG5doSJ OFPXpbZR0hu4A== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3d3ZBBz6tmY; Thu, 29 Dec 2022 19:14:21 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:20 +0000 Message-Id: <20221229181327.758-2-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1672341139; a=rsa-sha256; cv=none; b=sOH5177biAVAej1k2W2+pOiSznDzk9GQ8iJEIbN7lU3Uv7PgQI9PFdECdMaaip1fGa/lj7 rx1cYuDWD2J2eM+zu0GQcjuV4fs0tOCyoeYBUzRY58qTdZHpAJPVvwVfnoz9vcsUVoKphQ 6yxkn+gmfmKGvC6tzB9pCWk3QiLRuCcwKH4d9cAlGBJTxf/gMWu4/3gzv2/nD/WttSwfmy H15zRCTEkzPqI7fQ+Tk1/sFYn/39Oq+8YF4UvCBxQ2/5QbivKefXr0/NNQ6iIYeXnIDIx4 qqTDLqqkgY6GNtRYoH7RyeaMWN8mlm28U1dCYIBSdawY9Hht43HC85rLmgJ22g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b="rY/0OHum"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1672341139; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=JHOndYn5RNofgKLxtJAwiJC1hWEYkL+KYb83HMRYYco=; b=ptOjSqnUIctFLi+5z42brAQGQFdIyiCOoKt3Ws1r/ZQ1NlLxauLMPZ3vODGN0p80nHs1wR Zpk1vJzPSr6o7sm0m/v48iqIA8iPqrUkOkDdZ8mfJKm9ccePrG6hcUhwfHfMWt9fGj0ArD VHBkcvl/1hPaQfdQxkVINQFlbCBngwBfBfz3Os5/TTVsqlAkU+JuBWIzwvWSqQxa+00dw6 DtxGAssYwqub+FbySfjNvmk9vH+kS95epnh9/CLiGNlytZUu0T7WLmCIRaOn9exFO6eKBP 5oh9LMchb5UGuMcoGOL1oFbLdGJPqbMAiQAm9XpgcXQHjCN9pmF1sLQS4DWa6A== X-Spam-Score: 0.27 X-Migadu-Queue-Id: 19C311B068 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b="rY/0OHum"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) X-Migadu-Scanner: scn0.migadu.com X-Migadu-Spam-Score: 0.27 X-TUID: CdR9Oz7uNgo0 * guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars. (narinfo-string): Add #:eris-urn parameter and honor it. * guix/scripts/narinfo.scm: ()[eris-format,eris-urn]: New fields. (narinfo-maker): Handle ERIS URN and ERIS format. * configure.ac: (HAVE_GUILE_ERIS): New conditional. * gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris. --- configure.ac | 5 +++++ gnu/packages/package-management.scm | 1 + guix/narinfo.scm | 16 +++++++++----- guix/scripts/publish.scm | 34 ++++++++++++++++++++++++----- 4 files changed, 46 insertions(+), 10 deletions(-) diff --git a/configure.ac b/configure.ac index 92dede8014..52a37a6113 100644 --- a/configure.ac +++ b/configure.ac @@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)]) AM_CONDITIONAL([HAVE_GUILE_AVAHI], [test "x$have_guile_avahi" = "xyes"]) +dnl Check for Guile-eris. +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)]) +AM_CONDITIONAL([HAVE_GUILE_ERIS], + [test "x$have_guile_eris" = "xyes"]) + dnl Guile-newt is used by the graphical installer. GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)]) diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm index 2ffaa12247..3585f6c073 100644 --- a/gnu/packages/package-management.scm +++ b/gnu/packages/package-management.scm @@ -419,6 +419,7 @@ (define code ("guile-zstd" ,guile-zstd) ("guile-ssh" ,guile-ssh) ("guile-git" ,guile-git) + ("guile-eris" ,guile-eris) ;; XXX: Keep the development inputs here even though ;; they're unnecessary, just so that 'guix environment diff --git a/guix/narinfo.scm b/guix/narinfo.scm index 741c7ad406..d7f8c88f8f 100644 --- a/guix/narinfo.scm +++ b/guix/narinfo.scm @@ -45,6 +45,8 @@ (define-module (guix narinfo) narinfo-file-sizes narinfo-hash narinfo-size + narinfo-eris-format + narinfo-eris-urn narinfo-references narinfo-deriver narinfo-system @@ -68,8 +70,8 @@ (define-module (guix narinfo) (define-record-type (%make-narinfo path uri-base uris compressions file-sizes file-hashes - nar-hash nar-size references deriver system - signature contents) + nar-hash nar-size eris-format eris-urn references deriver + system signature contents) narinfo? (path narinfo-path) (uri-base narinfo-uri-base) ;URI of the cache it originates from @@ -79,6 +81,8 @@ (define-record-type (file-hashes narinfo-file-hashes) (nar-hash narinfo-hash) (nar-size narinfo-size) + (eris-format narinfo-eris-format) + (eris-urn narinfo-eris-urn) (references narinfo-references) (deriver narinfo-deriver) (system narinfo-system) @@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url) "Return a narinfo constructor for narinfos originating from CACHE-URL. STR must contain the original contents of a narinfo file." (lambda (path urls compressions file-hashes file-sizes - nar-hash nar-size references deriver system - signature) + nar-hash nar-size eris-format eris-urn references deriver + system signature) "Return a new object." (define len (length urls)) (%make-narinfo path cache-url @@ -157,6 +161,8 @@ (define len (length urls)) ((lst ...) (map string->number lst))) nar-hash (and=> nar-size string->number) + eris-format + (if eris-urn (string->uri eris-urn) #f) (string-tokenize references) (match deriver ((or #f "") #f) @@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url (narinfo-maker str url) '("StorePath" "URL" "Compression" "FileHash" "FileSize" "NarHash" "NarSize" - "References" "Deriver" "System" + "ERISFormat" "ERIS" "References" "Deriver" "System" "Signature") '("URL" "Compression" "FileSize" "FileHash")))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 3bf3bd9c7c..76b5a429f4 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -40,6 +40,7 @@ (define-module (guix scripts publish) #:use-module (srfi srfi-34) #:use-module (srfi srfi-37) #:use-module (srfi srfi-71) + #:use-module (srfi srfi-171) #:use-module (web http) #:use-module (web request) #:use-module (web response) @@ -57,6 +58,7 @@ (define-module (guix scripts publish) #:use-module (guix workers) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) + #:use-module (eris) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -145,6 +147,9 @@ (define %default-gzip-compression ;; Since we compress on the fly, default to fast compression. (compression 'gzip 3)) +(define %eris-zstd-compression + (compression 'zstd 19)) + (define (default-compression type) (compression type 3)) @@ -323,7 +328,8 @@ (define* (store-item->recutils store-item (define* (narinfo-string store store-path #:key (compressions (list %no-compression)) - (nar-path "nar") (file-sizes '())) + (nar-path "nar") (file-sizes '()) + eris-urn) "Generate a narinfo key/value string for STORE-PATH; an exception is raised if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs. @@ -345,10 +351,10 @@ (define* (narinfo-string store store-path "\ StorePath: ~a NarHash: sha256:~a -NarSize: ~d +NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~] References: ~a~%" store-path - hash size references)) + hash size eris-urn references)) ;; Do not render a "Deriver" line if we are rendering info for a ;; derivation. Also do not render a "System" line that would be ;; expensive to compute and is currently unused. @@ -633,6 +639,22 @@ (define (compressed-nar-size compression) (and stat (cons compression (stat:size stat))))) + (define (eris-encode-nar compressions) + (and (member %eris-zstd-compression compressions) + (let* ((nar (nar-cache-file cache item + #:compression %eris-zstd-compression)) + (stat (stat nar #f))) + (and stat + (call-with-input-file nar + (lambda (port) + (let ((eris-urn _ + (eris-encode port + #:block-size 'large + #:block-reducer rcount + #:convergence-secret + %null-convergence-secret))) + eris-urn))))))) + (let ((compression (actual-compressions item compressions))) (for-each (cut compress-nar cache item <>) compressions) @@ -640,7 +662,8 @@ (define (compressed-nar-size compression) (match compressions ((main others ...) (let ((narinfo (narinfo-cache-file cache item - #:compression main))) + #:compression main)) + (eris-urn (eris-encode-nar compressions))) (with-atomic-file-output narinfo (lambda (port) ;; Open a new connection to the store. We cannot reuse the main @@ -651,7 +674,8 @@ (define (compressed-nar-size compression) (display (narinfo-string store item #:nar-path nar-path #:compressions compressions - #:file-sizes sizes) + #:file-sizes sizes + #:eris-urn eris-urn) port))) ;; Make the cached narinfo world-readable, contrary to what -- 2.38.1